Subway lightweight video positioning method based on sparse optical flow algorithm

Abstract

The application provides a subway lightweight video positioning method based on a sparse optical flow algorithm, which comprises the following steps: collecting images in front of the subway during driving, and performing ROI region segmentation on the images; performing gray scale processing on the ROI images and performing corner point recognition; performing optical flow tracking on the recognized corner points by using an L-K optical flow recognition algorithm based on an image pyramid, estimating the new positions of the corner points in the current frame according to the pixel intensity change in the local window; screening the target points, filtering noise, screening the effective displacement trajectories of the target points, and assigning confidence values to the effective target points; converting the pixel coordinate system into a three-dimensional world coordinate system to obtain the three-dimensional coordinates of the target points; solving the real displacement and the instantaneous speed of each target point between adjacent frames; weightedly solving the actual displacement and the real-time speed of the whole vehicle; acquiring the actual distances between stations of a subway line, and judging whether the subway arrives at a station according to the driving distance between the stations and the real-time speed. The application realizes subway positioning by using only a front camera and a vehicle-mounted computer, reduces a large number of trackside equipment, realizes the lightweight of subway positioning, and reduces the equipment maintenance cost.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and subway positioning, and particularly relates to a subway lightweight video positioning method based on a sparse optical flow algorithm. BACKGROUND

[0002] Accurate real-time subway position information is a key to ensure safe operation of a subway in a subway operation control system. A commonly used subway positioning method is speed measurement positioning, which determines the running direction and distance of the subway through a speed measurement sensor, and then calculates the actual position of the subway on the line. However, because of the slippage of subway wheel pairs and other reasons, the speed measurement positioning error is large. In addition, track circuits, transponders, GPS, WLAN and LTE can also be used for subway positioning, but these positioning technologies have certain shortcomings, such as low positioning accuracy, great influence of external environment, high investment cost and difficult post-maintenance and the like. At present, there is a similar subway positioning system based on visible light communication and binocular vision in China, but the overall positioning system is relatively complex, needs more trackside equipment for auxiliary positioning, has a large maintenance amount and a high failure rate. SUMMARY

[0003] The application provides a subway lightweight video positioning method based on a sparse optical flow algorithm.

[0004] The technical scheme for achieving the object of the application is as follows: a subway lightweight video positioning method based on a sparse optical flow algorithm, comprising the following steps.

[0005] Step 1: collecting images in front of the subway during driving, and performing ROI region segmentation on the images;

[0006] Step 2: performing gray scale processing on the ROI images;

[0007] Step 3: performing corner point recognition on the ROI images after the gray scale processing;

[0008] Step 4: performing optical flow tracking on the recognized corner points by using an L-K optical flow recognition algorithm based on an image pyramid, and estimating the new position of the corner points in the current frame according to the pixel intensity change in the local window;

[0009] Step 5: screening the target points, filtering noise, screening the effective displacement trajectories of the target points, and assigning confidence values to the effective target points;

[0010] Step 6: converting the pixel coordinate system into a three-dimensional world coordinate system to obtain the three-dimensional coordinates of the target points;

[0011] Step 7: solving the real displacement and instantaneous speed of each target point between adjacent frames;

[0012] Step 8, weighted solution of the actual displacement and real-time speed of the whole vehicle;

[0013] Step 9, obtaining the actual distance between each station of the subway line, judging whether the subway arrives at the station according to the driving distance between the subway stations and the real-time speed.

[0014] Preferably, the specific method of corner point recognition for the ROI image after gray scale processing is:

[0015] Calculate the gradient of the ROI image after gray scale processing in the x and y directions:

[0016]

[0017] Wherein, I represents the pixel intensity of the image;

[0018] For each pixel point in the image, a self-correlation matrix M is constructed, representing the quadratic sum of the gradient in a window:

[0019]

[0020] Calculate two eigenvalues λ1 and λ2 of the matrix M;

[0021] Convert the self-correlation matrix M into a 2x2 symmetric matrix expression, as follows:

[0022]

[0023] Wherein: B = ΣI x I y ,

[0024] Eigenvalues are obtained by solving the characteristic equation det(M-λI)=0, and the characteristic equation is:

[0025]

[0026] Calculate the determinant to obtain:

[0027] (A-λ)(C-λ)-B 2 =0

[0028] Expand and simplify to obtain a quadratic equation in one unknown:

[0029] λ 2 -(A+C)λ+(AC-B 2 )=0

[0030] Use the root formula to solve the eigenvalues:

[0031]

[0032] The eigenvalues λ1 and λ2 are respectively:

[0033]

[0034]

[0035] The smaller eigenvalue is used as the corner response function:

[0036] R = min(λ1, λ2)

[0037] When the smaller eigenvalue is greater than a set threshold, the point is considered as a corner point.

[0038] Preferably, the identified corner points are tracked by using the L-K optical flow identification algorithm based on the image pyramid, and the new position of the corner point in the current frame is estimated according to the pixel intensity change in the local window, and the specific method is as follows:

[0039] An image pyramid is constructed, and the current frame image I t and the next frame image I t+1 are input into the image pyramid, and in each layer of the image pyramid, the new position of the corner point in the current frame is estimated according to the pixel intensity change in the local window, and the specific process is as follows:

[0040] The estimated optical flow (u i , v i ) of each layer of the image pyramid is calculated by the least square method from the top layer of the image pyramid to the bottom layer step by step;

[0041] On the image of each layer, the estimated optical flow is used as the initial optical flow, and the initial optical flow is locally refined and synthesized until the bottom layer to obtain the final optical flow estimation.

[0042] Preferably, the estimated optical flow (u i , v i ) of each layer of the image pyramid is calculated by the least square method from the top layer of the image pyramid to the bottom layer step by step, and the specific formula is as follows:

[0043]

[0044] Where A, B, and C are coefficients for image gradient calculation, and D and E are intensity change terms, and the specific formula is as follows:

[0045]

[0046] I x and I y are the gradients of the image in the x and y directions, I t is the time gradient, and w represents the window area in the optical flow calculation.

[0047] Preferably, on the image of each layer, the estimated optical flow is used as the initial optical flow, and the initial optical flow is locally refined and synthesized until the bottom layer to obtain the specific method of final optical flow estimation:

[0048] The optical flow estimation (u i ,v i ) calculated for the i-th layer is up-sampled as the initial value of the optical flow calculation iteration of the (i-1)-th layer

[0049] The following equation is solved using the least square method on the image gradient and the time gradient in the (i-1)-th layer:

[0050] I x Δu+I y Δv+I t =0

[0051] where I x and I y are the gradients of the image in the x and y directions, and I t is the time gradient.

[0052] The optical flow increment from the i-th layer to the (i-1)-th layer is obtained:

[0053]

[0054] where I x and I y are the gradients of the image in the x and y directions, and I t is the time gradient.

[0055] Preferably, the target points are screened, noise is filtered, the effective displacement trajectory of the target points is screened, and the specific method of assigning confidence values to the effective target points is as follows:

[0056] According to the tracking result status and the optical flow calculation error err, the target points are preliminarily screened to remove points with large tracking errors in the corner point sequence and corner points with failed tracking.

[0057] The motion slope and distance of each corner point in adjacent frames are calculated. If the slope is greater than 8 and the moving distance of adjacent frames is greater than 5 pixels, it is considered that the motion of the corner point is effective motion and is not random noise caused by the shaking of the subway itself. The motion trajectory is drawn on the mask image, and the corner point is marked on the current frame image.

[0058] If the tracking is effective motion in consecutive multiple frames, the tracking success times are normalized as weights as the confidence of the pixel displacement of the effective target points.

[0059] Preferably, the tracking success times of the i-th target point are T i , and the sum of the tracking success times of all effective target points is Tt otal, the weight w i of the i-th target point is:

[0060]

[0061] where T i is the number of tracking successes of the i-th target point, T total is the sum of the tracking successes of all valid target points.

[0062] Preferably, the specific method for solving the real displacement of each target point between adjacent frames and the instantaneous speed is:

[0063] For each target point, the displacement is calculated by the following formula:

[0064] Δx = x t - x t-1

[0065] Δy = y t - y t-1

[0066] where (x t , y t ) is the three-dimensional world coordinate corresponding to the target point in the current frame, (x t-1 , y t-1 ) is the three-dimensional world coordinate of the target point in the previous frame, and Δx and Δy represent the displacement in the x and y directions, respectively.

[0067] The Euclidean distance of the target point is:

[0068]

[0069] where d represents the displacement distance of the target point.

[0070] The average speed obtained within adjacent frames is taken as the real-time speed:

[0071]

[0072] where v is the real-time speed of the target point, d is the displacement distance of the target point, and Δt is the time difference between two frames.

[0073] Preferably, the specific method for solving the real displacement and real-time speed of the whole vehicle by weighting is:

[0074] Let Δx i and Δy i represent the displacement of the i-th valid target point in the x and y directions, respectively; w i represents the normalized weight of the i-th valid target point, Δx weighted , Δy weightedThis represents the weighted displacement of the entire vehicle in the x and y directions, and N represents the total number of valid target points.

[0075] Based on the displacement (Δx) of the i-th valid target point in the x and y directions i Δy i Calculate the vector displacement d of the i-th effective target point. i and instantaneous velocity v i :

[0076]

[0077] Weighted calculation of the vehicle's real-time speed:

[0078]

[0079] In the formula, w i Let be the weight of the i-th target point, and N be the total number of target points;

[0080] Weighted calculation of the vehicle's displacement in the x and y directions:

[0081]

[0082] Calculate the actual displacement of the entire vehicle:

[0083]

[0084] Compared with the prior art, the significant advantages of this invention are:

[0085] (1) Positioning method based on optical flow tracing: High-precision positioning is achieved by utilizing the continuous and stable image features in the subway operating environment and the relatively stable speed of the subway. This method is currently less common in subway positioning research.

[0086] (2) Confidence assessment of optical flow tracing: Image noise is filtered by filtering algorithm, and the confidence of feature points is assessed by using factors such as the slope of optical flow trajectory and the stability of successful tracking in multiple consecutive frames. Weights are assigned to the tracking results to improve the accuracy and reliability of positioning.

[0087] (3) The subway positioning method is less affected by the subway driving environment. It adopts stable and continuous visual features for tracking and positioning, and can be effectively positioned in both tunnel lines and ground lines. It can also be effectively identified and positioned in low visibility environments such as rain, snow, and heavy fog at night, and has good robustness.

[0088] (4) This invention uses only the front camera and onboard computer to achieve subway positioning, which reduces a large number of trackside equipment, realizes the lightweighting of subway positioning, and reduces equipment maintenance costs.

[0089] The application will be described in further detail below with reference to the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0090] Figure 1 Fig. 2 is a schematic diagram of ROI region selection and recognition effect, Figure 1 Fig. 2(a) is a schematic diagram of ROI region selection in a tunnel, Figure 1 Fig. 2(b) is a schematic diagram of ROI region optical flow recognition effect in a tunnel, Figure 1 Fig. 2(c) is a schematic diagram of ROI region selection in a ground line, Figure 1 Fig. 2(d) is a schematic diagram of ROI region optical flow recognition effect in a ground line.

[0091] Figure 2 Fig. 3 is a schematic diagram of filtering noise and screening effective target trajectory effect in an image, Figure 2 Fig. 3(a) is subway motion noise, Figure 2 Fig. 3(b) is subway station stopping noise, Figure 2 Fig. 3(c) is a schematic diagram of effective target point trajectory.

[0092] Figure 3 Fig. 4 is a flowchart of the application.

[0093] Figure 4 Fig. 5 is a structure of an image pyramid.

[0094] Figure 5 Fig. 6 is a flowchart of LK algorithm.

[0095] Figure 6 Fig. 7 is a screenshot of a tunnel video.

[0096] Figure 7 Fig. 8 is a test system interface.

[0097] Figure 8 Fig. 9 is a test effect in a low-visibility environment. DETAILED DESCRIPTION

[0098] A subway lightweight video positioning method based on a sparse optical flow algorithm, and the specific steps are as follows:

[0099] Step 1: The camera installed at the front of the subway train transmits the real-time image of the front of the subway train, and the image to be processed is segmented into an ROI region.

[0100] The subway driving environment can be divided into ground line and tunnel line, and different ROI segmentation methods are adopted for the two kinds of line environments:

[0101] For the ground line, since the surrounding environment light of the ground line changes greatly, the fixed ROI region is the steel rail region, and no dynamic adjustment is performed;

[0102] For the tunnel line, the tunnel walls on both sides of the subway driving environment are selected as the ROI region.

[0103] After selecting the ROI region, perspective transformation is needed to make the ROI region more suitable for subsequent corner recognition and optical flow tracking processing. Define the four corner points of the ROI region, and then define the positions of the four corner points in the ROI image (size: 200x500) to calculate the perspective transformation matrix. The warpPerspective function in the OpenCV library is used for perspective transformation.

[0104] Perspective transformation maps a point in an image to another image through a 3x3 homography matrix H.

[0105]

[0106] Given a point (x, y) in the source image and a point (x', y') in the target image,

[0107]

[0108] where w' is the normalization factor of the homogeneous coordinate.

[0109] Convert the homogeneous coordinate to the actual image coordinate by normalization. The normalized coordinate is calculated as:

[0110]

[0111] The effect diagram after ROI selection and perspective transformation is shown in the following figure: Figure 1 .

[0112] Step 2, perform grayscale processing on the ROI image to facilitate subsequent corner detection.

[0113] Step 3, perform corner recognition on the ROI image after grayscale processing to extract feature points for optical flow recognition.

[0114] The Shi-Tomasi corner detection algorithm is an improved algorithm of the Harris corner detection algorithm, which simplifies the calculation of the response function and only uses a smaller eigenvalue to judge the corner.

[0115] First, calculate the gradient of the ROI image after grayscale processing in the x and y directions:

[0116]

[0117] where I represents the pixel intensity of the image.

[0118] For each pixel in the image, construct an autocorrelation matrix M, which represents the quadratic sum of the gradient in a window:

[0119]

[0120] Then, two eigenvalues λ1 and λ2 of the matrix M are calculated.

[0121] Since the autocorrelation matrix M is a 2x2 symmetric matrix, it has the following form:

[0122]

[0123] Where: B = ∑I x I y ,

[0124] The eigenvalues are obtained by solving the characteristic equation det(M - λI) = 0. The characteristic equation is:

[0125]

[0126] The determinant is calculated to get:

[0127] (A - λ)(C - λ) - B 2 = 0

[0128] Expanding and simplifying, we get a quadratic equation in one variable:

[0129] λ 2 -(A + C) λ + (AC - B 2 ) = 0

[0130] The eigenvalues are solved using the root formula:

[0131]

[0132] Therefore, the eigenvalues λ1 and λ2 are:

[0133]

[0134] Finally, the smaller eigenvalue is used as the corner response function:

[0135] R = min(λ1, λ2)

[0136] When the smaller eigenvalue λ is greater than a certain threshold, the point is considered to be a corner.

[0137] In this method, the goodFeaturesToTrack function in the OpenCV library is used to input the grayscale image using the Shi-Tomasi algorithm, and the detected corner vector is output for subsequent optical flow recognition.

[0138] Where, consider the parameters as:

[0139] The number of detected corner points is set to be at most 100, and the quality level is set to be 0.25, indicating the minimum quality level that the detected corner points must reach. If the feature value of a certain point is lower than 0.25, the point will not be detected as a corner point. The minimum Euclidean distance between the detected corner points is set to be 7 pixels.

[0140] In order to ensure the stability and real-time performance of corner point detection, the corner points are re-detected every 10 frames or when the corner point vector p0 is empty.

[0141] Step 4, using the L-K optical flow recognition algorithm based on image pyramid to track the recognized corner points, estimating the new position of the corner points in the current frame according to the pixel intensity change in the local window, specifically including:

[0142] (1) For the current frame image I t and the next frame image I t+1 , construct an image pyramid. The image pyramid is generated by layer-by-layer down-sampling of the image.

[0143] Pyramid level setting:

[0144] (2) Set the number of layers of the image pyramid L = 3 and the scaling factor of each layer (default is 0.5). The bottom layer of the pyramid is the original image, and the other layers are images with gradually reduced resolution, with the top layer having the lowest resolution.

[0145] (3) Optical flow calculation:

[0146] In each layer of the image pyramid, the new position of the corner points in the current frame is estimated according to the pixel intensity change in the local window. The Lucas-Kanade method is used to calculate the optical flow, and the optical flow of the current layer is set to (u i ,v i ).

[0147] Set the pixel intensity change in the local window as:

[0148] I x Δu+I y Δv+I t =0

[0149] where I x and I y are the gradients of the image in x and y directions, I t is the temporal gradient, and (Δu, Δv) is the optical flow increment.

[0150] (4) Least squares optimization

[0151] The estimated optical flow (u i ,v i), minimizing the difference of pixel intensity, and getting the optimal optical flow vector in the image window of the layer, so that the difference of pixel intensity of the target point in the two frames is minimized.

[0152] The least square solution of optical flow in the i-th layer image is calculated (u i ,v i ), and the specific process is as follows:

[0153]

[0154] Where A, B, C are the coefficients of image gradient calculation, and D, E are the intensity change terms. Specifically:

[0155]

[0156] The summation sign indicates summation within the local window.

[0157] (5) Pyramid level processing:

[0158] Starting from the top layer of the image pyramid, then processing step by step down, using the least square method to calculate the estimated optical flow (u i ,v i ) of each layer, using the initial result of optical flow estimation (up-sampling result from the low resolution layer) as initialization on each layer of image, and performing local refinement and synthesis, until the bottom layer, to get the final optical flow estimation:

[0159] Assuming that the optical flow estimation of the i-th layer (lower resolution layer) calculated by the least square method is (u i ,v i ), after up-sampling, it is used as the initial value of the optical flow calculation iteration solution of the i-1-th layer (higher resolution layer) The purpose of up-sampling is to make it match the image resolution of the i-1-th layer, for example, when the scaling factor is 0.5, the image resolution is increased by 1 times from the i-th layer to the i-1-th layer, and the optical flow vector also needs to be enlarged accordingly, to get

[0160] When the scaling factor is 0.5, the up-sampling formula is:

[0161]

[0162] In the formula, u i-1 (x,y) 0 is the up-sampled optical flow vector u in the i-1-th layer; v i-1 (x,y) 0 is the up-sampled optical flow vector v in the i-1-th layer; is the corresponding optical flow vector u in the i-th layer; is the optical flow vector corresponding to the i-th layer.

[0163] The following equation is solved for the image gradient and the temporal gradient at the i-1-th layer:

[0164] I x Δu+I y Δv+I t = 0

[0165] where I x and I y are the gradients of the image in the x and y directions, I t is the temporal gradient.

[0166] The optical flow increment from the i-th layer to the i-1-th layer is calculated using the least square method:

[0167]

[0168] The initial optical flow estimate is added to the calculated increment to obtain the final optical flow estimate

[0169]

[0170] which is passed as input to the higher resolution optical flow calculation of the next layer (i-2-th layer). Through the layer-by-layer processing and synthesis, the accurate optical flow estimate is finally obtained on the highest resolution image.

[0171] (5) The new position estimate of each corner point in the current frame is output.

[0172] In the specific implementation, the optical flow tracking is performed using the OpenCV library function, and the specific steps are as follows:

[0173] The optical flow tracking is initialized, and two Point2f type containers (vectors) are defined to store the tracking state and tracking error of each corner point. Then the function calcOpticalFlowPyrLK(old_gray, frame_gray, p0, p1, status, err, Size(15, 15), 3, criteria) in the OpenCV library is used to perform optical flow tracking on the corner points. The input is the gray image of the previous frame ROI old_gray and the gray image of the current frame ROI frame_gray, and the corner point coordinates p0 in the previous frame; the output is the corner point coordinates p1 tracked in the current frame, the tracking result status (whether the tracking is successful), and the optical flow calculation error err.

[0174] Here, the parameters to be considered are "window size for computing optical flow" (Size(15,15)), "number of pyramid levels" (3), and "stopping criteria for optical flow computation" (criteria):

[0175] An image pyramid is a multi-scale representation of an image, consisting of images at different resolutions. The bottom layer (highest resolution) of the pyramid is the original image, while the upper layers are progressively downsampled images, with each layer having a lower resolution than the previous one. In each layer of the pyramid, the actual size of the window is the same (e.g., 15x15 pixels), but since the resolution of each layer is different, the actual area covered by the window varies with the level, allowing for the advantages of both large and small windows to be considered simultaneously. This ensures that the optical flow is computed at different scales while maintaining consistency in the window size, enabling the optical flow computation to effectively handle both local and global motion information. The stopping criteria for the optical flow computation include specifying a maximum number of iterations and an error threshold. Typically, the number of pyramid levels is set to 3, and the window size is Size(15,15).

[0176] The stopping criteria for the iterative algorithm are set using the TermCriteria class from the OpenCV library, which has a constructor that takes three arguments:

[0177] TermCriteria::COUNT: the maximum number of iterations is reached.

[0178] TermCriteria::EPS: the specified accuracy error is reached.

[0179] TermCriteria::COUNT + TermCriteria::EPS: both conditions are met.

[0180] Taking into account the time consumption and error requirements of the algorithm, the maximum number of iterations is set to 10, and the error threshold is set to 0.03.

[0181] Step 5: Select the target points, filter out noise, select the effective displacement trajectories of the target points, and assign confidence values to the effective target points (if visualization is required, draw them).

[0182] Based on the tracking results status and the optical flow computation error err, the target points are preliminarily screened to remove points with large tracking errors and failed corner points in the corner sequence.

[0183] The motion slope and distance of each corner point in adjacent frames are calculated. If the slope is greater than 8 and the moving distance of adjacent frames is greater than 5 pixels, it is considered that the motion of the corner point is effective motion, not random noise caused by the shaking of the subway itself. The OpenCV library function line(mask, p1[i], p0[i], colors[i], 2) is used to draw the motion trajectory on the mask image, and the OpenCV library function circle(frame, p1[i], 5, colors[i], -1) is used to mark the corner point on the current frame image:

[0184] The Line function is a function in the OpenCV library for drawing a straight line. Mask represents the mask image, p1[i], p0[i] represents the starting point and ending point of the i-th straight line. colors[i] represents the color of the i-th straight line. 2 represents the thickness of the straight line. Here, it means that the width of the straight line is set to 2 pixels.

[0185] The circle function is a function in the OpenCV library for drawing a circle. frame represents the target image for drawing a circle, p1[i] represents the center of the i-th circle to be drawn, 5 represents the radius of the circle, which is 5 pixels, colors[i] represents the color of the i-th circle to be drawn; -1 represents the filling method of the circle, which is solid.

[0186] If the tracking of consecutive multiple frames is effective motion, the number of successful tracking is normalized as a weight, which is the confidence of the pixel displacement of the effective target point, and is used for the calculation of the pixel displacement of the whole vehicle.

[0187] Let the number of successful tracking of the i-th target point be T i , and the sum of the number of successful tracking of all effective target points be T t otal, then the weight w i of the i-th target point can be calculated as:

[0188]

[0189] The random noise caused by the shaking of the subway itself and the effective trajectory effect diagram of the target point obtained after filtering are shown in the accompanying Figure 2 .

[0190] Continue to pass in the next frame of picture, use the pyramid L-K optical flow recognition algorithm to realize the continuous optical flow tracking of the target point, save the continuous frame pixel point coordinates of the effective target point, until all pictures are processed.

[0191] Step 6, convert the pixel coordinate system to a three-dimensional world coordinate system to obtain the three-dimensional coordinates of the target point.

[0192] The result obtained by optical flow tracking is limited to the displacement information at the image pixel level, and does not provide accurate motion information of the subway in the real world. By converting the pixel coordinates into coordinates in the world coordinate system, the displacement information obtained by optical flow tracking can be mapped to the real world, thereby obtaining the real motion of the subway, and further obtaining the actual displacement of the subway in the three-dimensional world.

[0193] The coordinates of the camera in the world can be represented by a rotation matrix R and a translation matrix T. If a camera is placed in the world coordinate system, the translation of the camera coordinates relative to the origin of the world coordinate system can be recorded as a vector, and the rotation angle can be recorded as R. Any translation includes three directions, which can be represented by a three-dimensional vector (x, y, z); for the rotation matrix R, the rotation matrix is:

[0194]

[0195]

[0196] Therefore, the transformation from the world coordinate system to the camera coordinate system can be written as:

[0197]

[0198] The imaging model of a general camera is a pinhole imaging model. By analyzing the geometric relationship, the relationship between the coordinates of a 3D target point in the camera coordinate system and the image pixel coordinates can be obtained. In order to facilitate calculation, a new dimension is introduced to convert the coordinates into homogeneous coordinates:

[0199]

[0200] (u0, v0) is the offset from the center of the image coordinate system to the center of the pixel coordinate system, which belongs to the camera intrinsic parameters. f x ,f y is the projection amount of the camera focal length on the x and y axes.

[0201] Therefore, the relationship between the pixel coordinate system and the world coordinate system can be represented as follows:

[0202]

[0203] wherein, is the camera intrinsic parameter, is the camera extrinsic parameter, i.e.:

[0204]

[0205] This method needs to calculate the real coordinates of the target feature points in the world coordinate system according to the pixel coordinates, i.e. the inverse transformation from the world coordinate system to the pixel coordinate system:

[0206]

[0207] (1) Since the application considers calculating the subway advancing distance through the displacement of feature points between adjacent frames, there is no need to obtain the absolute coordinates of the feature points, so the camera coordinates can be set as the origin of the world coordinate system, that is, the camera coordinate system and the origin of the world coordinate system coincide.

[0208] (2) By fixing the camera at a certain fixed position of the head of the subway, the intrinsic and extrinsic matrices of the camera can be calculated in advance. The intrinsic parameters of the camera include focal length, optical center and distortion coefficient; the extrinsic parameters include rotation matrix and translation vector.

[0209] The binocular camera is calibrated using Zhang Zhengyou method:

[0210] First, multiple checkerboard images are taken at different angles, and the coordinates of each corner point in the image are extracted.

[0211] Then, using the single target calibration algorithm, the intrinsic matrix K and the distortion parameter D of the left camera and the right camera are solved respectively.

[0212]

[0213] Wherein:

[0214] (X, Y, Z) is a point in three-dimensional space.

[0215] (u, v) is the corresponding two-dimensional image point.

[0216] s is the scaling factor.

[0217] K is the intrinsic matrix of the camera, defined as:

[0218] R is the rotation matrix, and t is the translation vector.

[0219] For each image, the homography matrix H is solved using the corner point coordinates, which satisfies the following relationship:

[0220]

[0221] The relationship between the homography matrix H and the intrinsic matrix K and the extrinsic matrix is:

[0222] H = K[r1 r2 t]

[0223] Where r1 and r2 are the first two columns of the rotation matrix R.

[0224] By optimizing the homography matrix H of multiple images, the intrinsic matrix K is solved:

[0225] v ij = [h i1h j1 h i1 h j2 +h i2 h j1 h i2 h j2 h i3 h j1 +h i1 h j3 h i3 h j2 +h i2 h j3 h i3 h j3 ]

[0226] Vb = 0

[0227] where V is a matrix composed of v ij and b is a vector containing the intrinsic matrix elements. Solve b by SVD decomposition, and further solve K.

[0228] After solving the intrinsic matrix K, use the homography matrix H to calculate the extrinsic matrix of each image:

[0229] r1 = K -1 h1

[0230] r2 = K -1 h2

[0231] t = K -1 h3

[0232] Orthogonalize r1 and r2, and calculate r3: r3 = r1 x r2

[0233] On the basis of single target calibration, solve the relative rotation matrix R and translation vector T between the two cameras by stereo calibration algorithm:

[0234]

[0235] Finally, perform stereo rectification. Align the image planes of the two cameras, so that the epipolar geometry is simplified. Calculate new rotation matrices R1 and R2 for stereo rectification, and project the images of the two cameras to the same view plane:

[0236] Calculate new projection matrices P1 and P2 to align the images to the same view plane:

[0237] P1 = K1 [I | 0]

[0238] P2 = K2 [R | T]

[0239] Wherein: K1, K2 are the intrinsic matrix of left and right cameras respectively; R, T are the rotation matrix and translation vector of the camera respectively.

[0240] Recalculate the new rotation matrix:

[0241]

[0242] Wherein R rect is a rotation matrix for aligning views, R 1′ and R 2′ are the original rotation matrix respectively.

[0243] Using the calculated rotation matrix and projection matrix, the image is transformed so that the same object in the two images has the same vertical coordinates.

[0244] x 1′ = R1x1

[0245] x 2′ = R2x2

[0246] Wherein x1, x2 are the coordinates of the points in the left and right images respectively, x 1′ , x 2′ are the transformed point coordinates.

[0247] Thus, the calibration of the binocular camera is completed.

[0248] (3) Because the depth information is lost in the conversion from the world coordinate to the camera picture, additional depth information is usually needed to perform 2D inverse transformation to the 3D coordinate system.

[0249] Using the binocular camera, the depth information can be obtained:

[0250] The binocular camera system consists of two cameras, the left camera and the right camera, and through calibration, their relative position and attitude are known. Assuming that the optical axes of the two cameras are parallel and the baseline (i.e. the distance between the centers of the two cameras) is B, the calculation formula of the depth Z of the target point is:

[0251]

[0252] Wherein,

[0253] f is the focal length of the camera.

[0254] B is the baseline length of the camera, which is calculated by the Euclidean norm of the translation vector T: B = ‖T‖.

[0255] d is the parallax, i.e. the horizontal displacement difference of the corresponding points in the left and right images, defined as d = x L -x R , wherein x L and xR The horizontal coordinates of the same scene point in the left and right images, respectively.

[0256] After the above theoretical analysis and simplified derivation, the transformation matrix of the target point from the pixel coordinate system to the world coordinate system is finally obtained:

[0257]

[0258] The translation matrix t is 0 (the origins of the camera coordinate system and the world coordinate system coincide).

[0259] Therefore, the pixel coordinates of the target point can be converted into three-dimensional world coordinates.

[0260] Step 7, solve the real displacement and instantaneous speed of each target point between adjacent frames.

[0261] In the optical flow calculation, the displacement of the target point refers to the movement of a specific point in two consecutive frames of images. For each target point, the displacement can be calculated by the following formula:

[0262] Δx = x t - x t-1

[0263] Δy = y t - y t-1

[0264] Where (x t , y t ) is the three-dimensional world coordinate corresponding to the target point in the current frame, (x t-1 , y t-1 ) is the three-dimensional world coordinate of the target point in the previous frame, and Δx and Δy represent the displacement in the x and y directions, respectively.

[0265] The Euclidean distance (displacement distance) of the target point is:

[0266]

[0267] Where d represents the displacement distance of the target point.

[0268] When the frame rate is high, the average speed obtained within a certain number of adjacent frames can be approximated as the real-time speed:

[0269]

[0270] Where v is the real-time speed of the target point, d is the displacement distance of the target point, and Δt is the time difference between two frames. The invention uses a 1080P60fps binocular camera, and the time interval between adjacent frames is about 0.01 seconds.

[0271] Step 8, weighted solution of the actual displacement and real-time speed of the whole vehicle.

[0272] According to the idea of step 7, the real-time speed v of each target point is calculated i and the displacement distance d i . Let Δx i and Δy i represent the displacement of the i-th effective target point in the x and y directions; w i represent the normalized weight of the i-th effective target point, Δx weighted , Δy weighted represent the weighted displacement of the whole vehicle in the x and y directions, and N represents the total number of effective target points.

[0273]

[0274] The real-time speed of the whole vehicle is calculated by weighting:

[0275]

[0276] The displacement of the whole vehicle in the x and y directions is calculated by weighting:

[0277]

[0278] The actual displacement of the whole vehicle is calculated:

[0279]

[0280] Step 9, subway station judgment.

[0281] The actual distance between each station of the subway line is obtained from the subway company.

[0282] When the calculated real-time speed of the subway is 0 and the cumulative distance between stations is the distance between two stations, the system determines that the subway has arrived at the station. The station information is updated using the station transponder; at the same time, the cumulative distance between stations is cleared (error is cleared). When the subway starts, the speed starts to change, and the subway positioning automatically enters the next interval.

[0283] The embodiment is described in more detail.

[0284] Reason for using LK algorithm for feature point tracking

[0285] Optical flow is a kind of apparent motion of image brightness pattern. Let the illumination of the image be I(x, y, t), and assume that the illumination of the adjacent two images remains unchanged, after time dt, we have:

[0286] I(x + uδt, y + vδt, t + δt) = I(x, y, t)

[0287] Where u, v represent the optical flow vector.

[0288] Lucas-Kanade (L-K) algorithm is a method for sparse optical flow. Based on the following assumptions:

[0289] (1) Local motion consistency assumption: The algorithm assumes that the motion in the image is relatively consistent within a small neighborhood. That is, the neighboring pixels in the image change similarly over time.

[0290] (2) Optical flow continuity assumption: The algorithm assumes that the optical flow is continuous near a pixel point. That is, the motion of neighboring pixel points is continuous relative to the motion of neighboring pixel points.

[0291] (3) Brightness constancy assumption: The algorithm assumes that the surface of the object in the image maintains the same brightness over a short period of time. That is, within a small neighborhood, the brightness of the image changes little over time.

[0292] When the subway runs in the underground tunnel, the image features around the tunnel are relatively continuous, the subway is illuminated by the car lights, and the running speed is relatively stable in stages. The tunnel wall and subway line have the characteristics of brightness stability, optical flow continuity, and relative local motion consistency with the subway, satisfying the above three assumptions.

[0293] When the subway runs on the ground line, the steel rail also has the characteristics of optical flow continuity, brightness stability, and relative local motion consistency with the subway, also satisfying the above assumptions.

[0294] Assuming that the motion vector remains constant in a small spatial neighborhood Ω, the weighted least squares method is used to estimate the optical flow.

[0295] v=(A T W 2 A) -1 A T W 2 b

[0296] where n points at time t

[0297]

[0298] W=diag[W(x1),W(x2),…,W(x n )],

[0299]

[0300] W(x) represents the window weight function. The L-K algorithm based on the image pyramid starts from the highest layer of the pyramid to calculate the optical flow, using the obtained motion estimation result as the starting point for the next layer of the pyramid, until the bottom layer, as shown in Figure 1 and Figure 2 .

[0301] Example:

[0302] Video acquisition and algorithm testing

[0303] After permission from the subway staff, the calibrated camera was used to shoot the subway line 7 (unmanned) subway driving video on January 27, 2024, and the video parameters were 1080P, 60 frames / s. The actual distance between each station of subway line 7 was obtained through the subway company. Since only unmanned lines can shoot subway driving videos after permission from subway staff, but the entire line 7 is an underground line, online public videos are used to test and verify ground lines.

[0304] Test system design

[0305] The test system interface is shown in Figure 8 , which draws a reference line for precision enhancement on the input image, and the right side is divided into four columns, which respectively show the real-time tracking effect of the ROI area optical flow; mileage and speed data; real-time location of the subway on the route map, and the selected running mode.

[0306] When the visual recognition detects changes in ambient brightness and confirms that it has entered the above-ground line part by comparing the subway location with the subway line, the system switches to the above-ground mode, and the ROI area is switched to the subway track area.

[0307] Testing under different scene conditions:

[0308] The characteristics of the tunnel line are relatively stable, while the ground line scene is more complex. The work carried out tests for scenes with low visibility such as night and rain and fog on the ground line, and the test results are shown in Figure 8 .

[0309] The test results show that since the ROI area selects the steel rail area within 2m-4m in front of the car, it is not easily disturbed by surrounding environment lights, meeting lights, etc., and the test effect is good under rain and fog weather conditions.

[0310] Test results and errors:

[0311] Since only the station-to-station distance between each station on the line can be obtained, the cumulative error of the station-to-station distance is used to verify the positioning effect of the system. The positioning error between multiple stations on the left line of the north section (section opening) of Nanjing subway line 7 is within 0.1%, the average time consumption of the system is 5ms, the real-time performance is strong, and the demand for real-time positioning and speed measurement of the subway is realized more accurately.

[0312] Table 1 Station-to-station error analysis

[0313] .

Claims

1. A metro lightweight video positioning method based on a sparse optical flow algorithm, characterized in that, The method comprises the following steps: Step 1: collecting images in front of the subway during driving, and performing ROI region segmentation on the images; Step 2: performing gray scale processing on the ROI images; Step 3: performing corner point recognition on the ROI images after the gray scale processing; Step 4: performing optical flow tracking on the recognized corner points by using an L-K optical flow recognition algorithm based on an image pyramid, estimating the new position of the corner points in the current frame according to the pixel intensity change in the local window, and the specific method is as follows: Constructing the image pyramid, the current frame image I t and the next frame image I t+1 Input the image pyramid, in each layer of the image pyramid, according to the pixel intensity variation in the local window, estimate the new position of the corner point in the current frame, the specific process is: The estimated optical flow (u, v) of each layer of the image pyramid is calculated by the least square method from the top layer of the image pyramid to the bottom layer step by step. i i )​ On each layer of the image, the estimated optical flow is used as the initial optical flow, and the initial optical flow is locally refined and synthesized until the bottom layer to obtain the final optical flow estimation; Step 5: screening the target points, filtering the noise, screening the effective displacement trajectories of the target points, and assigning confidence values to the effective target points, and the specific method is as follows: According to the tracking result status and the optical flow calculation error err, the target points are preliminarily screened, and the points with large tracking errors in the corner point sequence and the corner points with failed tracking are removed; The motion slope and distance of each corner point in adjacent frames are calculated, if the slope is greater than 8 and the moving distance of adjacent frames is greater than 5 pixels, it is considered that the motion of the corner point is effective motion, and is not random noise caused by the shaking of the subway itself, the motion trajectory is drawn on the mask image, and the corner point is marked on the current frame image; If the tracking of consecutive multiple frames is all effective motion, the tracking success times are normalized as weights, which are used as the confidence of the pixel displacement of the effective target points; Let the tracking success times of the i-th target point be T i , and the sum of the tracking success times of all the effective target points be T t otal, then the weight w i of the i-th target point is: In the formula, T i is the tracking success number of the i-th target point, T total is the sum of tracking success numbers of all valid target points Step 6: converting the pixel coordinate system into a three-dimensional world coordinate system to obtain the three-dimensional coordinates of the target points; Step 7: solving the real displacement and instantaneous speed of each target point between adjacent frames; Step 8: weightedly solving the actual displacement and real-time speed of the whole vehicle, and the specific method is as follows: Let Δx i and Δy i denote the displacement of the i-th valid target point in the x and y directions; w i denote the normalized weight of the i-th valid target point, Δx weighted , Δy weighted denote the weighted displacement of the whole vehicle in the x and y directions, and N denotes the total number of valid target points. The vector displacement d i and the instantaneous velocity v i of the i th valid target point are calculated according to the displacement (Δx i , Δy i ) of the i th valid target point in the x and y directions: Weightedly solving the real-time speed of the whole vehicle: In the formula, w i is the weight of the ith target point, and N is the total number of target points. Weightedly solving the displacement of the whole vehicle in the x and y directions: Solving the actual displacement of the whole vehicle: Step 9: obtaining the actual distance between each station on the subway line, and judging whether the subway arrives at a station according to the driving distance between the subway stations and the real-time speed.

2. The metro lightweight video positioning method based on sparse optical flow algorithm according to claim 1, characterized in that, The specific method of performing corner point recognition on the ROI images after the gray scale processing is as follows: Calculating the gradient of the ROI image after the gray scale processing in the x and y directions: Wherein, I represents the pixel intensity of the image; For each pixel point in the image, a self-correlation matrix M is constructed, which represents the quadratic sum of the gradients in a window: Two eigenvalues λ1 and λ2 of the matrix M are calculated; The self-correlation matrix M is converted into a 2x2 symmetric matrix expression, which is as follows: wherein: B = ∑ I x I y , The eigenvalues are obtained by solving the characteristic equation det(M-λI)=0, and the characteristic equation is as follows: The determinant is calculated to obtain: (A - λ)(C - λ) - B 2 = 0 Expand and simplify to obtain a quadratic equation in one unknown: λ 2 -(A+C)λ+(AC-B 2 )=0 The eigenvalues are solved by using the root formula: The eigenvalues λ1 and λ2 are as follows: The smaller eigenvalue is used as the corner point response function: R=min(λ1,λ2) When the smaller eigenvalue is greater than a set threshold, the point is considered to be a corner point.

3. The metro lightweight video positioning method based on sparse optical flow algorithm according to claim 1, characterized in that, The estimated optical flow (u i ,v i ) of each layer of the image pyramid is calculated by the least square method from the top layer of the image pyramid to the lower layer step by step, and the specific formula is: Wherein A, B, C are the coefficients of image gradient calculation, and D, E are intensity change items, which are as follows: I x and I y are the gradients of the image in the x and y directions, I t is the temporal gradient, and w denotes the window region in the optical flow computation.

4. The metro lightweight video positioning method based on sparse optical flow algorithm according to claim 1, characterized in that, The specific method of using the estimated optical flow as the initial optical flow, and locally refining and synthesizing the initial optical flow until the bottom layer to obtain the final optical flow estimation is as follows: The optical flow estimation (u i ,v i ) calculated for the i-th layer is up-sampled and used as the initial value for the iterative solution of the optical flow calculation for the (i-1)-th layer Solve the following equations using least squares method on image gradient and time gradient in i-1 layer: I x Δu+I y Δv+I t =0 where I x and I y are the gradients of the image in the x and y directions, I t is the temporal gradient, Obtain the optical flow increment from i layer to i-1 layer: where I x and I y are the gradients of the image in the x and y directions, I t is the temporal gradient.

5. The sparse optical flow algorithm based metro lightweight video positioning method according to claim 1, characterized in that, The specific method for solving the real displacement between adjacent frames and the instantaneous speed of each target point is: For each target point, the displacement is calculated by the following formula: Δx = x t - x t-1 Δy = y t - y t-1 where (x t ,y t ) is the three-dimensional world coordinate corresponding to the target point in the current frame, (x t-1 ,y t-1 ) is the three-dimensional world coordinate of the target point in the previous frame, and Δx and Δy represent the displacement in the x and y directions, respectively. The Euclidean distance of the target point is: Where d represents the displacement distance of the target point; The average speed obtained within adjacent frames is taken as the real-time speed: Where v is the real-time speed of the target point, d is the displacement distance of the target point, and Δt is the time difference between two frames.

Images