Subway lightweight video positioning method based on sparse optical flow algorithm

Abstract

The invention provides a subway lightweight video positioning method based on a sparse optical flow algorithm, and the method comprises the steps: collecting a front image in a subway driving process, and carrying out the ROI region segmentation of the image; carrying out gray processing on the ROI image and carrying out angular point identification; performing optical flow tracking on the identified angular point by using an L-K optical flow identification algorithm based on an image pyramid, and estimating a new position of the angular point in the current frame according to the pixel intensity change in the local window; the target points are screened, noise is filtered, effective displacement tracks of the target points are screened, and confidence coefficient assignment is carried out on the effective target points; converting the pixel coordinate system into a three-dimensional world coordinate system to obtain a three-dimensional coordinate of the target point; solving the real displacement and instantaneous speed of each target point between adjacent frames; weighting and solving the actual displacement and the real-time speed of the whole vehicle; the actual distance between the stations of the subway line is obtained, and whether the subway arrives at the station is judged according to the running distance and the real-time speed between the subway stations. According to the method, metro positioning is achieved only through the vehicle head camera and the vehicle-mounted computer, a large amount of trackside equipment is reduced, light weight of metro positioning is achieved, and the equipment maintenance cost is reduced.

Description

Technical Field

[0001] The present invention belongs to computer vision and subway positioning technology, and specifically is a subway lightweight video positioning method based on a sparse optical flow algorithm. Background Art

[0002] Accurate and real-time subway location information is the key to ensuring the safe operation of the subway in the subway operation control system. At present, the commonly used subway positioning method is speed measurement positioning, which determines the running direction and distance of the subway through speed sensors, and then calculates the actual position of the subway on the line. However, due to reasons such as slippage of subway wheels, the speed measurement positioning error is large. In addition, track circuits, transponders, GPS, WLAN and LTE can also be used for subway positioning, but these positioning technologies have certain disadvantages, such as low positioning accuracy, great influence by the external environment, high investment cost and difficulty in later maintenance. At present, there are similar subway positioning systems based on visible light communication and binocular vision in China, but special LED lights need to be installed on the entire line. The overall positioning system is relatively complex and requires more trackside equipment for auxiliary positioning. The maintenance workload is large and the failure rate is high. Summary of the invention

[0003] The present invention proposes a subway lightweight video positioning method based on a sparse optical flow algorithm.

[0004] The technical solution to achieve the purpose of the present invention is: a subway lightweight video positioning method based on a sparse optical flow algorithm, comprising:

[0005] Step 1: Collect the front image of the subway during its running and perform ROI region segmentation on the image;

[0006] Step 2, grayscale processing of the ROI image;

[0007] Step 3, identifying corner points of the ROI image after grayscale processing;

[0008] Step 4: Use the LK optical flow recognition algorithm based on the image pyramid to track the identified corner points, and estimate the new position of the corner points in the current frame according to the change of pixel intensity in the local window;

[0009] Step 5: Screen the target points, filter the noise, screen the effective displacement trajectories of the target points, and assign confidence values ​​to the effective target points;

[0010] Step 6: Convert the pixel coordinate system into a three-dimensional world coordinate system to obtain the three-dimensional coordinates of the target point;

[0011] Step 7: Calculate the real displacement and instantaneous velocity of each target point between adjacent frames;

[0012] Step 8: Calculate the actual displacement and real-time speed of the vehicle by weighted calculation;

[0013] Step 9: Get the actual distance between each station on the subway line, and determine whether the subway has arrived at the station based on the travel distance and real-time speed between subway stations.

[0014] Preferably, the specific method for identifying corner points of the ROI image after grayscale processing is:

[0015] Calculate the gradient of the grayscale processed ROI image in the x and y directions:

[0016]

[0017] Where I represents the pixel intensity of the image;

[0018] For each pixel in the image, construct an autocorrelation matrix M, which represents the quadratic sum of the gradient within a window:

[0019]

[0020] Calculate the two eigenvalues ​​λ1 and λ2 of the matrix M;

[0021] The autocorrelation matrix M is converted into a 2×2 symmetric matrix expression in the following form:

[0022]

[0023] in: B=ΣI x I y ,

[0024] The eigenvalue is obtained by solving the characteristic equation det(M-λI)=0. The characteristic equation is:

[0025]

[0026] Calculate the determinant and get:

[0027] (A-λ)(C-λ)-B 2 =0

[0028] Expand and simplify to get a quadratic equation:

[0029] λ 2 -(A+C)λ+(AC-B 2 )=0

[0030] Use the root-finding formula to solve for the eigenvalues:

[0031]

[0032] The eigenvalues ​​λ1 and λ2 are:

[0033]

[0034]

[0035] Use smaller eigenvalues ​​as corner response functions:

[0036] R=min(λ1,λ2)

[0037] When the smaller eigenvalue is greater than the set threshold, the point is considered to be a corner point.

[0038] Preferably, the LK optical flow recognition algorithm based on the image pyramid is used to perform optical flow tracking on the identified corner points, and the new position of the corner points in the current frame is estimated according to the change of pixel intensity in the local window. The specific method is as follows:

[0039] Construct an image pyramid and transform the current frame image I t . and the next frame image I t+1 Input the image pyramid. In each layer of the image pyramid, the new position of the corner point in the current frame is estimated according to the change of pixel intensity in the local window. The specific process is as follows:

[0040] Starting from the top layer of the image pyramid, the estimated optical flow (u i ,v i );

[0041] On each layer of the image, the estimated optical flow is used as the initialized optical flow, and the initialized optical flow is locally refined and synthesized until the bottom layer to obtain the final optical flow estimate.

[0042] Preferably, the estimated optical flow (u ) of each layer of the image pyramid is calculated step by step from the top layer of the image pyramid downwards by the least squares method. i ,v i ), the specific formula is:

[0043]

[0044] Among them, A, B, C are the coefficients of image gradient calculation, and D and E are intensity change terms, specifically:

[0045]

[0046] I x and I y is the gradient of the image in the x and y directions, I t is the temporal gradient, and w represents the window area in the optical flow calculation.

[0047] Preferably, on each layer of the image, the estimated optical flow is used as the initialization optical flow, and the initialization optical flow is locally refined and synthesized until the bottom layer, so as to obtain the final optical flow estimation. The specific method is:

[0048] The optical flow estimate (u i ,v i ) is upsampled and used as the initial value for the iterative solution of the optical flow calculation of the i-1th layer

[0049] At the i-1th layer, the least squares method is used to solve the following equations for the image gradient and the temporal gradient:

[0050] I x Δu+I y Δv+I t =0

[0051] Among them I x and I y is the gradient of the image in the x and y directions, I t is the time gradient,

[0052] Get the optical flow increment from layer i to layer i-1:

[0053]

[0054] In the formula, I x and I y is the gradient of the image in the x and y directions, I t is the time gradient.

[0055] Preferably, the specific method of screening the target points, filtering the noise, screening the effective displacement trajectory of the target points, and assigning confidence values ​​to the effective target points is:

[0056] According to the tracking result status and the optical flow calculation error err, the target points are initially screened to remove the points with large tracking errors and the corner points with failed tracking in the corner point sequence.

[0057] Calculate the motion slope and distance of each corner point in adjacent frames. If the slope is greater than 8 and the moving distance between adjacent frames is greater than 5 pixels, the motion of the corner point is considered to be valid, not random noise caused by the shaking of the subway itself. Draw the motion trajectory on the mask image and mark the corner point on the current frame image.

[0058] If multiple consecutive frame trackings are valid movements, the number of successful tracking is normalized into a weight as the confidence level of the valid target point pixel displacement.

[0059] Preferably, let the number of successful tracking of the i-th target point be T i , the sum of the number of successful tracking of all valid target points is Tt otal, then the weight w of the i-th target point i for:

[0060]

[0061] Where, T i is the number of successful tracking of the i-th target point, T total It is the sum of the number of successful tracking of all valid target points.

[0062] Preferably, the specific method for solving the real displacement and instantaneous velocity of each target point between adjacent frames is:

[0063] For each target point, the displacement is calculated using the following formula:

[0064] Δx=x t -x t-1

[0065] Δy=y t -y t-1

[0066] Among them, (x t ,y t ) is the 3D world coordinate of the target point in the current frame, (x t-1 ,y t-1 ) is the 3D world coordinate of the target point in the previous frame, Δx and Δy represent the displacement in the x and y directions, respectively;

[0067] The Euclidean distance of the target point is:

[0068]

[0069] Where d represents the displacement distance of the target point;

[0070] The average speed obtained within several consecutive frames is taken as the real-time speed:

[0071]

[0072] Among them, v is the real-time velocity of the target point, d is the displacement distance of the target point, and Δt is the time difference between two frames.

[0073] Preferably, the specific method for weighted solution of the actual displacement and real-time speed of the whole vehicle is:

[0074] Let Δx i and Δy i represents the displacement of the i-th valid target point in the x and y directions; w i represents the normalized weight of the i-th valid target point, Δx weighted , Δy weightedrepresents the weighted displacement of the vehicle in the x and y directions, N represents the total number of valid target points,

[0075] According to the displacement of the i-th valid target point in the x and y directions (Δx i , Δy i ) Calculate the vector displacement d of the i-th valid target point i and instantaneous velocity v i :

[0076]

[0077] Weighted solution for the real-time speed of the vehicle:

[0078]

[0079] In the formula, w i is the weight of the i-th target point, and N is the total number of target points;

[0080] Weighted solution for the vehicle's displacement in the x and y directions:

[0081]

[0082] Calculate the actual displacement of the vehicle:

[0083]

[0084] Compared with the prior art, the present invention has the following significant advantages:

[0085] (1) Positioning method based on optical flow tracking: Taking advantage of the continuous and stable image features in the subway operation environment and the relatively stable subway speed, optical flow tracking technology is used to achieve high-precision positioning. Currently, this method is less used in subway positioning research.

[0086] (2) Confidence assessment of optical flow tracking: Image noise is filtered through a filtering algorithm, and the confidence of feature points is assessed using factors such as the slope of the optical flow trajectory and the stability of successful tracking in multiple consecutive frames. Weights are assigned to the tracking results to improve the accuracy and reliability of positioning.

[0087] (3) The subway positioning method is less affected by the subway driving environment and adopts stable and continuous visual features for tracking and positioning. It can effectively locate in tunnel lines and ground lines. At the same time, it can also effectively identify and locate in environments with low visibility such as rainy and snowy weather and heavy fog at night, and has good robustness.

[0088] (4) The present invention realizes subway positioning by only using the front camera and the on-board computer, which reduces a large amount of trackside equipment, realizes lightweight subway positioning, and reduces equipment maintenance costs.

[0089] The present invention is described in further detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 This is a schematic diagram of ROI area selection and recognition effect, where: Figure 1 (a) is a schematic diagram of ROI area selection in the tunnel. Figure 1 (b) is a schematic diagram of the optical flow recognition effect of the ROI area in the tunnel. Figure 1 (c) is a schematic diagram of ground line ROI area selection. Figure 1 (d) is a schematic diagram of the optical flow recognition effect of the ground line ROI area.

[0091] Figure 2 Schematic diagram of the effect of filtering noise and selecting effective target trajectories in the image. Figure 2 (a) is the noise from subway movement. Figure 2 (b) is the stationary noise of the subway station. Figure 2 (c) in the figure is a schematic diagram of the effective target point trajectory.

[0092] Figure 3 It is a flow chart of the present invention.

[0093] Figure 4 is the structure of the image pyramid.

[0094] Figure 5 This is the flow chart of the LK algorithm.

[0095] Figure 6 Screenshot of the video inside the tunnel.

[0096] Figure 7 To test the system interface.

[0097] Figure 8 To test the effect in low visibility environment. DETAILED DESCRIPTION

[0098] A subway lightweight video positioning method based on sparse optical flow algorithm, the specific steps are as follows:

[0099] Step 1: The camera installed on the front of the subway car transmits the front image taken in real time during the subway travel, and performs ROI region segmentation on the image to be processed.

[0100] The subway driving environment can be divided into two types: ground lines and tunnel lines. Different ROI segmentation methods are adopted for these two line environments:

[0101] For ground lines, since the ambient light of ground lines varies greatly, the fixed ROI area is the rail area and no dynamic adjustment is performed;

[0102] For tunnel lines, the tunnel walls on both sides of the subway running environment are selected as the ROI area.

[0103] After selecting the ROI area, it is also necessary to perform perspective transformation on the ROI area to make it more conducive to subsequent corner point identification and optical flow tracking. Define the four corner points of the region of interest, and then calculate the perspective transformation matrix by defining the positions (four vertices) of these four corner points in the ROI image (size: 200x500), and use the warpPerspective function in the OpenCV library to perform perspective transformation.

[0104] The perspective transformation maps points in one image to another image through a 3x3 homography matrix H.

[0105]

[0106] Given a point (x,y) in the source image and a point (x′,y′) in the target image,

[0107]

[0108] where w' is the normalization factor for homogeneous coordinates.

[0109] The homogeneous coordinates are converted to actual image coordinates by normalization. The normalized coordinates are calculated as:

[0110]

[0111] See the attached picture for the effect after ROI selection and perspective transformation. Figure 1 .

[0112] Step 2: grayscale the ROI image to facilitate subsequent corner point detection.

[0113] Step 3: Perform corner recognition on the grayscale processed ROI image to extract feature points for optical flow recognition.

[0114] The Shi-Tomasi corner detection algorithm is an improved algorithm of the Harris corner detection algorithm. It simplifies the calculation of the response function and only uses smaller eigenvalues ​​to determine corners.

[0115] First, calculate the gradient of the grayscale processed ROI image in the x and y directions:

[0116]

[0117] Where I represents the pixel intensity of the image.

[0118] For each pixel in the image, construct an autocorrelation matrix M, which represents the quadratic sum of the gradient within a window:

[0119]

[0120] Then, the two eigenvalues ​​λ1 and λ2 of the matrix M are calculated.

[0121] Since the autocorrelation matrix M is a 2×2 symmetric matrix, it is in the following form:

[0122]

[0123] in: B=∑I x I y ,

[0124] The eigenvalue is obtained by solving the characteristic equation det(M-λI) = 0. The characteristic equation is:

[0125]

[0126] Calculate the determinant and get:

[0127] (A-λ)(C-λ)-B 2 =0

[0128] Expand and simplify to get a quadratic equation:

[0129] λ 2 -(A+C)λ+(AC-B 2 )=0

[0130] Use the root-finding formula to solve for the eigenvalue:

[0131]

[0132] Therefore, the eigenvalues ​​λ1 and λ2 are:

[0133]

[0134] Finally, use the smaller eigenvalue as the corner response function:

[0135] R=min(λ1,λ2)

[0136] When the smaller eigenvalue λ is greater than a certain threshold, the point is considered to be a corner point.

[0137] In this method, the goodFeaturesToTrack function in the OpenCV library is used, the Shi-Tomasi algorithm is used to input the grayscale image, and the detected corner point vector is output for subsequent optical flow recognition.

[0138] Among them, the parameters considered are:

[0139] Set the maximum number of detected corner points to 100 and the quality level to 0.25, which indicates the minimum quality level that the detected corner points must reach. If the feature value of a point is lower than 0.25, the point will not be detected as a corner point. Set the minimum Euclidean distance between detected corner points to 7 pixels.

[0140] In order to ensure the stability and real-time performance of corner point detection, the corner points are re-detected every 10 frames or when the corner point vector p0 is empty.

[0141] Step 4: Use the LK optical flow recognition algorithm based on the image pyramid to track the identified corner points. According to the change of pixel intensity in the local window, estimate the new position of the corner point in the current frame, including:

[0142] (1) is the current frame image I t . and the next frame image I t+1 Build an image pyramid. The image pyramid is generated by downsampling the image layer by layer.

[0143] Pyramid level settings:

[0144] (2) Set the number of layers of the image pyramid L = 3 and the scaling factor of each layer (default is 0.5). The bottom layer of the pyramid is the original image, and the other layers are images with gradually lower resolutions, with the top layer having the lowest resolution.

[0145] (3) Optical flow calculation:

[0146] In each layer of the image pyramid, the new position of the corner point in the current frame is estimated based on the change in pixel intensity within the local window. The Lucas-Kanade method is used to calculate the optical flow. Let the optical flow of the current layer be (u i ,v i ).

[0147] Assume that the pixel intensity change in the local window is:

[0148] I x Δu+I y Δv+I t =0

[0149] Among them I x and I y is the gradient of the image in the x and y directions, I t is the temporal gradient, and (Δu, Δv) is the optical flow increment.

[0150] (4) Least Squares Optimization

[0151] The estimated optical flow (u i ,v i), minimize the difference in pixel intensity, and obtain the optimal optical flow vector in the image window of this layer, so that the difference in pixel intensity of the target point between the previous and next frames is minimized.

[0152] Calculate the least squares solution of the optical flow in the i-th layer image (u i ,v i )The specific process is as follows:

[0153]

[0154] Among them, A, B, C are the coefficients of image gradient calculation, and D and E are intensity change terms. Specifically:

[0155]

[0156] The summation symbol indicates the summation within the local window.

[0157] (5) Pyramid level processing:

[0158] Starting from the top layer of the image pyramid, and then processing downwards, the estimated optical flow (u i ,v i ), on each layer of the image, use the initial result of the optical flow estimation (the upsampling result from the low-resolution layer) as initialization, and perform local refinement and synthesis until the bottom layer to obtain the final optical flow estimation:

[0159] Assume that the optical flow estimate calculated by the least squares method at the i-th layer (lower resolution layer) is (u i ,v i ), which is upsampled and used as the initial value for the iterative solution of the optical flow calculation of the i-1th layer (higher resolution layer) The purpose of upsampling is to make it Match the image resolution of the i-1th layer. For example, when the scaling factor is 0.5, the image resolution increases by 1 time from the i-th layer to the i-1th layer, and the optical flow vector also needs to be enlarged accordingly, resulting in

[0160] The scaling factor is 0.5, and the upsampling formula is:

[0161]

[0162] In the formula, u i-1 (x,y) 0 is the optical flow vector u after upsampling at the i-1th layer; v i-1 (x,y) 0 is the optical flow vector v after upsampling at the i-1th layer; is the optical flow vector u corresponding to the i-th layer; is the optical flow vector v corresponding to the i-th layer.

[0163] At the i-1th layer, the following equations are still solved for the image gradient and time gradient:

[0164] I x Δu+I y Δv+I t =0

[0165] Among them I x and I y is the gradient of the image in the x and y directions, I t is the time gradient.

[0166] Use the least squares method to calculate the optical flow increment from layer i to layer i-1:

[0167]

[0168] The initial optical flow estimate Added to the calculated increment, the final optical flow estimate of the i-1th layer is obtained

[0169]

[0170] It will be passed as input to the next layer (i-2 layer) for higher resolution optical flow calculation. Through layer-by-layer processing and synthesis, an accurate optical flow estimate is finally obtained on the highest resolution image.

[0171] (5) Output the new position estimation result of each corner point in the current frame.

[0172] In the specific implementation, OpenCV library functions are used for optical flow tracking. The specific steps are as follows:

[0173] Initialize optical flow tracking and define two containers (vector) of Point2f type to store the tracking status and tracking error of each corner point. Then use the function calcOpticalFlowPyrLK(old_gray,frame_gray,p0,p1,status,err,Size(15,15),3,criteria) in the OpenCV library to perform optical flow tracking on the corner points. Input the grayscale image old_gray of the previous frame ROI and the grayscale image frame_gray of the current frame ROI, as well as the corner point coordinates p0 in the previous frame; output the corner point coordinates p1 tracked by the optical flow in the current frame, the tracking result status (whether the tracking is successful), and the optical flow calculation error err.

[0174] Here, the parameters that need to be considered are "window size for calculating optical flow" (Size(15,15)), "number of pyramid layers" (3), and "stop condition for optical flow calculation" (criteria):

[0175] An image pyramid is a multi-scale representation of an image, consisting of images of different resolutions. The bottom layer (highest resolution) of the pyramid is the original image, while the upper layer is a gradually downsampled image, with the resolution of each layer decreasing layer by layer. In each layer of the pyramid, the actual size of the window is the same (for example, 15x15 pixels), but due to the different resolutions of each layer, the actual area covered by the window will also change with the level, so that the advantages of large windows and small windows can be taken into account at the same time, ensuring that the consistency of the window is maintained while calculating the optical flow at different scales, so that the optical flow calculation can effectively process motion information from local to global. The stopping conditions for the optical flow calculation include specifying the maximum number of iterations and the error threshold. Generally, the number of pyramid layers is set to 3 and the window size is Size(15,15).

[0176] Use the TermCriteria class in the OpenCV library to set the stopping condition of the iterative algorithm. Its constructor includes three parameters:

[0177] TermCriteria::COUNT: The specified maximum number of iterations was reached.

[0178] TermCriteria::EPS: Achieve the specified precision error.

[0179] TermCriteria::COUNT+TermCriteria::EPS: Both are satisfied.

[0180] Taking into account the algorithm's time consumption and error requirements, the algorithm sets the maximum number of iterations to 10 and the error threshold to 0.03.

[0181] Step 5: Filter the target points, filter the noise, filter the effective displacement trajectories of the target points, and assign confidence values ​​to the effective target points (if visualization is required, you can draw them).

[0182] According to the tracking result status and the optical flow calculation error err, the target points are initially screened to remove the points with large tracking errors and the corner points with failed tracking in the corner point sequence.

[0183] Calculate the motion slope and distance of each corner point in adjacent frames. If the slope is greater than 8 and the moving distance between adjacent frames is greater than 5 pixels, the motion of the corner point is considered to be valid, not random noise caused by the shaking of the subway itself. Use the OpenCV library function line(mask,p1[i],p0[i],colors[i],2) to draw the motion trajectory on the mask image, and use the OpenCV library function circle(frame,p1[i],5,colors[i],-1) to mark the corner point on the current frame image:

[0184] The Line function is a function used in the OpenCV library to draw a straight line. Mask represents the mask image, p1[i] and p0[i] represent the starting and ending points of the i-th line. colors[i] represents the color of the i-th line. 2 represents the thickness of the line. Here, the width of the drawn line is set to 2 pixels.

[0185] The circle function is a function used to draw circles in the OpenCV library. frame represents the target image for drawing circles, p1[i] represents the center of the i-th circle drawn, 5 represents the radius of the circle is 5 pixels, colors[i] represents the color of the i-th circle drawn; -1 represents the filling method of the circle is solid.

[0186] If multiple consecutive tracking frames are valid movements, the number of successful tracking is normalized into a weight as the confidence of the pixel displacement of the valid target point, which is used for the subsequent calculation of the pixel displacement of the entire vehicle.

[0187] Assume that the number of successful tracking of the i-th target point is T i , the sum of the number of successful tracking of all valid target points is T t otal, then the weight w of the i-th target point i It can be calculated as:

[0188]

[0189] The random noise caused by the shaking of the subway itself and the effective trajectory of the target point obtained after screening and filtering are shown in the attached figure. Figure 2 shown.

[0190] Continue to pass in the next frame of the picture, use the pyramid LK optical flow recognition algorithm to achieve continuous optical flow tracking of the target point, and save the continuous frame pixel coordinates of the valid target point until all pictures are processed.

[0191] Step 6: Convert the pixel coordinate system into a three-dimensional world coordinate system to obtain the three-dimensional coordinates of the target point.

[0192] The results of optical flow tracing are limited to displacement information at the pixel level of the image, and do not provide accurate movement information of the subway in the real world. By converting pixel coordinates into coordinates in the world coordinate system, the displacement information obtained by optical flow tracing can be mapped to the real world, thereby obtaining the real movement of the subway and further obtaining the actual displacement of the subway in the three-dimensional world.

[0193] The coordinates of the camera in the world can be represented by the rotation matrix R and the translation matrix T. If a camera is placed in the world coordinate system, the translation of the camera coordinates relative to the origin of the world coordinate system can be recorded as a vector, and the rotation angle is recorded as R. Any translation includes three directions, represented by a three-dimensional vector (x, y, z); for the rotation matrix R, the rotation is performed along three different axes, x, y, and z, and the rotation matrix is:

[0194]

[0195]

[0196] Therefore, the transformation from the world coordinate system to the camera coordinate system can be written as:

[0197]

[0198] The imaging model of an ordinary camera is a pinhole imaging model. By analyzing the geometric relationship, we can know the relationship between the coordinates of the 3D target point in the camera coordinate system and the image pixel coordinates. At the same time, in order to facilitate calculation, a new dimension is introduced to convert the coordinates into homogeneous coordinates:

[0199]

[0200] (u0, v0) is the offset from the center point of the image coordinate system to the center point of the pixel coordinate system, which is part of the camera intrinsic parameter. x ,f y It is the projection of the camera focal length on the x and y axes.

[0201] Therefore, the relationship between the pixel coordinate system and the world coordinate system can be expressed as follows:

[0202]

[0203] in, is the camera internal parameter, is the camera external parameter, that is:

[0204]

[0205] This method needs to calculate the real coordinates of the target feature point in the world coordinate system based on the pixel coordinates, that is, the inverse transformation from the world coordinate system to the pixel coordinate system:

[0206]

[0207] (1) Since the present invention considers calculating the subway advance distance by the displacement of feature points between adjacent frames, there is no need to obtain the absolute coordinates of the feature points. Therefore, the camera coordinates can be set as the origin of the world coordinate system, that is, the origin of the camera coordinate system coincides with the origin of the world coordinate system.

[0208] (2) By fixing the camera at a fixed position on the front of the subway, the camera's intrinsic and extrinsic parameters can be calibrated and calculated in advance. The camera's intrinsic parameters include focal length, optical center, and distortion coefficient; the extrinsic parameters include rotation matrix and translation vector.

[0209] Use Zhang Zhengyou's method to calibrate the binocular camera:

[0210] First, take multiple checkerboard images at different viewing angles and extract the coordinates of each corner point in the image.

[0211] Then, the single-target positioning algorithm is used to solve the intrinsic parameter matrix K and distortion parameter D of the left camera and the right camera respectively.

[0212]

[0213] in:

[0214] (X,Y,Z) is a point in three-dimensional space.

[0215] (u,v) is the corresponding two-dimensional image point.

[0216] s is the scaling factor.

[0217] K is the intrinsic parameter matrix of the camera, defined as:

[0218] R is the rotation matrix and t is the translation vector.

[0219] For each image, the corner point coordinates are used to solve the homography matrix H, satisfying the following relationship:

[0220]

[0221] The relationship between the homography matrix H, the intrinsic parameter matrix K and the extrinsic parameter matrix is:

[0222] H=K[r1 r2 t]

[0223] where r1 and r2 are the first two columns of the rotation matrix R.

[0224] By optimizing the homography matrix H of multiple images, the internal parameter matrix K is solved:

[0225] v ij =[h i1h j1 h i1 h j2 +h i2 h j1 h i2 h j2 h i3 h j1 +h i1 h j3 h i3 h j2 +h i2 h j3 h i3 h j3 ]

[0226] Use the following formula to calculate: Vb = 0

[0227] Where V is v ij The matrix is ​​composed of , and b is a vector containing the elements of the internal parameter matrix. Solve b through SVD decomposition and further solve K.

[0228] After solving the intrinsic parameter matrix K, the homography matrix H is used to calculate the extrinsic parameter matrix of each image:

[0229] r1=K -1 h1

[0230] r2=K -1 h2

[0231] t=K -1 h3

[0232] Orthogonalize r1 and r2, and calculate r3: r3 = r1 × r2

[0233] Based on single-target calibration, the relative rotation matrix R and translation vector T between the two cameras are solved by the stereo calibration algorithm:

[0234]

[0235] Finally, stereo correction is performed. The image planes of the two cameras are aligned to simplify the epipolar geometry. Stereo correction calculates new rotation matrices R1 and R2 to project the images of the two cameras onto the same view plane:

[0236] Calculate new projection matrices P1 and P2 to align the images to the same view plane:

[0237] P1=K1[I|0]

[0238] P2=K2[R|T]

[0239] Among them: K1, K2 are the intrinsic parameter matrices of the left and right cameras respectively; R, T are the rotation matrix and translation vector of the camera respectively.

[0240] Then calculate the new rotation matrix:

[0241]

[0242] Where R rect is a rotation matrix used to align the view, R 1′ and R 2′ are the original rotation matrices respectively.

[0243] Using the calculated rotation and projection matrices, transform the images so that rows of the same object in both images have the same vertical coordinates.

[0244] x 1′ =R1x1

[0245] x 2′ =R2x2

[0246] Among them, x1, x2 are the coordinates of the points in the left and right images respectively, and x 1′ ,x 2′ are the coordinates of the points after transformation.

[0247] At this point, the calibration of the binocular camera is completed.

[0248] (3) Since depth information is lost during the conversion from world coordinates to the camera image, it is usually necessary to obtain additional depth information to perform an inverse transformation from 2D to 3D coordinate system.

[0249] Using a binocular camera, you can get depth information:

[0250] The binocular camera system consists of two cameras, the left camera and the right camera. Through calibration, their relative positions and postures are known. Assuming that the optical axes of the two cameras are parallel and the baseline (i.e. the distance between the centers of the two cameras) is B, the calculation formula for the depth Z of the target point is:

[0251]

[0252] in,

[0253] f is the focal length of the camera.

[0254] B is the baseline length of the camera, which is calculated by the Euclidean norm of the translation vector T: B = ‖T‖.

[0255] d is the disparity, that is, the lateral displacement difference of the corresponding points in the left and right images, defined as d = x L -x R , where x L and xR are the horizontal coordinates of the same scene point in the left and right images respectively.

[0256] After the above theoretical analysis and simplified derivation, the transformation matrix of the target point from the pixel coordinate system to the world coordinate system is finally obtained:

[0257]

[0258] The translation matrix t is 0 (the origin of the camera coordinate system and the world coordinate system coincides).

[0259] Thus, the pixel coordinates of the target point can be converted into three-dimensional world coordinates.

[0260] Step 7: Calculate the true displacement and instantaneous velocity of each target point between adjacent frames.

[0261] In optical flow calculation, the displacement of a target point refers to the amount of movement of a specific point in two consecutive frames of images. For each target point, the displacement can be calculated using the following formula:

[0262] Δx=x t -x t-1

[0263] Δy=y t -y t-1

[0264] Among them, (x t ,y t ) is the 3D world coordinate of the target point in the current frame, (x t-1 ,y t-1 ) is the 3D world coordinate of the target point in the previous frame, Δx and Δy represent the displacement in the x and y directions, respectively.

[0265] The Euclidean distance (displacement distance) of the target point is:

[0266]

[0267] Where d represents the displacement distance of the target point.

[0268] When the frame rate is high, the average speed obtained within several consecutive frames can be approximated to the real-time speed:

[0269]

[0270] Wherein, v is the real-time speed of the target point, d is the displacement distance of the target point, Δt is the time difference between two frames. The present invention adopts a 1080P 60fps binocular camera, and the interval time between adjacent frames is about 0.01 seconds.

[0271] Step 8: Use weighted method to solve the actual displacement and real-time speed of the vehicle.

[0272] According to the idea of ​​step 7, calculate the real-time speed v of each target point i and displacement distance d i . Let Δx i and Δy i represents the displacement of the i-th valid target point in the x and y directions; w i represents the normalized weight of the i-th valid target point, Δx weighted , Δy weighted represents the weighted displacement of the vehicle in the x and y directions, and N represents the total number of valid target points.

[0273]

[0274] Weighted solution for the real-time speed of the vehicle:

[0275]

[0276] Weighted solution for the vehicle's displacement in the x and y directions:

[0277]

[0278] Calculate the actual displacement of the vehicle:

[0279]

[0280] Step 9: Determine when the subway arrives at the station.

[0281] Get the actual distance between stations on the subway line from the subway company.

[0282] When the calculated real-time subway speed is 0 and the accumulated mileage between stations is the mileage of two stations, the system determines that the subway has arrived at the station. The station transponder information is used to update the station; at the same time, the accumulated mileage between stations is reset to zero (error reset). When the subway starts, it detects that the speed begins to change and automatically enters the subway positioning in the next section.

[0283] The embodiments are described in more detail.

[0284] Reasons for using LK algorithm for feature point tracking

[0285] Optical flow is an apparent movement of the brightness pattern of an image. Assume that the illumination of the image is I(x, y, t), and assume that the illumination of two adjacent images remains unchanged. After a time dt, we have:

[0286] I(x+uδt,y+vδt,t+δt)=I(x,y,t)

[0287] Where u, v represents the optical flow vector.

[0288] The Lucas-Kanade (LK) algorithm is a method for calculating sparse optical flow. It is based on the following basic assumptions:

[0289] (1) Local motion consistency assumption: The algorithm assumes that the motion in an image is relatively consistent within a small neighborhood. That is, adjacent pixels in the image change similarly over time.

[0290] (2) Optical flow continuity assumption: The algorithm assumes that the optical flow near a pixel is continuous. That is, the motion of adjacent pixels is continuous relative to the motion of adjacent pixels.

[0291] (3) Brightness constancy assumption: The algorithm assumes that the surface brightness of objects in the image remains constant over a short period of time. That is, within a small neighborhood, the brightness of the image changes little over time.

[0292] When the subway is running in an underground tunnel, the image features around the tunnel are relatively continuous. The subway is illuminated by headlights, the driving speed is relatively stable in stages, and the tunnel wall and subway line have the characteristics of stable brightness, continuous optical flow, and consistency relative to the local motion of the subway, which meets the above three assumptions.

[0293] When the subway runs on the ground line, the rails also have the characteristics of continuous optical flow, stable brightness, and consistency of local motion relative to the subway, which also meets the above assumptions.

[0294] Assuming that the motion vector remains constant over a small spatial neighborhood Ω, the optical flow is estimated using weighted least squares.

[0295] v=(A T W 2 A) -1 A T W 2 b

[0296] Among them, at the n points at time t

[0297]

[0298] W=diag[W(x1),W(x2),…,W(x n )],

[0299]

[0300] W(x) represents the window weight function. The LK algorithm based on the image pyramid calculates the optical flow from the highest level of the pyramid, and uses the obtained motion estimation result as the starting point of the next level of the pyramid until the bottom level. Figure 1 and Figure 2 shown.

[0301] () Example:

[0302] Video acquisition and algorithm testing

[0303] With the permission of the subway staff, a calibrated camera was used to shoot a subway driving video of Metro Line 7 (driverless) on January 27, 2024. The video parameters were 1080P, 60 frames / s. The actual distance between stations on Metro Line 7 was obtained from the subway company. Since only driverless lines can shoot subway driving videos with the permission of subway staff, but the entire Line 7 is an underground line, the public video on the Internet was used to test and verify the ground line.

[0304] Test system design

[0305] Test system interface Figure 8 As shown, a baseline for precision enhancement is drawn on the input image. The right side is divided into four columns, which are the real-time tracking effect of optical flow in the ROI area; mileage and speed data; the real-time position of the subway on the route map, and the selected operation mode.

[0306] When the system visually recognizes a change in ambient brightness and confirms that it has entered the above-ground line by comparing the subway location with the subway line, the system switches to above-ground mode and switches the ROI area to the subway track area.

[0307] Tests under different scenarios:

[0308] The characteristics of tunnel lines are relatively stable, while the scenes of ground lines are more complex. The work conducted tests on scene conditions with low visibility such as night and rain and fog on ground lines. The test results are as follows: Figure 8 shown.

[0309] The test results show that since the ROI area is selected as the rail area within 2m-4m in front of the vehicle, it is not easily disturbed by the surrounding environment lights, oncoming vehicle lights, etc., and the test effect is good under rainy and foggy weather conditions.

[0310] Test results and errors:

[0311] Since only the distance between stations on the line can be obtained, the system positioning effect is verified by the cumulative error of the distance between stations. The positioning error between multiple stations on the left line of the northern section of Nanjing Metro Line 7 (opened in sections) is within 0.1%, and the system takes an average of 5ms, with strong real-time performance, which more accurately meets the needs of real-time positioning and speed measurement of the subway.

[0312] Table 1 Inter-station error analysis

[0313] .

Claims

1. A subway lightweight video positioning method based on sparse optical flow algorithm, characterized in that: include: Step 1: Collect the front image of the subway during its running and perform ROI region segmentation on the image; Step 2, grayscale processing of the ROI image; Step 3, identifying corner points of the ROI image after grayscale processing; Step 4: Use the LK optical flow recognition algorithm based on the image pyramid to track the identified corner points, and estimate the new position of the corner points in the current frame according to the change of pixel intensity in the local window; Step 5: Screen the target points, filter the noise, screen the effective displacement trajectories of the target points, and assign confidence values ​​to the effective target points; Step 6: Convert the pixel coordinate system into a three-dimensional world coordinate system to obtain the three-dimensional coordinates of the target point; Step 7: Calculate the real displacement and instantaneous velocity of each target point between adjacent frames; Step 8: Calculate the actual displacement and real-time speed of the vehicle by weighted calculation; Step 9: Get the actual distance between each station on the subway line, and determine whether the subway has arrived at the station based on the travel distance and real-time speed between subway stations.

2. The subway lightweight video positioning method based on sparse optical flow algorithm according to claim 1 is characterized in that: The specific method for identifying corner points of the ROI image after grayscale processing is as follows: Calculate the gradient of the grayscale processed ROI image in the x and y directions: Where I represents the pixel intensity of the image; For each pixel in the image, construct an autocorrelation matrix M, which represents the quadratic sum of the gradient within a window: Calculate the two eigenvalues ​​λ1 and λ2 of the matrix M; The autocorrelation matrix M is converted into a 2×2 symmetric matrix expression in the following form: in: B=ΣI x I y , The eigenvalue is obtained by solving the characteristic equation det(M-λI)=0. The characteristic equation is: Calculate the determinant and get: (A-λ)(C-λ)-B 2 =0 Expand and simplify to get a quadratic equation: l 2 -(A+C)λ+(AC-B 2 )=0 Use the root-finding formula to solve for the eigenvalues: The eigenvalues ​​λ1 and λ2 are: Use smaller eigenvalues ​​as corner response functions: R=min(λ1,λ2) When the smaller eigenvalue is greater than the set threshold, the point is considered to be a corner point.

3. The subway lightweight video positioning method based on sparse optical flow algorithm according to claim 1 is characterized in that: Use the LK optical flow recognition algorithm based on the image pyramid to track the identified corner points. According to the change of pixel intensity in the local window, estimate the new position of the corner point in the current frame. The specific method is as follows: Construct an image pyramid and transform the current frame image I t . and the next frame image I t+1 Input the image pyramid. In each layer of the image pyramid, the new position of the corner point in the current frame is estimated according to the change of pixel intensity in the local window. The specific process is as follows: Starting from the top layer of the image pyramid, the estimated optical flow (u i ,v i ); On each layer of the image, the estimated optical flow is used as the initialized optical flow, and the initialized optical flow is locally refined and synthesized until the bottom layer to obtain the final optical flow estimate.

4. The subway lightweight video positioning method based on sparse optical flow algorithm according to claim 3 is characterized in that: Starting from the top layer of the image pyramid, the estimated optical flow (u i ,v i ), the specific formula is: Among them, A, B, C are the coefficients of image gradient calculation, and D and E are intensity change terms, specifically: I x and I y is the gradient of the image in the x and y directions, I t is the temporal gradient, and w represents the window area in the optical flow calculation.

5. The subway lightweight video positioning method based on sparse optical flow algorithm according to claim 3 is characterized in that: On each layer of the image, the estimated optical flow is used as the initialization optical flow, and the initialization optical flow is locally refined and synthesized until the bottom layer to obtain the final optical flow estimation. The specific method is: The optical flow estimate (u i ,v i ) is upsampled and used as the initial value for the iterative solution of the optical flow calculation of the i-1th layer At the i-1th layer, the least squares method is used to solve the following equations for the image gradient and the temporal gradient: I x Δu+I y Δv+I t =0 Among them I x and I y is the gradient of the image in the x and y directions, I t is the time gradient, Get the optical flow increment from layer i to layer i-1: In the formula, I x and I y is the gradient of the image in the x and y directions, I t is the time gradient.

6. The subway lightweight video positioning method based on sparse optical flow algorithm according to claim 1 is characterized in that: The specific method of screening target points, filtering noise, screening effective displacement trajectories of target points, and assigning confidence values ​​to effective target points is as follows: According to the tracking result status and the optical flow calculation error err, the target points are initially screened to remove the points with large tracking errors and the corner points with failed tracking in the corner point sequence. Calculate the motion slope and distance of each corner point in adjacent frames. If the slope is greater than 8 and the moving distance between adjacent frames is greater than 5 pixels, the motion of the corner point is considered to be valid, not random noise caused by the shaking of the subway itself. Draw the motion trajectory on the mask image and mark the corner point on the current frame image. If multiple consecutive frame trackings are valid movements, the number of successful tracking is normalized into a weight as the confidence level of the valid target point pixel displacement.

7. The subway lightweight video positioning method based on sparse optical flow algorithm according to claim 6 is characterized in that: Assume that the number of successful tracking of the i-th target point is T i , the sum of the number of successful tracking of all valid target points is T t otal, then the weight w of the i-th target point i for: Where, T i is the number of successful tracking of the i-th target point, T total It is the sum of the number of successful tracking of all valid target points.

8. The subway lightweight video positioning method based on sparse optical flow algorithm according to claim 1 is characterized in that: The specific method for solving the true displacement and instantaneous velocity of each target point between adjacent frames is: For each target point, the displacement is calculated using the following formula: Δx=x t -x t-1 Δy=y t -y t-1 Among them, (x t ,y t ) is the 3D world coordinate of the target point in the current frame, (x t-1 ,y t-1 ) is the 3D world coordinate of the target point in the previous frame, Δx and Δy represent the displacement in the x and y directions, respectively; The Euclidean distance of the target point is: Where d represents the displacement distance of the target point; The average speed obtained within several consecutive frames is taken as the real-time speed: Among them, v is the real-time velocity of the target point, d is the displacement distance of the target point, and Δt is the time difference between two frames.

9. The subway lightweight video positioning method based on sparse optical flow algorithm according to claim 1 is characterized in that: The specific method for weighted solution of the actual displacement and real-time speed of the vehicle is: Let Δx i and Δy i represents the displacement of the i-th valid target point in the x and y directions; w i represents the normalized weight of the i-th valid target point, Δx weighted , Δy weighted represents the weighted displacement of the vehicle in the x and y directions, N represents the total number of valid target points, According to the displacement of the i-th valid target point in the x and y directions (Δx i , Δy i ) Calculate the vector displacement d of the i-th valid target point i and instantaneous velocity v i : Weighted solution for the real-time speed of the vehicle: In the formula, w i is the weight of the i-th target point, and N is the total number of target points; Weighted solution for the vehicle's displacement in the x and y directions: Calculate the actual displacement of the vehicle:

Images