A method and device for visually measuring the speed of moving objects in complex outdoor scenes

Abstract

The present invention discloses a visual measurement method and device for the speed of a moving object in a complex outdoor scene, including: calibrating a binocular camera; using the Shi-Tomasi corner point method to extract feature points in an image taken by the binocular camera, and using the LK optical flow method to track the feature points of the left and right images to obtain a left image feature point set and a right image feature point set; using a pyramid LK optical flow tracking algorithm based on dynamic ROI to track the feature points of the binocular images of the previous frame and the current frame to obtain a left image feature point set and a right image feature point set of the current frame; selecting a new feature point set with successful optical flow tracking and a small optical flow error from the corresponding feature point set tracked from the previous frame to the current frame; performing binocular three-dimensional reconstruction on the new feature point set to obtain a three-dimensional point set of the previous frame and the current frame; and calculating the speed of the moving object according to the three-dimensional point set of the previous frame and the current frame. The speed measurement method of the present invention has high robustness and high efficiency.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a method and device for visually measuring the speed of a moving object in a complex outdoor scene. Background Art

[0002] The 360° train fault detection equipment cannot be installed on the main line due to the high-voltage wires above the railway main line, and can only be installed in the depot section of the maintenance station. The imaging equipment in the 360° train fault detection equipment generally uses a linear array camera with a high resolution. In order to ensure that the linear array camera samples at equal intervals in the direction of train movement, the line trigger signal frequency of the linear array camera is required to be proportional to the train speed value. Therefore, speed measurement plays a key role in the 360° train fault detection equipment installed in the depot section.

[0003] The speed of a train varies greatly when traveling in the depot section, generally fluctuating within the range of -30 to 80 km / h. There is currently no reliable speed measurement method for this complex scene of non-uniform speed, low speed (relative to the normal speed of the train), turning and outdoor conditions. There are currently two main methods for speed measurement in the railway industry: Doppler radar speed measurement and magnetic steel speed measurement. Doppler radar speed measurement will be affected by the uneven surface of the train and the gap between the carriages; magnetic steel speed measurement will have a large delay when the train is traveling slowly; both of the current speed measurement methods cannot meet the requirements of the triggering photo signal of the linear array camera and 3D profile scanner in the train's 360° fault detection equipment. Summary of the invention

[0004] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a method for visually measuring the speed of a moving object in a complex outdoor scene, so as to solve the problem of difficulty in measuring the speed of a moving object in a complex outdoor scene.

[0005] To solve the above technical problems, the present invention is implemented by adopting the following solutions:

[0006] The present invention provides a method for visually measuring the speed of a moving object in a complex outdoor scene, comprising:

[0007] Calibrate the binocular camera;

[0008] The Shi-Tomasi corner point method is used to extract the feature points in the image taken by the binocular camera, and the LK optical flow method is used to track the feature points of the left and right images to obtain the feature point set of the left image and the feature point set of the right image;

[0009] The dynamic ROI-based pyramid LK optical flow tracking algorithm is used to track the feature points of the previous and current binocular images to obtain the left and right feature point sets of the current frame.

[0010] Filter out a new feature point set with successful optical flow tracking and small optical flow error from the corresponding feature point set tracked from the previous frame to the current frame;

[0011] Perform binocular 3D reconstruction on the new feature point set to obtain the 3D point sets of the previous frame and the current frame;

[0012] Calculate the speed of a moving object based on the 3D point sets of the previous frame and the current frame.

[0013] The present invention also provides a measuring device for implementing the above-mentioned visual measurement method of the speed of a moving object in a complex outdoor scene, comprising a binocular camera, a stroboscopic light source and a PC; the binocular camera is electrically connected to the stroboscopic light source and the PC respectively; the binocular camera comprises a left camera and a right camera, and the left camera and the right camera take pictures synchronously when the stroboscopic light source is turned on; the PC comprises a CPU and a GPU, the CPU is used to execute logical operations, and the GPU is used to accelerate feature point extraction and optical flow tracking algorithms.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] 1. The present invention has high robustness for measuring the speed of moving objects in complex outdoor scenes, and can adapt to changes in all regions, day and night, cloudy, rainy, snowy and all-weather conditions.

[0016] 2. The present invention uses a pyramid LK optical flow tracking algorithm based on dynamic ROI. This method is not affected by the speed of the moving object and can be applied to any speed change within the range of 0-80km / h.

[0017] 3. The present invention not only uses the LK optical flow tracking algorithm for tracking feature points between previous and next frame images, but also uses it for tracking feature points of binocular left and right images, replacing the traditional feature point matching of binocular images with descriptors, greatly improving the efficiency of feature point matching of binocular cameras.

[0018] 4. The present invention has a small speed measurement delay and high precision, and can meet the speed measurement requirements of the railway industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of a method for visually measuring the speed of a moving object in a complex outdoor scene provided by an embodiment of the present invention;

[0020] Figure 2 It is a schematic diagram of binocular image optical flow tracking of a method for visually measuring the speed of a moving object in a complex outdoor scene provided by an embodiment of the present invention;

[0021] Figure 3 It is a structural schematic diagram of a measuring device for implementing a method for visually measuring the speed of a moving object in a complex outdoor scene provided by an embodiment of the present invention; DETAILED DESCRIPTION

[0022] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0023] Embodiment 1:

[0024] This embodiment provides a method for visually measuring the speed of a moving object in a complex outdoor scene. Figure 1 As shown, the following steps are included:

[0025] Step 1: Stereo camera calibration

[0026] The main purpose of binocular calibration is to obtain the intrinsic parameter matrix of each camera and the distortion coefficient matrix D = [K1K2 K3 P1 P2] and the relative position relationship of the two cameras, that is, the rotation matrix R and translation matrix T of the right camera relative to the left camera; where f is the focal length, unit: mm; d x and d y are the widths of a single pixel of the camera sensor in the x and y directions, respectively, in mm; u0 and v0 are the pixel coordinates of the camera principal point, in pixels; K1, K2 and K3 are the radial distortion coefficients; P1 and P2 are the tangential distortion coefficients.

[0027] For a stereo camera, there is the following formula:

[0028]

[0029] Among them, P w is the coordinate of the point on the calibration plate in the world coordinate system; P l , P r are the coordinates of the left and right cameras in the world coordinate system; R l ,T l are the rotation matrix and translation matrix from the calibration plate coordinate system to the left camera coordinate system; R r ,T r They are the rotation matrix and translation matrix from the calibration plate coordinate system to the right camera coordinate system;

[0030] By transforming it into P w The forms are:

[0031]

[0032] Subtract the two equations and multiply both sides by R r Then we can get:

[0033]

[0034] Finally, the relative position relationship of the left and right cameras can be obtained:

[0035]

[0036] Where R and T are the rotation matrix and translation matrix of the right camera relative to the left camera, respectively.

[0037] Take multiple pictures, R r ,R l ,T r ,T l It can be obtained by calculating the monocular image and intrinsic parameters, substituting them into the above formula, and solving R and T using the least squares method or singular value decomposition method. After obtaining the various parameters of the camera through binocular positioning, the stereo correction algorithm in OpenCV is used to obtain the reprojection matrix where c x 、c y is the coordinate of the left camera principal point in the image, f is the focal length, T x is the translation between the left and right camera projection centers, c x ′ is the coordinate of the right camera principal point in the image. The Q matrix is ​​used to realize the conversion between the world coordinate system and the image pixel coordinate system.

[0038] Step 2: Use the Shi-Tomasi corner point method to extract the feature points in the image taken by the binocular camera, and use the LK optical flow method to track the feature points of the left and right images to obtain the feature point set of the left image and the feature point set of the right image.

[0039] Each time a frame of image is collected, feature points need to be extracted from the left or right image as input to the LK optical flow tracking algorithm. The LK optical flow method only tracks the feature points in the image, so the accuracy of the image feature point extraction directly affects the final calculation result of the optical flow vector. The image feature point extraction in this method uses the Shi-Tomasi corner point method, which constructs the calculation formula E by calculating the grayscale change of the pixel point, that is, solving the maximum value of E:

[0040]

[0041] Where I(x,y) represents the grayscale value, w(x,y) is the window function, and u and v are the pixels that change horizontally and vertically. Taylor expansion of the above formula is:

[0042]

[0043] In the formula, Among them I x ,I y are the grayscale differences in the x and y directions respectively.

[0044] After the G matrix is ​​diagonalized, the change components in two orthogonal directions are extracted, namely λ1 and λ2 (eigenvalues). Shi-Tomasi found that the stability of the corner point is related to the smaller eigenvalue of the matrix G, so the smaller eigenvalue is directly used as the score. If the score is greater than the set threshold, it is considered a corner point. Finally, the score of each pixel is sorted from large to small, and the first N corner points are selected as the feature point set. Figure 2 As shown, the present invention selects the left image for feature point extraction and selects the first 600 corner points as the feature point set of the left image. The feature point set of the right image is obtained from the feature point set of the left image through the optical flow tracking algorithm. Since the positional relationship between the left camera and the right camera is fixed, the optical flow direction from the left image to the right image is consistent, and the erroneous optical flow can be filtered out by the optical flow direction. The filtered feature point set of the left image is recorded as The feature point set of the right image is recorded as and The feature point set obtained in this step is used to update the feature points on the latest acquired binocular image. The point set is first stored in the computer's memory and then used for optical flow tracking of the previous and next frame images after the next frame image is acquired.

[0045] Step 3: Use the pyramid LK optical flow tracking algorithm based on dynamic ROI (region of interest) to track the feature points of the previous frame and the current frame binocular image to obtain the left and right feature point sets of the current frame.

[0046] LK optical flow tracking has three assumptions: grayscale invariance assumption, perturbation invariance assumption, and spatial consistency assumption. Based on the first two assumptions, there are the following constraint equations:

[0047] I(x,y,t)=I(x+δx,y+δy,t+δt)

[0048] Where I(x,y,t) refers to the grayscale value of the pixel coordinate (x,y) at time t;

[0049] After performing a first-order Taylor expansion on the constraint equation and ignoring higher-order terms, we can obtain:

[0050]

[0051] Dividing both sides by δt, we have

[0052]

[0053] in, and is the speed of the pixel along the x and y directions, that is, the displacement divided by time. The above formula can be simplified as:

[0054] I x v x +I yv y +I t =0

[0055] In matrix form,

[0056]

[0057] Using the third assumption, we can assume that within an m*m window, the optical flow is a constant value, that is:

[0058]

[0059] …

[0060]

[0061] The optical flow v can be directly solved using the least squares method x , v y .

[0062] The LK optical flow method is based on the assumption of continuous small motions. In practical applications, large-scale discontinuous motions will occur when the moving object is fast or the camera capture frame rate is low. The pyramid-based LK optical flow tracking algorithm can solve this problem. The specific method is to calculate the optical flow from the highest level of the image pyramid, and then use the calculation result of this level as the initial value for the next level calculation, and repeat this process until the original image. After adopting the pyramid technology, large-scale discontinuous motions can meet the assumption of continuous small motions, thereby realizing optical flow tracking for faster motions. However, the pyramid-based method will cause some weaker feature points in the image to disappear in the high-level pyramid image. When the calculation reaches the lower level pyramid, some weaker feature points cannot be correctly tracked because there is no good initial value. The present invention uses a pyramid LK optical flow tracking algorithm based on dynamic ROI. When there is no moving target in the field of view, a high-level pyramid image is used to track the optical flow of the front and rear frame images. When a moving object is detected, the object's moving speed is calculated, and the ROI range of the rear frame image is dynamically adjusted according to the current speed value, so that the moving object is roughly aligned in the front and rear frame images. On this basis, a low-level pyramid image is used to perform LK optical flow tracking. This algorithm can capture fast-moving objects and make full use of the subtle feature points on the surface of the object for optical flow tracking. Specifically, Figure 2 As shown in FIG. 1 , the ROI range of the current frame is set using the pixel offset value of the moving object calculated in the previous frame image, so that the pixel coordinates of the moving object in the previous frame image and the current frame image are roughly aligned, and then the feature point set corresponding to the previous frame image is And the previous binocular image (ImgL n-1 , ImgR n-1 ), the current frame binocular image (ImgLn , ImgR n ) is used as the input of the pyramid-based LK optical flow tracking algorithm to calculate the feature point set of the left image corresponding to the current frame image And the feature point set on the right

[0063] It should be understood that the above steps 2 and 3 avoid the two time-consuming operations of feature descriptor calculation and feature point matching commonly used in traditional binocular reconstruction, and use the optical flow tracking algorithm to track the feature points corresponding to the left and right images, thereby greatly improving the efficiency of binocular image feature point matching.

[0064] Step 4: 3D reconstruction and velocity calculation

[0065] In step 3, during the process of tracking the optical flow of the feature point set of the previous binocular image to the feature point set of the current binocular image, the optical flow obtained by the LK optical flow tracking algorithm includes the image background optical flow and the optical flow of the moving object. The background optical flow (v x , v y ) is zero, so the feature point set corresponding to the moving object can be screened out by the modulus of the optical flow. It should be noted that the optical flow calculated by the LK optical flow tracking algorithm may contain some errors. We can use the physical law that the moving direction and displacement length of the feature points on the surface of the moving object are consistent, that is, the optical flow (v x , v y ) direction and modulus length, and the point set corresponding to the filtered optical flow is recorded as the feature point set of the left image of the new previous frame Feature point set of the right image of the new previous frame The feature point set of the left image of the new current frame And the feature point set of the right image of the new current frame

[0066] Traditional binocular 3D reconstruction generally involves correcting the binocular image and then calculating the parallax to complete the 3D point reconstruction. The binocular image correction in the process is relatively time-consuming, which will seriously affect the real-time performance of the system. The present invention uses an optical flow tracking algorithm based on LK to use distorted images for calculations, and finally corrects the feature points that are successfully tracked by the optical flow. In this way, only sparse point sets are corrected, avoiding the time-consuming operation of correcting the entire image, and effectively improving the measurement efficiency.

[0067] The new feature point set is reconstructed by binocular 3D to obtain the 3D point set of the previous frame and the current frame, including: And the feature point set of the right image of the new previous frame After stereo calibration, binocular 3D reconstruction is performed to obtain the 3D point set of the previous frame. Set the feature point set of the left image of the new current frame And the feature point set of the right image of the new current frame After stereo calibration, binocular 3D reconstruction is performed to obtain the 3D point set of the current frame. Among them, stereo correction includes distortion correction and parallel binocular correction; the calculation formula for binocular three-dimensional reconstruction is as follows:

[0068]

[0069]

[0070] Substituting the feature point coordinates, disparity and Q matrix obtained in step 1 into the above formula, we can get the three-dimensional point set and Where d(u,v) is the disparity of the feature point at pixel coordinates u,v, W is the scale factor, Q is the reprojection matrix; X′, Y′, Z′ are the intermediate quantities calculated, and X, Y, Z are the final three-dimensional point coordinates.

[0071] Finally, the speed of the moving object is calculated based on the 3D point set of the previous frame and the current frame, including: and the 3D point set of the current frame Subtract and obtain the spatial displacement vector of the same feature point set within the time interval of two frame image acquisitions Filter out the spatial displacement vectors with zero displacement; perform median filtering on the directions and modulus lengths of the remaining vectors to remove the spatial displacement vectors with large errors; Taking the derivative of the time difference T between the two frames of images, we can get the speed V of the moving object and the spatial displacement vector The direction is the direction of movement of the object.

[0072] Embodiment 2:

[0073] This embodiment provides a measuring device for implementing the method for visually measuring the speed of a moving object in a complex outdoor scene described in the first embodiment. Figure 3 As shown, it includes a binocular camera, a stroboscopic light source and a PC; the binocular camera is electrically connected to the stroboscopic light source and the PC respectively, and the binocular camera is electrically connected to the PC through an optical fiber cable; the binocular camera includes a left camera and a right camera, and the left camera and the right camera take pictures synchronously when the stroboscopic light source is turned on; the PC includes but is not limited to a CPU and a GPU, the CPU is used to perform logical operations, and the GPU is used to accelerate feature point extraction and optical flow tracking algorithms.

[0074] Specifically, the device is divided into an outdoor part and an indoor part; the outdoor part mainly includes a set of high-speed binocular cameras and a high-brightness stroboscopic light source. During acquisition, the left camera collects images at a fixed time interval, and then uses the Strobe signal of the left camera as the input signal of the right camera and the light source controller to trigger the right camera to take pictures and control the light source to turn on and off. By adjusting the delay of the Strobe signal, the left and right cameras can take pictures synchronously after the light source turns on. The indoor part is a PC, including a computer, a display and a graphics card. The CPU in the computer is mainly used to implement some logical operations, the GPU on the graphics card is mainly used to accelerate feature point extraction and optical flow tracking algorithms, the display is used to display the calculated speed of the moving object, and the memory in the computer is used to store the captured images and the process quantity of the speed measurement operation.

[0075] Speed ​​measurement principle: The same feature point is reconstructed in three dimensions in the binocular images taken at the first and second moments respectively, and the spatial displacement of the feature point is calculated using the three-dimensional point at the second moment and the three-dimensional point at the first moment; the time difference is calculated using the exposure timestamps of the binocular cameras at the first and second moments; finally, the moving speed of the object is calculated by taking the time derivative of the displacement.

[0076] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for visually measuring the speed of a moving object in a complex outdoor scene, characterized in that: include: Calibrate the binocular camera; The Shi-Tomasi corner point method is used to extract the feature points in the image taken by the binocular camera, and the LK optical flow method is used to track the feature points of the left and right images to obtain the feature point set of the left image and the feature point set of the right image; The dynamic ROI-based pyramid LK optical flow tracking algorithm is used to track the feature points of the previous and current binocular images to obtain the left and right feature point sets of the current frame. Filter out a new feature point set with successful optical flow tracking and small optical flow error from the corresponding feature point set tracked from the previous frame to the current frame; The new feature point set is subjected to binocular 3D reconstruction to obtain the 3D point sets of the previous frame and the current frame; Calculate the speed of the moving object based on the 3D point set of the previous frame and the current frame; The pyramid LK optical flow tracking algorithm based on dynamic ROI is used to track the feature points of the binocular images of the previous frame and the current frame to obtain the feature point set of the left image and the feature point set of the right image of the current frame, including: The pixel offset value of the moving object calculated in the previous frame is used to set the ROI range of the current frame, and the pixel coordinates of the moving object in the previous frame and the current frame are roughly aligned; The left and right feature point sets corresponding to the previous frame image, and the binocular images of the previous frame and the current frame are used as the input of the pyramid-based LK optical flow tracking algorithm to calculate the left and right feature point sets corresponding to the current frame image; The dynamic ROI-based pyramid LK optical flow tracking algorithm uses a high-level pyramid image to perform LK optical flow tracking of the front and rear frame images when there is no moving object in the field of view; when a moving object is detected, the object's moving speed is calculated and the ROI range of the rear frame image is dynamically adjusted according to the current speed value, and the moving objects in the front and rear frame images are roughly aligned, and then the low-level pyramid image is used to perform LK optical flow tracking of the front and rear frame images.

2. The method for visually measuring the speed of a moving object in a complex outdoor scene according to claim 1, characterized in that: The binocular cameras are calibrated to obtain the intrinsic parameter matrix and distortion coefficient matrix of each camera, the relative position relationship between the two cameras, and the reprojection matrix; The internal parameter matrix is: Where f is the focal length, d is x and d y are the widths of a single pixel in the camera sensor in the x and y directions, respectively, and u0 and v0 are the pixel coordinates of the camera principal point; The distortion coefficient matrix is: D=[K1 K2 K3 P1 P2] Where K1, K2 and K3 are radial distortion coefficients, P1 and P2 are tangential distortion coefficients; The relative position relationship of the two cameras is: Where R and T are the rotation matrix and translation matrix of the right camera relative to the left camera respectively; R l ,T l is the rotation matrix and translation matrix from the calibration plate coordinate system to the left camera coordinate system, R r ,T r The rotation matrix and translation matrix from the calibration plate coordinate system to the right camera coordinate system; The reprojection matrix is: In the formula, c x and c y is the coordinate of the left camera principal point in the image, f is the focal length, T x is the translation between the left and right camera projection centers, c′ x are the coordinates of the right camera principal point in the image.

3. The method for visually measuring the speed of a moving object in a complex outdoor scene according to claim 1 is characterized in that: The Shi-Tomasi corner point method is used to extract feature points in the image taken by the binocular camera, and the LK optical flow method is used to track the feature points of the left and right images to obtain the feature point set of the left image and the feature point set of the right image, including: Corner points are detected by calculating the grayscale changes of pixels in the image, and the top N corner points in stability ranking are selected as feature points; The LK optical flow method is used to track the feature points of the left and right images.

4. The method for visually measuring the speed of a moving object in a complex outdoor scene according to claim 3 is characterized in that: The feature points are detected and selected from the image taken by the left camera; The LK optical flow method is used to track the feature points of the left and right images, including: tracking from the left image to the right image by the LK optical flow method and filtering the optical flow to obtain a left image feature point set and a right image feature point set.

5. The method for visually measuring the speed of a moving object in a complex outdoor scene according to claim 1, characterized in that: The new feature point sets are respectively recorded as the feature point sets of the left image of the new previous frame Feature point set of the right image of the new previous frame The feature point set of the left image of the new current frame And the feature point set of the right image of the new current frame 6. The method for visually measuring the speed of a moving object in a complex outdoor scene according to claim 5, characterized in that: The new feature point set is subjected to binocular 3D reconstruction to obtain the 3D point sets of the previous frame and the current frame, including: The feature point set of the left image of the new previous frame And the feature point set of the right image of the new previous frame After stereo calibration, binocular 3D reconstruction is performed to obtain the 3D point set of the previous frame. Set the feature point set of the left image of the new current frame And the feature point set of the right image of the new current frame After stereo calibration, binocular 3D reconstruction is performed to obtain the 3D point set of the current frame. Among them, the calculation formula for binocular 3D reconstruction is: Where d(u,v) is the disparity of the feature point at pixel coordinates u,v, W is the scale factor, and Q is the reprojection matrix; X ′ ,Y ′ ,Z ′ To calculate the intermediate quantity, X, Y, and Z are the final three-dimensional point coordinates.

7. The method for visually measuring the speed of a moving object in a complex outdoor scene according to claim 1, characterized in that: Calculate the speed of the moving object based on the 3D point set of the previous frame and the current frame, including: Subtract the 3D point set of the previous frame from the current frame to obtain the spatial displacement vector of the same feature point set within the time interval between the two frames of images; Remove the spatial displacement vector with a displacement of 0; Perform median filtering on the vector modulus and vector direction to remove spatial displacement vectors with large errors; The spatial displacement vector is divided by the time difference between the two frames of images to obtain the speed of the moving object in the two frames of images. The direction of the spatial displacement vector is the moving direction of the moving object.

8. A measuring device for implementing the visual measurement method of the speed of a moving object in an outdoor complex scene as described in any one of claims 1 to 7, comprising a binocular camera, a stroboscopic light source and a PC; the binocular camera is electrically connected to the stroboscopic light source and the PC respectively; the binocular camera comprises a left camera and a right camera, and the left camera and the right camera take pictures synchronously when the stroboscopic light source is turned on; the PC comprises a CPU and a GPU, the CPU is used to perform logical operations, and the GPU is used to accelerate feature point extraction and optical flow tracking algorithms.

Images