Slope monitoring method based on optical flow estimation and binocular vision

Abstract

A slope monitoring method based on optical flow estimation and binocular vision belongs to the technical field of deformation monitoring and measurement, and adopts the technical scheme that a binocular stereoscopic vision system is built, and camera acquisition parameters are calibrated; transmitting the image to a server in real time through the Internet of Things; distortion correction and stereo matching are carried out to obtain a depth map; monitoring points are manually or automatically arranged on the corrected image, and dense optical flow estimation tracking pixel point movement is carried out; world coordinates of observation points and movement conditions of each frame are obtained, and displacement changes of slope monitoring points are obtained by combining coordinates before and after displacement; and obtaining a displacement rate by combining monitoring time and displacement, judging slope safety, obtaining a displacement track, and performing linear interpolation to obtain a slope deformation cloud picture, thereby realizing the monitoring purpose. The slope displacement monitoring method has the advantages that slope monitoring can be carried out more flexibly and efficiently, the safety of workers is improved, the slope monitoring cost is reduced, real-time slope displacement monitoring is achieved, and the intensive degree of slope monitoring points is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deformation monitoring and measurement, and relates to a non-contact surface deformation quantitative monitoring and early warning method and a monitoring system, and in particular to a slope monitoring method based on optical flow estimation and binocular vision. Background Art

[0002] Slope instability refers to the situation where the slope is affected by various adverse external factors, such as rainfall, groundwater action, earthquakes, human engineering activities, etc., which causes the structure of the slope soil or rock to change, lose its original stability, and cause destructive phenomena such as sliding and collapse, resulting in landslides.

[0003] When a landslide occurs, it may cause the collapse of buildings such as houses and bridges, as well as possible losses of farmland and damage to infrastructure, causing huge losses to people's lives and property, and may also have long-term impacts on the ecological environment and social economy.

[0004] By monitoring slopes, we can detect signs of landslides in advance, issue early warnings, and implement preventive measures to reduce casualties and property losses. After a landslide occurs, monitoring data can provide real-time information to guide emergency rescue efforts and minimize the damage caused by the disaster.

[0005] Traditional slope monitoring methods, such as the Global Navigation Satellite System (GNSS), LiDAR (TLS), physical monitoring, and Interferometry (InSAR), all have limitations. For example, TLS, InSAR, and GNSS are relatively expensive, while GNSS and some physical monitoring methods have limited monitoring point density. Consequently, non-contact measurement methods based on machine vision have emerged in recent years. However, due to the lack of distinct slope feature points, it is difficult to match monitoring points before and after deformation. Existing methods rely on inserting artificial markers on the slope. These markers have distinct features that can be easily perceived by computers. Slope monitoring is achieved by monitoring the displacement of these markers. However, this method has significant limitations. Some slope markers are difficult to insert, the density of monitoring points is limited, it cannot represent the actual deformation of the slope surface, it is difficult to ensure that marker displacement is always caused by slope changes, and the durability of the markers needs to be considered.

[0006] Therefore, a slope monitoring method with high efficiency, dense monitoring points, low price and high flexibility is needed. Summary of the Invention

[0007] In order to solve the above-mentioned technical problems existing in the prior art, the present invention provides a slope monitoring method based on optical flow estimation and binocular vision. This method provides a high-efficiency, low-cost and intensive slope monitoring means to overcome the limitations of traditional monitoring methods and the shortcomings of marker methods; by combining optical flow estimation and binocular vision technology, real-time monitoring of slope deformation is achieved; the accuracy and real-time nature of the monitoring data are ensured, providing strong support for landslide warning and emergency rescue.

[0008] To achieve the above object, the present invention provides the following technical solutions:

[0009] A slope monitoring method based on optical flow estimation and binocular vision, the steps are as follows:

[0010] S1. Build a binocular stereo vision system and calibrate the built binocular stereo camera to obtain camera parameter information;

[0011] S2. Place the monitoring system opposite the slope to be monitored and transmit the images captured by binocular vision to the server in real time through the Internet of Things.

[0012] S3, dividing the photos captured by the binocular camera into two groups according to the left and right camera shots, and performing distortion correction and binocular stereo matching on the two groups of photos to obtain depth maps;

[0013] S4. Manually or automatically arrange monitoring points on the corrected image, perform dense optical flow estimation on the monitoring points, and track the movement of the monitoring pixels;

[0014] S5. The world coordinates of the observed point can be obtained through step S3. The movement of the monitoring point in each frame can be obtained through step S4. Combined with the world coordinates before and after the displacement of the monitoring point, the displacement change of the slope monitoring point can be obtained;

[0015] S6. By combining the monitoring time and the monitoring displacement, the displacement rate of a certain monitoring point can be obtained to judge the safety of the slope. By obtaining the slope displacement trajectory, linear interpolation is performed on the displacement of the slope monitoring point array to obtain a more intuitive slope deformation cloud map, thereby achieving the purpose of slope monitoring.

[0016] Furthermore, in step S1, an existing binocular camera calibration algorithm is used to write a program by oneself, or the Stereo Camera Calibrator toolkit of MATLAB is used to perform camera calibration; the intrinsic parameter matrices, distortion parameters, rotation matrices, and translation matrices of the left and right cameras are obtained.

[0017] Furthermore, in step S2, the monitoring system is placed opposite the slope to be monitored, and the images captured by binocular vision are transmitted to the server in real time through the Internet of Things.

[0018] Furthermore, in step S3, the images captured by the left and right cameras are subjected to distortion correction according to the camera parameters obtained in step S1, and stereo matching is performed and disparity is obtained by the SGBM algorithm, and the world coordinates of the observation point and the depth map are obtained according to the disparity and coordinate conversion formula.

[0019] Furthermore, in step S4, monitoring points are manually or automatically arranged on the corrected image obtained in step S3, and dense optical flow estimation based on deep learning is performed on these monitoring points to track the movement of the monitoring pixels.

[0020] Furthermore, in step S5, the pixel coordinates of the i-th monitoring point at the t-th monitoring time point obtained in step S4 are And the corresponding world coordinates obtained in step S3 Save, according to the world coordinate changes of the same monitoring point at time t1 and t2, the displacement change of the monitoring point in the time period t1 and t2 can be obtained

[0021] Furthermore, in the step S6, the displacement change of the monitoring point in the time period t1 and t2 is calculated. Dividing the time |t1 - t2| yields the deformation rate. Using the above method, we obtain the displacement of each point. Linearly interpolating the displacements of adjacent pixels yields the displacement of each point in the monitored area. Normalizing the displacement values ​​to a range of 0-255, converting them to RGB values, and plotting them yields a more intuitive slope displacement cloud map.

[0022] Beneficial effects of the present invention:

[0023] Compared with the existing technology, the slope monitoring method based on optical flow estimation and binocular vision described in the present invention has the following technical features or beneficial effects:

[0024] (1) Significantly reduce monitoring costs: Compared with traditional slope monitoring methods, such as the Global Navigation Satellite System (GNSS) and LiDAR (TLS), the present invention only requires a depth camera to complete the monitoring task, which greatly reduces the equipment cost and makes slope monitoring more economical and feasible.

[0025] (2) Improved monitoring efficiency and accuracy: By combining optical flow estimation and binocular vision technology, the present invention enables high-precision, real-time monitoring of slope deformation. The binocular stereo vision system can obtain three-dimensional information about the slope, while dense optical flow estimation can accurately track the movement of monitoring points, thereby improving monitoring efficiency and accuracy.

[0026] (3) Enhanced density and flexibility of monitoring points: The density of monitoring points in the present invention depends on hardware conditions (such as camera resolution and server performance) and can be adjusted according to actual needs. Furthermore, since no artificial markers are required on the slope, the monitoring method of the present invention is more flexible and applicable to various complex terrains and slopes where markers are difficult to place.

[0027] (4) Ensure the safety of the monitoring process: The present invention adopts a non-contact measurement method, and the monitoring equipment will not affect the slope, thus ensuring the safety of the monitoring process. At the same time, since monitoring personnel do not need to enter the dangerous area of ​​the slope, the risk of casualties is greatly reduced.

[0028] (5) Realize real-time monitoring and early warning functions: The images captured by binocular vision are transmitted to the server in real time through the Internet of Things technology. The present invention can obtain the deformation of the slope in real time, and combine information such as displacement rate and displacement trajectory to issue early warnings in time, providing strong support for landslide early warning and emergency rescue.

[0029] (6) Providing intuitive monitoring results: The present invention can display the deformation of the slope in an intuitive and easy-to-understand manner by drawing a slope displacement cloud map, which helps monitoring personnel quickly understand the stability of the slope and provides a strong basis for subsequent decision-making and response measures.

[0030] The present invention not only reduces the cost of slope monitoring, improves monitoring efficiency and accuracy, but also enhances the density and flexibility of monitoring points, ensures the safety of the monitoring process, and realizes real-time monitoring and early warning functions, bringing significant progress and beneficial effects to the field of slope monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and detailed embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be derived from these drawings without inventive effort. Among them:

[0032] Figure 1 Schematic diagram of the pinhole imaging model;

[0033] Figure 2 Schematic diagram of the camera sensor pixel structure;

[0034] Figure 3 This is the transformation relationship diagram between the camera coordinate system and the world coordinate system;

[0035] Figure 4 Schematic diagram of binocular stereo camera imaging;

[0036] Figure 5(a) is the original image in Example 1;

[0037] Figure 5(b) is the depth map generated in Example 1;

[0038] Figure 6 This is the optical flow estimation effect diagram in Example 1. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Figure 1-6 The slope monitoring method based on optical flow estimation and binocular vision is further explained, and the technical solutions in the embodiments of the present application are clearly and completely described; obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0040] The steps of the present invention include:

[0041] S1: Build a binocular stereo vision system and calibrate the built binocular stereo camera to obtain camera parameter information.

[0042] S2, the monitoring system is placed opposite the slope to be monitored, and the images captured by binocular vision are transmitted to the server in real time through the Internet of Things.

[0043] In S3, the photos captured by the binocular camera are divided into two groups according to the left and right camera shots, and the two groups of photos are subjected to distortion correction and binocular stereo matching to obtain depth maps.

[0044] S4, manually or automatically arranges monitoring points for the rectified image, performs dense optical flow estimation on the monitoring points, and tracks the movement of the monitoring pixels.

[0045] S5. The world coordinates of the observed point can be obtained through step S3. The movement of the monitoring point in each frame can be obtained through step S4. Combined with the world coordinates before and after the displacement of the monitoring point, the displacement change of the slope monitoring point can be obtained.

[0046] S6, combining monitoring time and monitoring displacement, can determine the displacement rate of a monitoring point and judge the slope safety. By obtaining the slope displacement trajectory, linear interpolation is performed on the displacement of the slope monitoring point array to obtain a more intuitive slope deformation cloud map, thus achieving the purpose of slope monitoring.

[0047] Example 1

[0048] The present invention is further described in detail below with reference to the embodiments so that those skilled in the art can implement the invention with reference to the description.

[0049] S1. Build a binocular stereo vision system. Based on the approximate distance from the measured slope to the binocular camera system, select the resolution of the two cameras and the distance between the cameras, i.e., the baseline length b. Use a checkerboard or other calibration board for calibration. The specific method is as follows:

[0050] In the camera imaging model, there are two coordinate systems, one is the world coordinate system, that is, the coordinates of the real world, and the other is the camera coordinate system. This coordinate system is established to facilitate the use of the pinhole imaging model to establish the connection between pixel coordinates and the real world. We first complete this step, such as Figure 1 As shown in the figure, it is a top view of the pinhole imaging model. Assume that the coordinates of a point in the camera coordinate system are (x c ,y c ,z c ), according to the triangle similarity principle:

[0051]

[0052] In the above formula (x i ,y i ) is the coordinate of the camera point on the imaging plane, and now convert the coordinate into the pixel coordinate of the image. Figure 2 As shown, the pixel coordinate origin of the image is generally the first pixel in the upper left corner. Assume that the coordinate of the intersection point of the camera coordinate system Z axis and the sensor plane is (o x ,o y ), then the pixel coordinates of the imaging coordinate point on the image plane are:

[0053]

[0054] where m x 、m y Represents the pixel density of the camera in the X and Y directions, that is, the number of pixels per unit length. x 、o y represents the horizontal and vertical coordinates of the intersection of the camera coordinate system Z axis and the sensor plane, and f represents the focal length of the camera. (u, v) is the pixel coordinate of the imaging point. Here, m x f、m y f are considered as a whole x 、f y , these two values ​​can be regarded as the effective focal length in the X and Y directions. The relationship between the camera coordinate system and the image pixel coordinate system can be established through formula 3-4. The above variables (f x ,f y ,o x ,o y) is called the intrinsic parameter of the camera. In order to convert the camera model into a linear model, the second coordinate transformation is used, and the relationship between the image coordinates and the camera coordinate system coordinates can be written in the form of a matrix:

[0055]

[0056] The matrix on the right side of the equal sign in the above formula is the camera's intrinsic parameter matrix M in , the right side is the coordinate C in the camera coordinate system w The relationship between the camera coordinate system and the world coordinate system is established below. Figure 3 As shown in the figure, the world coordinate system can coincide with the camera coordinate system after rotation and translation.

[0057] Therefore, the transformation formula between the world coordinate system and the camera coordinate system is:

[0058] C=RC w +T (6)

[0059] In the above formula, R and T represent the rotation matrix and translation matrix respectively.

[0060]

[0061] Therefore, the conversion formula from the world coordinate system to the image pixel coordinate is:

[0062]

[0063] Through camera calibration, the above unknown parameters can be solved and the camera internal and external parameter matrix can be obtained. For a binocular camera, its model is as follows: Figure 4 As shown, according to the above principle, the coordinates (x, y, z) of a point in the camera coordinate system can be solved by calibration as follows:

[0064]

[0065] In the above formula (u l ,v l )、(u r ,v r ) are the pixel coordinates of point P in the left and right cameras. (u l -u r ) is called parallax d.

[0066] A camera lens is composed of multiple lenses of varying focal lengths. Lenses positioned at different focal lengths bend light to varying degrees. This curvature error caused by the lenses results in image distortion known as radial distortion. During camera production, inaccurate lens and sensor assembly can lead to misalignment, causing the imaging plane to be non-parallel to the camera plane. This ultimately leads to an error in the resulting image's position, known as tangential distortion.

[0067] The components of radial distortion in the X and Y directions can be expressed as follows:

[0068]

[0069] in: k1, k2, and k3 are the first-order, second-order, and third-order radial distortion coefficients of the camera, respectively. In practical applications, the third-order radial distortion coefficient k3 of conventional cameras is extremely small. Therefore, k3 is generally considered to be 0 during calibration and distortion correction, and only the first two order distortion coefficients are calculated. The components of tangential distortion in the X and Y directions can be expressed as follows:

[0070]

[0071] Where p1 and p2 are the tangential distortion coefficients of the camera.

[0072] Place the checkerboard calibration plate in front of the camera and take a series of photos. When calibrating, keep in mind three key points: First, change the calibration plate's position while taking photos, ensuring it is captured in every possible part of the frame. Second, rotate the calibration plate in three directions to maximize the angle variation. Third, keep the calibration plate as close as possible to the distance from the slope to be monitored to the camera, which may make the calibration plate very large.

[0073] Use the calibration program of OpenCV library or MATLAB's Stereo Camera Calibrator toolkit to complete the camera calibration. Get the camera's intrinsic parameter matrix M in1 、M in2 , and the camera's distortion parameters k1, k2, k3, p1, p2, where k3 defaults to 0.

[0074] Furthermore, in step S2, the monitoring system is placed opposite to the slope to be monitored, and the images captured by binocular vision are transmitted to the server in real time through the Internet of Things.

[0075] Furthermore, in step S3, the image is first corrected using the camera's distortion parameters, and stereo matching is performed using the SGBM algorithm to obtain the disparity d. The camera coordinates can be calculated according to formulas 9-11, and the depth calculation formula in the scene is:

[0076]

[0077] The depth map can be obtained by calculating the depth of each pixel. The depth map in this example is as follows Figure 5b As shown. Then each pixel in the image can obtain the world coordinates at time t through its pixel coordinates (u, v) i represents the pixel index.

[0078] Furthermore, in step S4, traditional optical flow estimation often performs poorly in complex scenes, occlusions, illumination changes, and areas with little texture, and relies on the selection of good feature points (such as corners and edges). It is not suitable for slope monitoring tasks, so the present invention adopts dense optical flow estimation based on deep learning. The neural network model used in this embodiment adopts the Transformer architecture, which can encode the tracking information of each point in the video and iteratively update the positions of these points. In addition, a window mechanism is introduced to divide the time axis into sliding windows, use the output of the previous window to initialize subsequent windows, and run multiple Transformer iterations on each window. Note that the scope of this version of the present invention should not be limited to the neural network model, but should include all dense optical flow estimation methods. This optical flow estimation method can theoretically monitor any pixel point, and the tracking effect of the pixel point is as follows. Figure 6 shown.

[0079] Furthermore, in step S5, the pixel coordinates of the i-th monitoring point at the t-th monitoring time point obtained in step S4 are converted to And the corresponding world coordinates Save the displacement change of the monitoring point within a certain period of time The calculation formula is:

[0080]

[0081] The displacement change of the slope monitoring point can be obtained through the above formula.

[0082] Furthermore, in step S6, the displacement change rate is:

[0083]

[0084] This allows us to determine the displacement rate of a monitoring point and assess the safety of the slope. Furthermore, optical flow estimation and pixel tracking can be used to obtain the slope displacement trajectory, providing a more intuitive understanding of the slope's historical movement.

[0085] Perform linear interpolation on the displacement of the slope monitoring point matrix, normalize the displacement value to between 0 and 255, convert it into RGB value and draw it to obtain a more intuitive slope displacement cloud map, thereby achieving the purpose of slope monitoring.

[0086] The present invention has the following beneficial effects: low cost, only one depth camera is needed; convenient and easy to install; only the camera needs to be facing the slope; high safety, the monitoring equipment will not affect the slope, non-contact measurement can ensure the safety of personnel; monitoring points are dense (density is optional), depending on hardware conditions (camera resolution, server performance, etc.); it is real-time, with fast data processing speed, and can also be used as a camera for real-time monitoring.

[0087] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A slope monitoring method based on optical flow estimation and binocular vision, characterized in that: Here are the steps: S1. Build a binocular stereo vision system and calibrate the built binocular stereo camera to obtain camera parameter information; S2. Place the monitoring system opposite the slope to be monitored and transmit the images captured by binocular vision to the server in real time through the Internet of Things method; S3, dividing the photos captured by the binocular camera into two groups according to the left and right camera shots, and performing distortion correction and binocular stereo matching on the two groups of photos to obtain depth maps; S4. Manually or automatically arrange monitoring points on the corrected image, perform dense optical flow estimation on the monitoring points, and track the movement of the monitoring pixels; S5. Obtain the world coordinates of the observed point through step S3, obtain the movement of the monitoring point in each frame through step S4, and obtain the displacement change of the slope monitoring point by combining the world coordinates before and after the monitoring point is displaced; S6. Combine the monitoring time and monitoring displacement to obtain the displacement rate of a monitoring point and judge the safety of the slope by obtaining the slope displacement trajectory; perform linear interpolation on the displacement of the slope monitoring point array to obtain a more intuitive slope deformation cloud map, thereby achieving the purpose of slope monitoring.

2. The slope monitoring method based on optical flow estimation and binocular vision according to claim 1, characterized in that: In the step S1, a binocular camera calibration algorithm is used, or the Stereo Camera Calibrator toolkit of MATLAB is used to perform camera calibration; the intrinsic parameter matrices, distortion parameters, rotation matrices, and translation matrices of the left and right cameras are obtained.

3. The slope monitoring method based on optical flow estimation and binocular vision according to claim 1, characterized in that: In the step S2, the monitoring system is placed opposite the slope to be monitored, and the images captured by binocular vision are transmitted to the server in real time through the Internet of Things method.

4. The slope monitoring method based on optical flow estimation and binocular vision according to claim 1, characterized in that: In step S3, the images captured by the left and right cameras are corrected for distortion according to the camera parameters obtained in step S1, and stereo matching is performed using the SGBM algorithm to obtain disparity. The world coordinates of the observation point and the depth map are obtained according to the disparity and coordinate conversion formula.

5. The slope monitoring method based on optical flow estimation and binocular vision according to claim 1, characterized in that: In step S4, monitoring points are manually or automatically arranged for the corrected image obtained in step S3; dense optical flow estimation based on deep learning is performed on these monitoring points to track the movement of the monitoring pixels.

6. The slope monitoring method based on optical flow estimation and binocular vision according to claim 1, characterized in that: In step S5, the pixel coordinates of the i-th monitoring point at the t-th monitoring time point obtained in step S4 are And the corresponding world coordinates obtained in step S3 Save, and based on the world coordinate changes of the same monitoring point at time t1 and t2, obtain the displacement change of the monitoring point in the time period t1 and t2 7. The slope monitoring method based on optical flow estimation and binocular vision according to claim 6, characterized in that: In step S6, the displacement change of the monitoring point in the time period t1 and t2 is calculated. The deformation rate can be obtained by dividing by the time |t1-t2|. The displacement of each point is obtained by the above method, and the displacement of each point in the monitoring area is obtained by linear interpolation of the displacements of adjacent pixels. The displacement value is normalized to between 0 and 255, converted to RGB value and plotted to obtain a more intuitive slope displacement cloud map.

Images