A dynamic and static landslide monitoring method based on monocular camera

Abstract

The present invention provides a method for static and dynamic landslide monitoring based on a monocular camera, comprising the following steps: collecting image data of a slope to be monitored as a dataset for a convolutional neural network model; using the convolutional neural network model to segment and select noise-free image data from the dataset; performing pre-processing on the noise-free image data; performing static change monitoring based on image difference and structural similarity index, and dynamic change monitoring based on a triple optical flow algorithm on the pre-processed image data, to calculate the displacement present in the image data of the slope to be monitored; and determining whether a landslide has occurred based on the displacement velocity calculated from the static and dynamic change monitoring. By combining static and dynamic changes, this method accurately and real-timely monitors slope displacement and deformation, effectively improving landslide monitoring capabilities, reducing losses caused by landslides, and being widely applicable in practical engineering applications.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and in particular to a method for dynamic and static landslide monitoring based on a monocular camera. Background Art

[0002] my country is a mountainous country, especially in the central and western regions, where the topography and geology are complex and located at the junction of many plates. This region is significantly affected by geological activity, coupled with the influence of climate and human activities, leading to frequent geological disasters. Landslides are a major concern among geological disasters, causing enormous losses. Therefore, research on landslide monitoring technology is of great practical significance.

[0003] Traditional monitoring methods require a large number of technicians and have played a role in early landslide monitoring. However, they lack real-time performance and require significant human, material, and financial resources. Monitoring results are also affected by the technicians' knowledge and experience. With the advancement of science and technology, landslide monitoring methods are continuously moving towards automation, networking, and high precision. However, practical applications still have many limitations. Current landslide monitoring methods mostly utilize various sensors, but these sensors are difficult to install and their universality cannot be guaranteed.

[0004] After research, the inventors of this application found that landslides usually manifest as displacement deformation of the slope. Therefore, monitoring the displacement state of the slope is a reliable means. Therefore, how to innovatively provide a landslide monitoring method based on the slope deformation characteristics during landslide has become a problem that needs to be solved urgently. Summary of the Invention

[0005] Most existing landslide monitoring methods use various sensors for monitoring, but the installation of sensors is difficult and their versatility cannot be guaranteed, resulting in many technical limitations. The present invention provides a dynamic and static landslide monitoring method based on a monocular camera. This method can combine static and dynamic changes to accurately and in real time monitor the displacement and deformation of the slope, effectively improving the monitoring capability of landslide disasters, reducing the losses caused by landslide disasters, and can be widely used in actual engineering.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] A method for dynamic and static landslide monitoring based on a monocular camera, comprising the following steps:

[0008] S1. Collect image data of the slope to be monitored as a data set for a convolutional neural network model, and use the convolutional neural network model to segment and select image data without noise influence from the data set;

[0009] S2, performing pre-processing operations on the image data without noise influence;

[0010] S3, performing static change monitoring based on image difference and structural similarity index and dynamic change monitoring based on triple optical flow algorithm on the image data after pre-processing operation, and calculating the displacement speed of the image data of the slope to be monitored;

[0011] S4. Determine whether a landslide disaster has occurred based on the displacement velocity calculated from dynamic and static change monitoring.

[0012] Furthermore, the step S1 specifically includes:

[0013] S11. Use a monocular camera to periodically collect image data of the slope to be monitored and establish a data set;

[0014] S12. Select a convolutional neural network algorithm to establish a convolutional neural network model, and use the convolutional neural network model to classify the image data in the data set, dividing the image data into two categories: those with noise influence and those without noise influence.

[0015] Furthermore, the convolutional neural network model established in step S12 includes a convolutional layer, a pooling layer, a first fully connected layer, a Dropout layer and a second fully connected layer connected sequentially. The number of convolution kernels in the convolution layer is 32, the convolution kernel size is 3×3, the pooling kernel size of the pooling layer is 2×2, the number of neurons in the first fully connected layer is 256, the dropout rate of the Dropout layer is 0.25, and the number of neurons in the second fully connected layer is 2.

[0016] Furthermore, the step S2 specifically includes:

[0017] S21. Calculate a transformation matrix between the images based on the mutual matching characteristics between the images, and perform image conversion based on the transformation matrix to map one image to another image to achieve image alignment;

[0018] S22, normalizing the image pixels and then performing intensity stretching to enhance contrast and improve image quality;

[0019] S23. Perform a histogram equalization operation on the image data, that is, widen the grayscale values of pixels with a large number of pixels and merge the grayscale values of pixels with a small number of pixels, so as to make the image clear and achieve image enhancement.

[0020] Furthermore, the step S3 specifically includes:

[0021] S31. Perform pixel-to-pixel subtraction on the image at a given time t. For a specific date Im1t and a different date Im2t, the image difference is calculated as follows:

[0022] Image Difference=Im1t-Im2t

[0023] S32, the structural similarity index measures the similarity of images based on a combination of three independent elements: brightness value, contrast value, and regional structure. Assuming that x and y are the positions of two regions of the same size in an image, the similarity of brightness value is l(x, y), the similarity of contrast is c(x, y), and the similarity of regional structure is s(x, y), the calculation formula of the structural similarity index is as follows:

[0024]

[0025] Among them, μ x and μ y are the local sample means of regions x and y, σ x and σ y are the local standard deviations of the x and y regions, σ xy is the sample cross-correlation of x and y after removing the mean of the regions x and y, C1, C2, and C3 are small positive constants;

[0026] S33. Use the optical flow algorithm to monitor the dynamic changes of the target pixel in the image caused by the movement of the object in the continuous frame image. The most important thing in the optical flow rate is to assume that the brightness of the image is constant, that is, the position of the object can change in a short time, but the reflectivity and illumination will remain unchanged. The optical flow equation includes:

[0027] I(x,y,t)=I(x+△x,y+△y,t+△t)

[0028] Where I(x,y,t) is the image intensity at position (x,y) at time t, and the change in position is represented by the change in Δx and Δy at time Δt. After applying Taylor expansion on the right side of the optical flow equation, the following equation is obtained:

[0029]

[0030] By solving this equation, we can get the optical flow constraint equation:

[0031] I x u+I y ·v+I t =0

[0032] Among them, I x , I y , I t is the partial derivative of the pixel grayscale along the x, y, and t directions, and u and v are the velocity vectors of the optical flow along the x and y directions respectively;

[0033] S34. After the above steps S31 to S33, a velocity vector is assigned to each pixel in the image, thus forming a motion vector field. At a specific moment, the points on the image correspond one-to-one with the points on the actual slope. This correspondence can be calculated through projection, and the actual displacement velocity of each point on the slope can be obtained based on this.

[0034] S35. Map the displacement velocity v of the slope to be monitored by using the image difference matrix and the structural similarity index between the images calculated by the static transformation monitoring algorithm and the velocity vector of each pixel calculated by the dynamic transformation monitoring algorithm.

[0035] Furthermore, the recommended values of parameters C1, C2 and C3 in step S32 are as follows:

[0036] C1=(K1·L) 2

[0037] C2=(K2·L) 2

[0038] C3=C2 / 2

[0039] Among them, K1 is a small constant, K2 is another small constant, and L is the dynamic range of pixel values.

[0040] Furthermore, the step S35 further includes calculating the displacement S of the slope to be monitored according to the displacement velocity v and the time difference t between the images, and the calculation formula is S=v·t.

[0041] Furthermore, the step S4 specifically includes: if the displacement speed of the slope to be monitored exceeds a preset displacement speed threshold, determining that a landslide disaster has occurred.

[0042] Compared to existing technologies, the monocular camera-based method for dynamic and static landslide monitoring provided by this invention uses a monocular camera to periodically collect image data of the slope to be monitored. A convolutional neural network model is then established using a convolutional neural network algorithm to classify the image data. The classified usable data, i.e., image data free of cloud and fog noise, undergoes pre-processing operations such as image alignment, image enhancement, and histogram equalization. Static change monitoring is then performed using a combination of image difference and structural similarity indices, while dynamic change monitoring is performed using a triple optical flow algorithm. This method monitors the displacement velocity of the slope and thus determines whether a landslide has occurred. This method can effectively improve existing landslide monitoring capabilities, reduce losses caused by landslides, and better protect the lives and property of people in landslide-affected areas. It is therefore suitable for widespread engineering application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1It is a flow chart of the method for dynamic and static landslide monitoring based on a monocular camera provided by the present invention.

[0044] Figure 2 3 is a diagram showing the accuracy of the image classification training set and the verification set in the embodiment provided by the present invention.

[0045] Figure 3 It is the pixel value histogram of the original image data and the pixel value histogram after histogram equalization processing in the embodiment provided by the present invention. DETAILED DESCRIPTION

[0046] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the present invention is further described below with reference to specific illustrations.

[0047] Please refer to Figure 1 As shown, the present invention provides a method for dynamic and static monitoring of landslides based on a monocular camera, comprising the following steps:

[0048] S1. Collect image data of the slope to be monitored as a data set for the convolutional neural network model, and use the convolutional neural network model to divide and select image data without noise influence from the data set.

[0049] As a specific embodiment, the step S1 specifically includes:

[0050] S11. Use a monocular camera to periodically collect image data of the slope to be monitored and establish a dataset. In a specific embodiment, the monocular camera can be fixed in front of the slope to be monitored and collect images every hour. No human intervention is required during the collection process. However, the collected images may be affected by noise such as clouds and fog, so it is necessary to establish a convolutional neural network model to automatically distinguish the usability of the image data. To this end, this specific embodiment also includes the steps of:

[0051] S12. Select a convolutional neural network algorithm to establish a convolutional neural network model, use the convolutional neural network model to classify the image data in the data set, and divide the image data into two categories: noise-affected and noise-free. As a specific implementation method, the convolutional neural network model established in step S12 includes a convolutional layer, a pooling layer, a first fully connected layer, a Dropout layer (discarding layer) and a second fully connected layer connected in sequence. The number of convolution kernels in the convolution layer is 32, and the convolution kernel size is 3×3; the pooling kernel size of the pooling layer is 2×2, that is, a 2×2 pooling window is used for maximum pooling operation, and the two-dimensional feature map is expanded into a one-dimensional vector and then input into the first fully connected layer; the number of neurons in the first fully connected layer is 256; the dropout layer has a dropout rate of 0.25 to reduce the risk of overfitting; the number of neurons in the second fully connected layer is 2. The accuracy of the validation set is used as the criterion for judging the prediction accuracy. The classification accuracy of the convolutional neural network model can reach 0.95 on the validation set. Specifically, the accuracy of the training set and validation set of the convolutional neural network model in this embodiment is as follows: Figure 2 shown.

[0052] S2. Pre-processing the image data without noise influence, specifically including the following steps:

[0053] S21. Since the monocular camera is used for outdoor shooting, the influence of the external environment (such as wind) may cause the camera to shake, resulting in distortion and rotation of the collected image data. Therefore, a set of feature points must be detected in one image and matched with the feature points in another image. Then, based on the characteristics of the mutual matching between the images, a transformation matrix between the images is calculated, and the image conversion is performed based on the transformation matrix, mapping one image to another image to achieve mutual alignment between the images. In addition, in order to better realize subsequent change monitoring, it is necessary to perform image enhancement operations on the data. To this end, the step S2 specifically includes the following steps:

[0054] S22, first normalizing the image pixels so that the image pixel range is [0, 1], and then performing intensity stretching to map the image pixel range to [0, 255] to enhance contrast and improve image quality;

[0055] S23, then perform histogram equalization on the image data, that is, widen the grayscale values of pixels with a large number of pixels and merge the grayscale values of pixels with a small number of pixels, so as to make the image clear and achieve image enhancement. The pixel value histogram of the original image and the pixel value histogram after histogram equalization in this embodiment is as follows: Figure 3 Since only the slope area needs to be focused on when monitoring dynamic and static changes, it is necessary to perform mask processing on the image data to extract the area of interest.

[0056] S3. Perform static change monitoring based on image difference and structural similarity index and dynamic change monitoring based on triple optical flow algorithm on the image data after pre-processing operation, and calculate the displacement speed of the image data of the slope to be monitored.

[0057] As a specific embodiment, step S3 specifically includes:

[0058] S31. To reduce the impact of sunlight intensity and angle on images collected at different times, this embodiment uses image data collected at 1 p.m. each day for comparative analysis. Pixel-to-pixel subtraction is performed on the images at a given time t (e.g., two days before and after) to obtain the image difference. For a specific date Im1t and a different date Im2t, the image difference is calculated as follows:

[0059] Image Difference=Im1t-Im2t

[0060] S32, the structural similarity index measures the similarity of images based on a combination of three independent elements: brightness value, contrast value, and regional structure. Assuming that x and y are the positions of two regions of the same size in an image, the similarity of brightness value is l(x, y), the similarity of contrast is c(x, y), and the similarity of regional structure is s(x, y), the calculation formula of the structural similarity index is as follows:

[0061]

[0062] Among them, μ x and μ y are the local sample means of regions x and y, σ x and σ y are the local standard deviations of the x and y regions, σ xy is the sample cross-correlation of x and y after removing the average of the x and y regions. C1, C2, and C3 are small positive constants. The structural similarity index map is generated using a sliding window that moves pixel by pixel across the entire image. A single structural similarity score is calculated for the entire image by averaging the structural similarity index values across the entire image to determine the similarity between images. Alternatively, the mean square error method can be used to measure image quality similarity.

[0063] S33. Use the optical flow algorithm to monitor the dynamic changes of the target pixel in the image caused by the movement of the object in the continuous frame image. The most important thing in the optical flow rate is to assume that the brightness of the image is constant, that is, the position of the object can change in a short time, but the reflectivity and illumination will remain unchanged. The optical flow equation includes:

[0064] I(x,y,t)=I(x+△x,y+△y,t+△t)

[0065] Where I(x,y,t) is the image intensity at position (x,y) at time t, and the change in position is represented by the change in Δx and Δy at time Δt. After applying Taylor expansion on the right side of the optical flow equation, the following equation is obtained:

[0066]

[0067] By solving this equation, we can get the optical flow constraint equation:

[0068] I x u+I y ·v+I t =0

[0069] Among them, I x , I y , I t It is the partial derivative of the pixel grayscale along the x, y, and t directions, and u and v are the velocity vectors of the optical flow along the x and y directions respectively.

[0070] In particular, to better compensate for longer time intervals, this embodiment calculates optical flow for triplets of frames rather than pairs of images. For example, {image n, image n+1 and image n, image n+2} form the first triplet, {image n+1, image n+2 and image n+1, image n+3} form the second triplet, and so on for the remaining image sequences. The optical flow is calculated from each triplet and then stacked to produce Δx / Δt in the horizontal direction and Δy / Δt in the vertical direction. Because the calculation involves more image data, the stacking process can calculate more accurate optical flow motion.

[0071] S34. After the above steps S31 to S33, a velocity vector is assigned to each pixel in the image, thus forming a motion vector field. At a specific moment, the points on the image correspond one-to-one with the points on the actual slope. This correspondence can be calculated through projection, based on which the actual displacement velocity of each point on the slope can be obtained.

[0072] S35. Map the displacement velocity v of the slope to be monitored using the image difference matrix (image difference) and structural similarity index between the images calculated by the static transformation monitoring algorithm and the velocity vector of each pixel calculated by the dynamic transformation monitoring algorithm.

[0073] In order to avoid numerical instability when calculating sample mean, variance and correlation, this method introduces small positive constants C1, C2 and C3. As a specific embodiment, the recommended values of parameters C1, C2 and C3 in step S32 are as follows:

[0074] C1=(K1·L) 2

[0075] C2=(K2·L) 2

[0076] C3=C2 / 2

[0077] Here, K1 is a small constant, such as 0.01; K2 is another small constant, such as 0.03; and L is the dynamic range of pixel values, such as 255 or 1. These constants can be fine-tuned based on actual application and requirements. Selecting a smaller constant helps make the SSIM metric more sensitive to differences in human perception, but too small a constant may lead to computational instability.

[0078] S4. Determine whether a landslide disaster has occurred based on the displacement calculated from the dynamic and static change monitoring, specifically including: if the displacement speed of the slope to be monitored exceeds a preset displacement speed threshold, determine that a landslide disaster has occurred.

[0079] As a specific embodiment, step S35 further includes calculating the displacement S of the monitored slope based on the displacement velocity v and the time difference t between the images, using the formula S = v·t. Correspondingly, step S4 may also determine whether a landslide disaster has occurred based on the displacement S, specifically including determining that a landslide disaster has occurred if the displacement of the monitored slope exceeds a preset displacement threshold.

[0080] Compared to existing technologies, the monocular camera-based method for dynamic and static landslide monitoring provided by this invention uses a monocular camera to periodically collect image data of the slope to be monitored. A convolutional neural network model is then established using a convolutional neural network algorithm to classify the image data. The classified usable data, i.e., image data free of cloud and fog noise, undergoes pre-processing operations such as image alignment, image enhancement, and histogram equalization. Static change monitoring is then performed using a combination of image difference and structural similarity indices, while dynamic change monitoring is performed using a triple optical flow algorithm. This method monitors the displacement velocity of the slope and thus determines whether a landslide has occurred. This method can effectively improve existing landslide monitoring capabilities, reduce losses caused by landslides, and better protect the lives and property of people in landslide-affected areas. It is therefore suitable for widespread engineering application.

[0081] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not limiting. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the purpose and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A method for dynamic and static monitoring of landslides based on a monocular camera, characterized in that: The following steps are involved: S1. Collect image data of the slope to be monitored as a data set for a convolutional neural network model, and use the convolutional neural network model to segment and select image data without noise influence from the data set; S2, performing pre-processing operations on the image data without noise influence; S3, performing static change monitoring based on image difference and structural similarity index and dynamic change monitoring based on triple optical flow algorithm on the image data after pre-processing operation, and calculating the displacement speed of the image data of the slope to be monitored; S4. Determine whether a landslide disaster has occurred based on the displacement velocity calculated by dynamic and static change monitoring; Wherein, the step S3 specifically includes: S31. Perform pixel-to-pixel value subtraction on the image at a given time t. For a specific date Im1t and a different date Im2t, the image difference is calculated as follows: Image Difference=Im1t-Im2t S32, the structural similarity index measures the similarity of images based on a combination of three independent elements: brightness value, contrast value, and regional structure. Let x and y be the positions of two regions of the same size in the image, the similarity of brightness value is l(x, y0), the similarity of contrast is c(x, y), and the similarity of regional structure is s(x, y). The calculation formula of the structural similarity index is as follows: Among them, μ x and μ y are the local sample means of regions x and y, σ x and σ y are the local standard deviations of the x and y regions, σ xy is the sample cross-correlation of x and y after removing the mean of the regions x and y, C1, C2, and C3 are small positive constants; S33. Use the optical flow algorithm to monitor the dynamic changes of the target pixel in the image caused by the movement of the object in the continuous frame image. The most important thing in the optical flow rate is to assume that the brightness of the image is constant, that is, the position of the object can change in a short time, but the reflectivity and illumination will remain unchanged. The optical flow equation includes: I(x,y,t)=I(x+Δx,y+Δy,t+Δt) Where I(x,y,t) is the image intensity at position (x,y) at time t, and the change in position is represented by Δx and Δy when the time change Δt occurs. After applying Taylor expansion to the right side of the optical flow equation, the following equation is obtained: By solving this equation, we can get the optical flow constraint equation: I x ·u+I y ·v+I t =0 Among them, I x , I y , I t is the partial derivative of the pixel grayscale along the x, y, and t directions, and u and v are the velocity vectors of the optical flow along the x and y directions respectively; S34. After the above steps S31 to S33, a velocity vector is assigned to each pixel in the image, thus forming a motion vector field. At a certain moment, the points on the image correspond one-to-one with the points on the actual slope. This correspondence can be calculated by projection, and the actual displacement velocity of each point on the slope can be obtained based on this. S35. Map the displacement velocity v of the slope to be monitored by using the image difference matrix and the structural similarity index between the images calculated by the static transformation monitoring algorithm and the velocity vector of each pixel calculated by the dynamic transformation monitoring algorithm.

2. The method for dynamic and static landslide monitoring based on a monocular camera according to claim 1, characterized in that: The step S1 specifically includes: S11. Use a monocular camera to periodically collect image data of the slope to be monitored and establish a data set; S12. Select a convolutional neural network algorithm to establish a convolutional neural network model, and use the convolutional neural network model to classify the image data in the data set, dividing the image data into two categories: those with noise influence and those without noise influence.

3. The method for dynamic and static landslide monitoring based on a monocular camera according to claim 2, characterized in that: The convolutional neural network model established in step S12 includes a convolutional layer, a pooling layer, a first fully connected layer, a Dropout layer and a second fully connected layer connected sequentially, the number of convolution kernels in the convolution layer is 32, the convolution kernel size is 3×3, the pooling kernel size of the pooling layer is 2×2, the number of neurons in the first fully connected layer is 256, the dropout layer has a dropout rate of 0.25, and the number of neurons in the second fully connected layer is 2.

4. The method for dynamic and static landslide monitoring based on a monocular camera according to claim 1, characterized in that: The step S2 specifically includes: S21. Calculate a transformation matrix between the images based on the mutual matching characteristics between the images, and perform image conversion based on the transformation matrix to map one image to another image to achieve image alignment; S22, normalizing the image pixels and then performing intensity stretching to enhance contrast and improve image quality; S23. Perform a histogram equalization operation on the image data, that is, widen the grayscale values of pixels with a large number of pixels and merge the grayscale values of pixels with a small number of pixels, so as to make the image clear and achieve image enhancement.

5. The method for dynamic and static landslide monitoring based on a monocular camera according to claim 1, characterized in that: The values of parameters C1, C2 and C3 in step S32 are as follows: C1=(K1·L) 2 C2=(K2·L) 2 C3=C2 / 2 Where K1 and K2 are constants, and L is the dynamic range of pixel values.

6. The method for dynamic and static landslide monitoring based on a monocular camera according to claim 1, characterized in that: The step S35 also includes calculating the displacement S of the slope to be monitored according to the displacement velocity v and the time difference t between the images, and the calculation formula is S=v·t.

7. The method for dynamic and static landslide monitoring based on a monocular camera according to claim 1, characterized in that: The step S4 specifically includes: if the displacement speed of the slope to be monitored exceeds a preset displacement speed threshold, determining that a landslide disaster has occurred.

Images