Dual-modal learning slope risk detection method integrating laser ranging and monitoring images

Abstract

The present invention discloses a dual-modal learning slope risk detection method that integrates laser ranging and surveillance images, and relates to the technical field of slope safety monitoring. The present invention provides a dual-modal learning slope risk detection method that integrates laser ranging and surveillance images. The method first combines surveillance camera image data with three-dimensional position data collected by a single, single-point laser rangefinder to fuse image features with laser ranging features. Laser ranging supplements the third-dimensional features, significantly improving the three-dimensional perception of slopes, providing greater information and stronger recognition capabilities. The method then constructs a dual-modal network and performs multimodal learning to detect the type and area of ​​slope risks, particularly local risk changes. This method provides accurate, real-time early warning services for slope detection systems, reducing the probability of false alarms.

Description

Technical Field

[0001] The present invention relates to the technical field of slope safety monitoring, and in particular to a dual-modal learning slope risk detection method integrating laser ranging and monitoring images. Background Art

[0002] Slopes are common in numerous production and construction sites across my country, such as mines, mountain slopes adjacent to roads and bridges, and slopes at hydropower stations. Slope safety monitoring, primarily used to detect various safety incidents and potential safety hazards occurring on slopes and provide timely warnings, is crucial for ensuring safe production and protecting the environment. With the emergence of more application scenarios, it is necessary not only to simply determine the presence of landslides and rockfalls but also to detect surface cracks, ground swelling, subsidence, collapse, and building deformation. With an increasing number of application scenarios, many tasks require not only classifying the presence of a risk but also locating the area where the risk occurs. In machine learning, the former corresponds to classification tasks, while the latter corresponds to target detection tasks.

[0003] There are a growing number of methods for detecting slopes. Commonly used include surveillance cameras, which use image recognition to determine slope risk. Laser rangefinders are also commonly used, analyzing single-point or multi-point displacement data to determine slope risk. In addition, there are various specialized devices, such as displacement sensors, tilt sensors, and stress sensors. Each of these methods has its own shortcomings when used individually, including poor noise immunity, as follows:

[0004] (1) Manual observation

[0005] The cost is high, and continuous observation is impossible. The measurement work is time-consuming and labor-intensive. More importantly, the personal safety of on-site measurement personnel cannot be guaranteed.

[0006] (2) Surface sensor observation

[0007] For example, installing displacement sensors, tilt sensors, stress sensors, etc. on the surface to collect data in real time has the disadvantages of high cost, inconvenient installation, and the sensors are easily damaged when risks occur on the slope.

[0008] (3) Long-distance observation equipment

[0009] 1) Camera surveillance monitoring

[0010] Remote observation requires human visual assistance, making it difficult to automatically identify risk factors with high precision. This is especially true for small changes, such as cracks and surface soil erosion. Furthermore, surveillance camera recognition is significantly affected by lighting and other factors, such as weather.

[0011] 2) Laser ranging equipment monitoring

[0012] Laser ranging equipment determines slope displacement by scanning intervals of positional changes. This point-by-point approach makes it difficult to detect detailed local changes. Furthermore, due to the complex conditions of the slope's rock, soil, and vegetation, not only does the distance measurement itself contain errors, but the detected points also vary. These two factors combined can lead to a high number of false alarms when simply using data from two scans of the same point. Even with the use of multi-point assessment and three-dimensional slope reconstruction techniques, it is still difficult to detect detailed local changes such as cracks, localized ground swelling, and subsidence. Summary of the Invention

[0013] The technical problem to be solved by the present invention is to provide a dual-modal learning slope risk detection method that integrates laser ranging and monitoring images. By combining the image data of the monitoring camera and the three-dimensional position data collected by a single single-point laser rangefinder, multi-modal learning is performed to detect the type and area of ​​slope risks, especially some local risk changes, to provide accurate and real-time early warning services for the slope detection system and reduce the probability of false alarms.

[0014] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0015] A dual-modal learning slope risk detection method integrating laser ranging and monitoring images includes:

[0016] Install four corner reference point markers in the designated area of ​​the slope so that they can be clearly identified by the camera and serve as the four corner reference points for laser ranging. Use the laser ranging equipment and camera to simultaneously collect data and images of the designated area.

[0017] The interval point data collected by the laser ranging device for the first time is used as the reference data, outliers are excluded using the four corner reference points, and the remaining valid point position data are used to construct a reference three-dimensional slope surface using an interpolation algorithm; the interval point data collected after the first time is used as the monitoring data, and the monitoring three-dimensional slope surface is obtained by the same method, and then the difference between the monitoring point data and the reference three-dimensional slope surface is calculated to obtain a three-dimensional difference slope surface map;

[0018] An affine transformation relationship is established based on the coordinates of the four reference points on the captured image and the three-dimensional positions of the four reference points of the ranging device. Then, using perspective transformation, the points on the three-dimensional difference slope map are projected onto the captured image plane. An interpolation algorithm is used to construct the three-dimensional slope difference corresponding to all pixels on the captured image to obtain a fused and aligned slope difference map.

[0019] Slope risk categories and risk areas are marked on the images captured by the camera for training; a dual-modal network is constructed, wherein the dual-modal network includes a landslide perception network, a first neural network, a first modal fusion network, a second neural network, a second modal fusion network, and a perception fusion network; the inputs of the landslide perception network and the first neural network are both fused and aligned slope difference maps, and the input of the second neural network is a slope capture image; the perception fusion network is used to perform multimodal fusion on the outputs of the first modal fusion network, the second modal fusion network, and the landslide perception network; the training images and the fused and aligned slope difference maps are then used as inputs for training the dual-modal network, and after the training is completed, a dual-modal learning slope risk detection model is obtained;

[0020] During the detection deployment, the camera images and the laser ranging equipment data are obtained respectively. The out-of-region points and outliers are excluded according to the same data processing method as in training. The image to be detected and the fused and aligned slope difference map are obtained. These are input into the dual-modal learning slope risk detection model. After detection and identification, the slope risk category and area are output.

[0021] Furthermore, an affine transformation relationship is established based on the coordinates of the four reference points on the training image and the three-dimensional positions of the four reference points of the ranging device, specifically including:

[0022] Input the xy plane positions (x[i], y[i]) of the four reference points on the laser rangefinder, corresponding to the XY positions (X[i], Y[i]) on the image, and calculate the perspective matrix H. The matrix size is 3×3, with a total of 8 variables. According to the standard solution of the perspective matrix, 8 equations need to be listed:

[0023]

[0024] The equation constructed by a pair of corresponding points is:

[0025]

[0026]

[0027] There are 8 equations in total for the four pairs of points, which can be used to solve the 8 parameters in the matrix H;

[0028] The points on the laser ranging xy plane are transformed to the image plane XY through the above transformation formula, and the corresponding points of the pixels on the image have the difference data of the three-dimensional slope surface.

[0029] Furthermore, the method for constructing the three-dimensional difference slope map includes:

[0030] Step 1: On the XY plane, construct an interpolation point network Tnet in the X-axis interval [Xmin, Xmax] and the Y-axis interval [Ymin, Ymax] every t meters in the XY direction, where t is a set value and t>0;

[0031] Step 2: Input the slope scan point data after removing abnormal points and outliers, project the data points on the XY plane, and construct a Delaunay triangulation network using the Delaunay triangulation algorithm;

[0032] Step 3. Calculate which triangle each Tnet point falls within in the Delaunay triangulation, then use the xyz coordinates of the three points in the triangle to calculate the plane equation z = f(x, y), substitute the xy values ​​of the interpolated Tnet point into z = f(x, y) to calculate the corresponding Z value, and traverse the interpolation to calculate the Z values ​​of all points in Tnet;

[0033] Step 4: Replace the xy values ​​of Tnet with the index coordinates to obtain a new data point set (m, n, z) of the interpolation points.

[0034] Furthermore, the dual-modal network also includes: the landslide perception network includes a slice operation module, a convolution module and three fully connected modules connected in sequence, and the output of the landslide perception network is a perception coefficient; the perception fusion network is a fully connected module, and the first neural network and the second neural network are YOLOv5 networks.

[0035] The technical solutions of the embodiments of the present invention have at least the following technical effects or advantages:

[0036] 1. Fusion of image features and laser ranging features is equivalent to three-dimensional information, with greater information volume and stronger recognition capabilities:

[0037] Landslides, cracks, and deformations on slopes are three-dimensional processes. When mapped onto two-dimensional images captured by cameras, much information is lost, making them difficult to detect. Laser ranging adds a third dimension, significantly improving the three-dimensional perception of slopes. Using appropriate algorithms can enhance the ability to identify various slope risks.

[0038] 2. Automatic feature extraction replaces manual feature design and automatically extracts optimal features based on data, which is more accurate and reliable:

[0039] A significant advantage of deep learning is its data-driven approach, automatically discovering the most effective features for classification and detection. This data-driven, automatic feature extraction method significantly avoids the blindness and randomness inherent in traditional, manually designed feature extraction methods. As the number of slope risk types increases, manual features become increasingly inadequate. Data-driven approaches, on the other hand, can be effective simply by increasing the number of training samples.

[0040] 3. Avoid complex noise processing and avoid relying on experience to adjust various parameters such as thresholds, so the algorithm has better adaptability:

[0041] When using traditional machine learning methods for slope risk classification or regional location, noise interference can severely impact performance. Complex noise filters are often required before feature extraction. These designs are often tied to empirical knowledge and pre-defined scenarios, and can easily fail when these knowledge and pre-defined scenarios are no longer valid in real-world applications.

[0042] Furthermore, various parameters must be designed for both the feature extractor and the final classifier. To simplify the algorithm, various experimental parameter values, or even empirical values, are often used as preset values. The scientificity, effectiveness, and robustness of these parameters remain to be tested.

[0043] The data-driven deep learning method can automatically focus on the most core features of classification, automatically avoid the influence of noise, and does not have too many preset parameter values, which greatly improves the effectiveness of the algorithm.

[0044] 4. For new target detection tasks in new scenarios, simply adding labeled samples and retraining can meet the requirements, improving the scalability of the model:

[0045] A significant advantage of deep learning is its data-driven, automated training of neural networks. For new scenarios, simply adding training samples for that scenario allows for iterative model updates, significantly improving the model's scalability.

[0046] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0048] Figure 1 is a flow chart of a method according to an embodiment of the present invention;

[0049] Figure 2 This is a flowchart of multimodal detection training according to an embodiment of the present invention;

[0050] Figure 3 This is a flow chart of reconstructing a three-dimensional slope image using an interpolation algorithm according to an embodiment of the present invention;

[0051] Figure 4 Schematic diagram of a dual-mode network structure according to an embodiment of the present invention;

[0052] Figure 5 This is a flow chart of multimodal detection and classification according to an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The embodiments of the present invention provide a dual-modal learning slope risk detection method that integrates laser ranging and monitoring images. By combining monitoring camera image data and three-dimensional position data collected by a single single-point laser rangefinder, multimodal learning is performed to detect the type and area of ​​slope risks, especially some local risk changes, providing accurate and real-time early warning services for the slope detection system and reducing the probability of false alarms.

[0054] The technical solution in the embodiment of the present invention has the following general ideas:

[0055] The algorithm has two main processes: training process and classification process. First, collect labeled training samples, align images and ranging multimodal data, and train the model. Figure 2 , and then provided to the detection process.

[0056] The key modules in the training process are the 3D slope map construction method (including the 3D interpolation method), the perspective transformation of the image reference points and the ranging reference points, and the multimodal learning method, which are described below.

[0057] 1. 3D slope map construction method (including 3D interpolation method), such as Figure 3 and as shown in Table 1.

[0058] Table 1 Interpolation point coordinate matrix of XY plane interpolation point network Tnet

[0059]

[0060] Note: The corresponding index coordinates of the matrix are (m, n), where m is an integer between [0, (Xmax-Xmin) / t] and n is an integer between [0, (Ymax-Ymin) / t].

[0061] 2. Perspective transformation of image reference points and ranging reference points:

[0062] The four reference corner points in both sets of data are calibrated. The laser ranging function measures three-dimensional spatial positions, with the xy coordinates corresponding to the ground plane and the z axis corresponding to the point's height relative to the ranging device. To achieve this, simply align the four corner points in the ranging space with the xy plane and the plane in the surveillance image. Then, by interpolating the slope difference information corresponding to each pixel in the image, the image and slope difference map can be aligned and fused. Areas where the slope image changes also show changes in the laser ranging information.

[0063] The method of aligning the image plane and the distance measurement plane of the present invention is to use perspective transformation under the known position information of four reference points. The specific method is as follows:

[0064] Input the xy plane positions (x[i], y[i]) of the four reference points on the laser rangefinder, corresponding to the XY positions (X[i], Y[i]) on the image, and calculate the perspective matrix H. The matrix size is 3×3, with a total of 8 variables. According to the standard solution of the perspective matrix, 8 equations need to be listed:

[0065]

[0066] The equation constructed by a pair of corresponding points is:

[0067]

[0068]

[0069] There are 8 equations in total for the four pairs of points, which can be used to solve the 8 parameters in the matrix H;

[0070] Using the above transformation formula, the points on the laser ranging xy plane are transformed to the image plane xy. The corresponding points on the image pixel contain the difference data of the 3D slope surface. Then, using the Delaunay triangulation interpolation technique described above, the difference data of the 3D slope surface corresponding to all pixels is interpolated. This completes the alignment of the two modal data.

[0071] 3. Model Training

[0072] 1) After the above processing, the fused and aligned data (image, slope difference between image points and corresponding points) is obtained. A large number of paired data samples are collected, and the images are labeled with risk types and areas to prepare training sample data.

[0073] 2) Since the data positions of the two are only roughly aligned, a multimodal data late fusion strategy is used to add a layer of network as a multimodal fusion network after the two-way target detection algorithm structure. The features extracted from the difference slope map reconstructed by the more accurate distance measurement information are used as perception information to adjust the perception network. Figure 4 The bimodal network shown in the figure is trained by a large amount of labeled data. Focus\conv\linear are the slice operation module\convolution module\fully connected module in yolov5 respectively.

[0074] 4. Model Detection

[0075] Deploy the trained model and follow the same data processing method as in training, except that the data annotation step is removed. Figure 5As shown in the figure, pre-processed data is input, and slope risk categories and areas are output after detection and identification.

[0076] A specific embodiment of the present invention is implemented as follows Figure 1 As shown:

[0077] A dual-modal learning slope risk detection method integrating laser ranging and monitoring images includes:

[0078] Four corner reference point markers are installed in the designated area of ​​the slope so that they can be clearly identified on the camera and serve as the four corner reference points for laser ranging. Data and images of the designated area are collected simultaneously by the laser ranging equipment and the camera.

[0079] The interval point data collected by the laser ranging device for the first time is used as the reference data, the four corner reference points are used to exclude outliers, and the remaining valid point position data are used to construct a reference three-dimensional slope surface using an interpolation algorithm; the interval point data collected after the first time is used as the monitoring data, and the monitoring three-dimensional slope surface is obtained by the same method, and then the difference with the reference three-dimensional slope surface is calculated to obtain a three-dimensional difference slope surface map.

[0080] An affine transformation relationship is established based on the coordinates of the four reference points on the acquired image and the three-dimensional positions of the four reference points of the ranging device. Then, using perspective transformation, the points on the three-dimensional difference slope map are projected onto the acquired image plane. The interpolation algorithm is used to construct the three-dimensional slope difference corresponding to all pixel points on the acquired image to obtain a fused and aligned slope difference map.

[0081] Slope risk categories and risk areas are marked on the images captured by the camera for training; Figure 4 As shown, a dual-modal network is constructed, which includes a landslide perception network, a first neural network, a first modal fusion network, a second neural network, a second modal fusion network and a perception fusion network; the inputs of the landslide perception network and the first neural network are both fused and aligned slope difference maps, and the input of the second neural network is a slope acquisition image; the perception fusion network is used to perform multimodal fusion on the output ends of the first modal fusion network, the second modal fusion network and the landslide perception network; then the training image and the fused and aligned slope difference map are used as inputs of the dual-modal network for training, and after the training is completed, a dual-modal learning slope risk detection model is obtained.

[0082] The dual-modal network also includes: the landslide perception network includes a slice operation module (Focus), a convolution module (Conv) and three fully connected modules (Linear) connected in sequence, and the output of the landslide perception network is a perception coefficient; the perception fusion network is a fully connected module, and the first neural network and the second neural network are Yolov5 networks.

[0083] During the detection deployment, the camera images and the laser ranging equipment data are obtained respectively. The out-of-region points and outliers are excluded according to the same data processing method as in training. The image to be detected and the fused and aligned slope difference map are obtained. These are input into the dual-modal learning slope risk detection model. After detection and identification, the slope risk category and area are output.

[0084] In one possible implementation, an affine transformation relationship is established based on the coordinates of four reference points on the training image and the three-dimensional positions of four reference points of the ranging device, specifically including:

[0085] Input the xy plane positions (x[i], y[i]) of the four reference points on the laser rangefinder, corresponding to the XY positions (X[i], Y[i]) on the image, and calculate the perspective matrix H. The matrix size is 3×3, with a total of 8 variables. According to the standard solution of the perspective matrix, 8 equations need to be listed:

[0086]

[0087] The equation constructed by a pair of corresponding points is:

[0088]

[0089]

[0090] There are 8 equations in total for the four pairs of points, which can be used to solve the 8 parameters in the matrix H;

[0091] The points on the laser ranging xy plane are transformed to the image plane XY through the above transformation formula, and the corresponding points of the pixels on the image have the difference data of the three-dimensional slope surface.

[0092] like Figure 3 As shown, the method for constructing the three-dimensional difference slope map includes:

[0093] Step 1: On the XY plane, construct an interpolation point network Tnet in the X-axis interval [Xmin, Xmax] and the Y-axis interval [Ymin, Ymax] every t meters in the XY direction, where t is a set value and t>0;

[0094] Step 2: Input the slope scan point data after removing abnormal points and outliers, project the data points on the XY plane, and construct a Delaunay triangulation network using the Delaunay triangulation algorithm;

[0095] Step 3. Calculate which triangle each Tnet point falls within in the Delaunay triangulation, then use the xyz coordinates of the three points in the triangle to calculate the plane equation z = f(x, y), substitute the xy values ​​of the interpolated Tnet point into z = f(x, y) to calculate the corresponding Z value, and traverse the interpolation to calculate the Z values ​​of all points in Tnet;

[0096] Step 4: Replace the xy values ​​of Tnet with the index coordinates to obtain a new data point set (m, n, z) of the interpolation points.

[0097] The present invention provides a dual-modal learning slope risk detection method that integrates laser ranging and monitoring images. The method first combines monitoring camera image data with three-dimensional position data collected by a single single-point laser rangefinder to fuse image features and laser ranging features. Laser ranging supplements the third-dimensional features, greatly improving the three-dimensional perception of slopes, with greater information content and stronger recognition capabilities. Then, by constructing a dual-modal network and performing multi-modal learning, the method detects the types and areas of slope risks, especially some local risk changes, providing accurate and real-time early warning services for the slope detection system and reducing the probability of false alarms.

[0098] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A dual-modal learning slope risk detection method integrating laser ranging and monitoring images, characterized in that: include: Install four corner reference point markers in the designated area of ​​the slope so that they can be clearly identified by the camera and serve as the four corner reference points for laser ranging. Use the laser ranging equipment and camera to simultaneously collect data and images of the designated area. The interval point data collected by the laser ranging device for the first time is used as the reference data, and the coordinates of the four corner reference points are used to exclude outliers outside the area. The remaining valid point position data are used to construct a reference three-dimensional slope surface using an interpolation algorithm; the interval point data collected after the first time is used as the monitoring data, and the monitoring three-dimensional slope surface is obtained by the same method, and then the difference between the monitoring point data and the reference three-dimensional slope surface is calculated to obtain a three-dimensional difference slope surface map; An affine transformation relationship is established based on the coordinates of the four reference points on the captured image and the three-dimensional positions of the four reference points of the ranging device. Then, using perspective transformation, the points on the three-dimensional difference slope map are projected onto the captured image plane. An interpolation algorithm is used to construct the three-dimensional slope difference corresponding to all pixels on the captured image to obtain a fused and aligned slope difference map. marking slope risk categories and risk areas on images captured by the camera for training; A dual-modal network is constructed, comprising a landslide perception network, a first neural network, a first modal fusion network, a second neural network, a second modal fusion network, and a perception fusion network; the inputs of the landslide perception network and the first neural network are both fused and aligned slope difference maps, and the input of the second neural network is a slope acquisition image; the perception fusion network is used to perform multimodal fusion on the outputs of the first modal fusion network, the second modal fusion network, and the landslide perception network; the training images and the fused and aligned slope difference maps are then used as inputs for training the dual-modal network, and after the training is completed, a dual-modal learning slope risk detection model is obtained; During the detection deployment, the camera images and the laser ranging equipment data are obtained respectively. The out-of-region points and outliers are excluded according to the same data processing method as in training. The image to be detected and the fused and aligned slope difference map are obtained. These are input into the dual-modal learning slope risk detection model. After detection and identification, the slope risk category and area are output.

2. The method according to claim 1, wherein: Establish an affine transformation relationship based on the coordinates of the four reference points on the training image and the three-dimensional positions of the four reference points of the ranging device, specifically including: Enter the xy plane positions of the four reference points on the laser rangefinder ( x [ i ], y [ i ]), corresponding to the XY position on the image ( X [ i ], Y [ i ]), find the perspective matrix H, the matrix size is 3×3, with a total of 8 variables. According to the standard solution of the perspective matrix, 8 equations need to be listed: ; The equation constructed by a pair of corresponding points is: ; There are 8 equations in total for the four pairs of points, which can be used to solve the 8 parameters in the matrix H; The points on the laser ranging xy plane are transformed to the image plane XY through the above transformation formula, and the corresponding points of the pixels on the image have the difference data of the three-dimensional slope surface.

3. The method according to claim 1 or 2, characterized in that: The method for constructing the three-dimensional difference slope map includes: Step 1: On the XY plane, construct an interpolation point network Tnet in the X-axis interval [Xmin, Xmax] and the Y-axis interval [Ymin, Ymax] every t meters in the XY direction, where t is a set value and t>0; Step 2: Input the slope scan point data after removing abnormal points and outliers, project the data points on the XY plane, and construct a Delaunay triangulation network using the Delaunay triangulation algorithm; Step 3. Calculate which triangle each Tnet point falls within in the Delaunay triangulation, and then use the xyz coordinates of the three points in the triangle to calculate the plane equation z=f(x,y). Substitute the xy values ​​of the interpolated Tnet point into z=f(x,y) to calculate the corresponding z value, and traverse the interpolation to calculate the z values ​​of all points in Tnet. Step 4. Replace the xy values ​​of Tnet with index coordinates to obtain a new data point set (m, n, z) of the interpolation point, where m is an integer between [0, (Xmax- Xmin) / t] and n is an integer between [0, (Ymax- Ymin) / t].

4. The method according to claim 1, wherein: The dual-modal network also includes: the landslide perception network includes a slice operation module, a convolution module and three fully connected modules connected in sequence, the output of the landslide perception network is the perception coefficient; the perception fusion network is a fully connected module, and the first neural network and the second neural network are YOLOv5 networks.

Images