Landslide monitoring method, device, system and electronic equipment based on data fusion

Abstract

The application provides a landslide monitoring method, device and system based on data fusion and an electronic device. The method comprises the following steps: acquiring a slope image and point cloud data; projecting the point cloud data into the slope image; performing grid division on the slope image and the point cloud data projected into the slope image, and extracting features of grid point cloud; comparing features of grid point cloud at different time points, and determining whether a landslide occurs according to displacement of a center point of the grid point cloud. The scheme fuses image data of a slope surface and a slope surface model constructed based on point cloud data, can take the whole slope surface as a monitoring object, and can timely monitor continuous and slight displacement of the slope surface, so that the landslide can be effectively monitored.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a landslide monitoring method, device and system based on data fusion and an electronic device. BACKGROUND

[0002] Landslide is a kind of global natural disaster that occurs frequently. Although landslide disaster occurs almost instantaneously, before the occurrence of landslide disaster, the slope will go through a slow sliding process. In this process, the surface of the slope will generally have obvious displacement. With the accumulation of small displacement over time, the slope becomes more and more unstable. When the deformation of the slope accumulates to a certain extent and is affected by some specific external factors (such as heavy rainfall, earthquake, human activity, etc.), the landslide accident will eventually occur.

[0003] In the past, the prediction of slope disaster mainly relied on simple manual observation. The width of ground fissure and the change of surface feature information were measured by manual ruler, and the change of underground water level and surrounding plants and animals were used for judgment. This method is not only low in efficiency, but also requires high labor cost.

[0004] The existing landslide measurement method, such as monocular and binocular measurement method based on camera, arranges marker points on the slope, detects the position of the marker points using image algorithm, and judges whether landslide will occur by whether the position of the marker points changes.

[0005] The above methods have certain defects. The method of detecting marker points to judge landslide cannot monitor the entire slope, but only monitors the area near the marker points, and the monitoring range is limited.

[0006] The above defects are expected to be overcome by those skilled in the art. SUMMARY

[0007] (I) Technical problems to be solved

[0008] In order to solve the above problems of the prior art, the present application provides a landslide monitoring method, device, system and electronic device based on data fusion, which solves the problem of limited monitoring range of landslide in the prior art based on marker points.

[0009] (II) Technical solutions

[0010] In order to achieve the above purpose, the main technical solutions adopted by the present application include:

[0011] In a first aspect, the present application provides a landslide monitoring method based on data fusion, comprising:

[0012] obtaining slope image and point cloud data;

[0013] projecting the point cloud data into the slope image;

[0014] grid the slope image and the point cloud data projected to the slope image, and extract features of the grid point cloud;

[0015] compare the features of the grid point cloud at different time points, and determine whether a landslide occurs according to the displacement of the center point of the grid point cloud.

[0016] In an exemplary embodiment of the present application, the slope image and the point cloud data are obtained by:

[0017] acquiring point cloud data of a certain integration time based on a laser radar;

[0018] acquiring a slope image based on a camera, and the acquisition time of the slope image is within a time period corresponding to the integration time of the point cloud data.

[0019] In an exemplary embodiment of the present application, before projecting the point cloud data into the slope image, further comprising:

[0020] calibrating the intrinsic parameters and distortion of the camera;

[0021] acquiring the slope image of the camera and the point cloud data of the laser radar, and performing extrinsic parameter calibration;

[0022] changing the scene and verifying the calibration result.

[0023] In an exemplary embodiment of the present application, the feature extraction of the grid point cloud comprises:

[0024] determining M*N grid blocks for the divided grid;

[0025] extracting local features and global features of the point cloud in the grid block, the local feature being the center coordinates of the point cloud in each grid block, and the global feature being the distance from the center coordinates of each divided grid block to the plane.

[0026] In an exemplary embodiment of the present application, the extraction of the local feature comprises:

[0027] selecting any three points in the point cloud in the grid block to form an internal plane;

[0028] calculating the distance of all point clouds in the grid block to the plane, screening out the internal points corresponding to the internal plane, and determining the number of internal points;

[0029] cycling for a preset number W of times to obtain W internal planes and corresponding numbers of internal points;

[0030] selecting the internal plane with the largest number of internal points as the fitting plane, and recording the internal point number of the fitting plane;

[0031] Calculate the average coordinates of all inner points based on the inner point numbering to determine the position of the grid block;

[0032] Remove the points with a distance to the plane greater than a preset value, and calculate the center point of the remaining point cloud in the grid block as the center point of the grid block.

[0033] In an exemplary embodiment of the present application, the extraction of the global feature comprises:

[0034] Perform global plane fitting on the center points of all grid blocks to obtain the plane equation of the entire slope surface;

[0035] Calculate the distance from the center point of each grid block to the plane, where a positive distance indicates a protruding slope surface, and a negative distance indicates a concave slope surface.

[0036] In an exemplary embodiment of the present application, the comparison of the features of the grid point clouds at different time points before and after the comparison, and the determination of whether a landslide has occurred according to the displacement of the center points of the grid point clouds comprises:

[0037] Calculate the displacement according to the distance from the center points of the grid blocks at different time points before and after the comparison to the plane;

[0038] Determine that the grid block has deformed when the displacement is greater than a threshold value;

[0039] Determine whether a landslide has occurred according to the degree of deformation.

[0040] In a second aspect, the present application also provides a landslide monitoring device based on data fusion, comprising:

[0041] A data acquisition module for acquiring slope images and point cloud data;

[0042] A point cloud projection module for projecting the point cloud data into the slope images;

[0043] A feature extraction module for performing grid division on the slope images and the point cloud data projected into the slope images, and extracting the features of the grid point clouds;

[0044] A landslide determination module for comparing the features of the grid point clouds at different time points before and after the comparison, and determining whether a landslide has occurred according to the displacement of the center points of the grid point clouds.

[0045] In a third aspect, the present application also provides a landslide monitoring system based on data fusion, comprising:

[0046] A calibration subsystem, a monitoring subsystem, and a backend control and display subsystem;

[0047] The calibration subsystem is used to calibrate the camera and the laser radar in the monitoring subsystem;

[0048] The monitoring subsystem is configured to collect, fuse and process the slope monitoring data, and perform the method steps described above.

[0049] The back-end control display subsystem is configured to receive the data from the monitoring subsystem, display the data, and send control commands to the monitoring subsystem to control the collection, fusion and processing of the monitoring data.

[0050] In a fourth aspect, the present application provides an electronic device, comprising:

[0051] one or more processors;

[0052] a storage device configured to store one or more programs;

[0053] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described above.

[0054] (Three) beneficial effects

[0055] The beneficial effects of the embodiments of the present application are that a landslide monitoring method, device, system and electronic device based on data fusion are provided, the image data of the slope surface and the slope surface model constructed based on the point cloud data are fused, the entire slope surface is taken as a monitoring object, the continuous small displacement of the slope is monitored in time, the landslide is monitored more widely, the monitoring limitation caused by monitoring only the landmark points is broken, and effective monitoring is achieved. BRIEF DESCRIPTION OF DRAWINGS

[0056] Figure 1 A step flowchart of a landslide monitoring method based on data fusion provided in an embodiment of the present application;

[0057] Figure 2 An implementation process diagram of a landslide monitoring method provided in an embodiment of the present application;

[0058] Figure 3 A mapping table and grid division diagram provided in an embodiment of the present application;

[0059] Figure 4 A diagram of the distance from the center point of the grid block to the plane shown in an embodiment of the present application;

[0060] Figure 5 A composition diagram of a landslide monitoring device based on data fusion provided in another embodiment of the present application;

[0061] Figure 6 A composition diagram of a landslide monitoring system based on data fusion provided in another embodiment of the present application;

[0062] Figure 7 A schematic diagram of a landslide monitoring system based on data fusion provided in another embodiment of the present application is shown in FIG. 1.

[0063] Figure 8 An indoor image collected by the photoelectric instrument;

[0064] Figure 9 An effect diagram of a back-end fusion display for the indoor image; Figure 8

[0065] Figure 10 A slope image collected by the photoelectric instrument for landslide detection in an embodiment of the present application;

[0066] Figure 11 An effect diagram of a back-end fusion display for the slope image; Figure 10

[0067] A schematic diagram of an internal structure of a computer system of an electronic device provided in another embodiment of the present application. Figure 12 DETAILED DESCRIPTION

[0068] In order to better explain the present application and facilitate understanding, the present application is described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] All technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0070] In the related embodiments of the present application, since the visible light camera is used for slope monitoring, all-weather monitoring cannot be performed, and low-light conditions such as foggy days or nights cannot be effectively monitored. The monitoring principle can only monitor the landslide for the marker points, cannot cover the entire slope, the monitoring range is not enough, and the monitoring efficiency and effect are poor.

[0071] In order to improve the landslide monitoring efficiency, the monitoring system needs to continuously and automatically monitor the slope. When the slope has local deformation, the monitoring system can give an early warning and timely inform the professional personnel to confirm, which brings a good early warning reminding effect in the aspect of landslide prevention.

[0072] The embodiments of the present application provide a landslide monitoring method, device, system and equipment based on data fusion, which can automatically monitor the landslide and give an early warning.

[0073] Figure 1 ​​A step flow chart of a landslide monitoring method based on data fusion provided in an embodiment of the present application is shown in Figure 1 The method comprises the following steps:

[0074] In step S110, slope image and point cloud data are acquired.

[0075] In step S120, the point cloud data is projected into the slope image.

[0076] In step S130, the slope image and the projected point cloud data are meshed, and the features of the mesh point cloud are extracted.

[0077] In step S140, the features of the mesh point cloud at different time points are compared, and whether a landslide occurs is determined according to the displacement of the center point of the mesh point cloud.

[0078] Based on the above steps S110-S140, the method provided in the embodiment of the present application can monitor the entire slope surface as a monitoring object by fusing the image data of the slope surface and the slope surface model constructed based on the point cloud data, and can timely monitor the continuous small displacement of the slope surface, so as to more widely monitor the landslide and break through the monitoring limitation caused by monitoring only the landmark point, thereby achieving effective monitoring.

[0079] The specific steps of the landslide monitoring method based on data fusion are described below in combination with embodiments.

[0080] As shown in Figure 1 In step S110, slope image and point cloud data are acquired.

[0081] In an exemplary embodiment of the present application, the point cloud data of a certain integration time is acquired based on the laser radar in this step; the slope image is acquired based on the camera, and the collection time of the slope image is within the time period corresponding to the integration time of the point cloud data. For example, when processing, a point cloud integrated for a certain time (for example, 30s) and an image are input, and the shooting time of the image should be within the integration time of the point cloud.

[0082] As shown in Figure 1 In step S120, the point cloud data is projected into the slope image.

[0083] In an exemplary embodiment of the present application, before the point cloud data is projected into the slope image, the step of system parameter calibration is further included, which specifically comprises:

[0084] The internal parameters and distortion of the camera are calibrated;

[0085] The slope image of the camera and the point cloud data of the laser radar are acquired, and the external parameters are calibrated;

[0086] Change the scenario and verify the calibration results.

[0087] After calibration is completed in a scene, in order to verify the quality of the calibration results, the monitored photoelectric instrument can be moved from one scene to another to acquire camera images and LiDAR point clouds. Then, the point cloud is colored. If the coloring effect is the same as the actual scene, the calibration is considered successful and accurate.

[0088] like Figure 1 As shown, in step S130, the slope image and the point cloud data projected onto the slope image are divided into grids, and the features of the grid point cloud are extracted.

[0089] In one exemplary embodiment of this application, the features of extracting the mesh point cloud include:

[0090] For each grid segment, determine M*N grid blocks;

[0091] Local and global features of point clouds within a grid block are extracted. The local features are the center coordinates of the point cloud within each grid block, and the global features are the distances from the center coordinates of each grid block to the plane.

[0092] In one exemplary embodiment of this application, the extraction of the local features includes:

[0093] Take any three points from the point cloud within the grid block to form an internal plane;

[0094] Calculate the distance from all point clouds within the grid block to the plane, filter out the interior points corresponding to the internal plane, and determine the number of interior points;

[0095] The loop is repeated a preset number of times W to obtain W internal planes and the corresponding number of internal points;

[0096] Select the interior plane with the most interior points as the fitting plane, and record the interior point numbers of the fitting plane.

[0097] Calculate the average coordinates of all interior points based on their interior point numbers to determine the location of the grid block;

[0098] Points with a distance greater than a preset value from the fitted plane are removed, and the center point of the remaining point cloud within the grid block is calculated and used as the center point of the grid block.

[0099] In the above steps, points that are too far from the fitting plane are removed by filtering. Here, "points that are too far from the fitting plane" means points that are more than a certain preset value away from the fitting plane. The preset value can be selected as 0.3 meters or 0.5 meters as needed.

[0100] In one exemplary embodiment of this application, the extraction of the global features includes:

[0101] A global plane fitting is performed on the center points of all the grid blocks to obtain a plane equation of the entire slope surface.

[0102] A distance of each grid block center point to the plane is calculated, and the distance is positive at a protruding slope surface and negative at a recessed slope surface.

[0103] As shown in FIG. 1, in step S140, the features of the grid point clouds at different time points are compared, and whether a landslide occurs is determined according to the displacement of the center points of the grid point clouds. Figure 1

[0104] In an exemplary embodiment of the present application, the comparison of the features of the grid point clouds at different time points and the determination of whether a landslide occurs according to the displacement of the center points of the grid point clouds include:

[0105] The displacement is calculated according to the distance of the center points of the grid blocks at different time points to the plane.

[0106] When the displacement is greater than a threshold value, it is determined that the grid block is deformed.

[0107] Whether a landslide occurs is determined according to the degree of deformation.

[0108] Thus, the landslide monitoring is completed based on the above steps.

[0109] Figure 2 As shown in FIG. 2, an implementation process diagram of the landslide monitoring method provided in an embodiment of the present application includes a preprocessing stage and a monitoring stage, and specifically includes: Figure 2 In the preprocessing stage T0, the calibration of the camera intrinsic parameters and extrinsic parameters, and the collection of the slope image and point cloud data are completed.

[0110] In the monitoring stage T1, the processing of the data is performed to determine the monitoring result, which mainly includes the following steps:

[0111] Step S201, projecting the point cloud data into the slope image to generate a mapping table. Based on the intrinsic parameters of the camera and the extrinsic parameters of the camera relative to the laser radar, the radar is first converted into the camera coordinate system, and then the point clouds located in front of the camera and within the field of view angle of the camera are selected in the camera coordinate system based on the field of view angle of the camera. Then, the point cloud data is projected onto the plane of the slope image to obtain a mapping table of the pixel points on the image to the laser radar points.

[0112] Step S202, image grid division. The slope image is divided into M*N grid blocks in the horizontal and vertical directions. At the same time, the point cloud data projected onto the slope image is also divided into M*N blocks.

[0113]

[0114] Figure 3 ​​As shown in the mapping table and grid partitioning schematic diagram provided in the embodiments of the present application, a 12*8 mapping table is taken as an example and is divided into 3*2 grid blocks. Figure 3 P1-P14 in the mapping table represent radar point clouds projected into image pixels. Figure 3 P1-P14 in the mapping table represent radar point clouds projected into image pixels.

[0115] Next, the point cloud data in the grid block also needs to be denoised. In each grid block, the distance of other points to each point cloud point (i.e., the point of the point cloud data) is calculated, and the number of distances less than a certain threshold is counted. For example, the distance threshold can be 2 meters, and if the number is less than a certain threshold, for example, the number threshold can be 10, it means that there are not many point clouds near the point, and it may be a noise point, which needs to be excluded to achieve denoising.

[0116] Step S203, grid point cloud feature extraction, extracting local features and global features of the grid block. The local feature refers to the center coordinates of the point cloud in each grid block, and the global feature refers to the distance of these center coordinates to the global plane.

[0117] Step A. First, calculate the local center point coordinates in each grid block. In each grid block, use the RANSAC plane fitting method to fit the local plane of the point cloud in the grid, remove the points far from the plane, and then calculate the center point of the remaining point cloud. As shown in FIG. 8, the local plane fitting of the first grid block is to fit a plane using P1, P2, P3, and P4. The specific method is as follows: Figure 3

[0118] a) For the point cloud in the grid block, randomly take three points to calculate an internal plane, then calculate the distance of all point clouds in the grid to the internal plane, and if a distance is less than a predetermined threshold (for example, 0.3 meters), the point is considered as an internal point. After determining the internal point, record the number and number of internal points.

[0119] b) Repeat step a) 100 times, and select the one with the most internal points as the final fitting result to obtain the fitting plane and internal point number.

[0120] c) Based on the number of internal points, calculate the average coordinates of all internal point clouds, which can represent the position of the current grid block. Using the RANSAC method to fit the plane can effectively reduce the influence of noise points on the plane fitting, and then the point clouds belonging to the slope plane in the grid block can be screened out.

[0121] ​Step B. Calculate the distance from the center point of the grid block to the slope plane. In order to show whether the center point of the grid block is below or above the plane, the distance also needs to be signed, that is, the distance of the part protruding from the slope plane is positive, and the distance of the part recessed from the slope plane is negative. The specific method is as follows:

[0122] For the mapping table in Figure 3 , a total of 6 grid blocks are divided, and a center point is calculated for each grid block. The 6 center points are used to fit a plane, and the distance from the 6 points to the plane is calculated.

[0123] a) Perform global plane fitting using all grid point cloud center points, and use the RANSAC method to fit the plane to obtain the plane equation of the entire slope. Assume that the plane equation is:

[0124] ax+by+cz+d=0,

[0125] In order to ensure that the normal direction points to the side of the laser radar, a>0 needs to be ensured. When the calculated plane coefficient a<0, multiply the four coefficients a, b, c, and d of the plane by -1.

[0126] b) Calculate the signed distance from each grid center point to the plane. For the center point of each grid block, first calculate the projection point of the center point on the plane, and then calculate the distance from the center point of the grid block to the projection point. The distance is positive. Then calculate the vector from the projection point to the center point of the grid block. If the inner product of the vector and the normal vector [a, b, c] of the plane is greater than 0, it means that the center point of the grid block is above the plane, and the final signed distance is positive. If the inner product is less than 0, it means that it is below the plane, and the final signed distance is negative. Figure 4 The schematic diagram of the distance from the center point of the grid block to the plane in an embodiment of the present application is shown in Figure 4 , and the center point with the normal vector pointing in the same direction has a distance greater than 0, and the center point with the normal vector pointing in the opposite direction has a distance less than 0.

[0127] c) Send the image and the slope grid features to the back end for display.

[0128] Step S204, landslide detection. In the adjacent two frames, the displacement of the center point of the grid block is calculated, and if the displacement is greater than the set distance threshold, deformation may have occurred in the grid. Further, two threshold values can be set to distinguish different levels of detection structure or crisis degree. For example, the low threshold can be set to 0.1m, and the high threshold can be set to 0.2m. Two landslide detections are performed, one using the low threshold and the other using the high threshold. If deformation is detected by the low threshold, an orange warning is issued; if deformation is detected by the high threshold, a red warning is issued.

[0129] To prevent false positives, multiple adjacent grid blocks need to be deformed to report an alarm, the specific method is as follows: generate an M*N black mask image, if the displacement of a block is greater than the threshold, then the position in the mask image is set to white. When all the grid blocks are calculated, the white part in the mask image is the displacement area, use the function connectedComponmentsWithStats() of opencv to analyze the connected domain of the mask image, this function will separate the white blocks connected together, if the area of the connected domain is greater than a certain threshold, for example 3, it means that there is a continuous area of more than 3 grid blocks deformed, then it is considered that these areas may have landslides.

[0130] Through the above, the landslide monitoring method based on data fusion provided by the application, by using a camera and a laser radar to respectively acquire a slope image and point cloud data, then performing grid division, extracting features in the grid point cloud, and judging whether there is a landslide based on the features at different times. Different times can be two adjacent frames, which can be used to detect instantaneous landslides, or can be compared with data one week ago and one month ago to detect the continuous small displacement of the slope. These small displacements are not obvious in a short time, but can be detected in a long time.

[0131] The application also provides a landslide monitoring device based on data fusion, Figure 5 For the composition schematic diagram of the landslide monitoring device based on data fusion provided in another embodiment of the application, as Figure 5 shown, the landslide monitoring device 500 comprises a data acquisition module 510, a point cloud projection module 520, a feature extraction module 530 and a landslide judgment module 540.

[0132] The data acquisition module 510 is used to acquire a slope image and point cloud data; the point cloud projection module 520 is used to project the point cloud data into the slope image; the feature extraction module 530 is used to perform grid division on the slope image and the point cloud data projected into the slope image, and extract the features of the grid point cloud; and the landslide judgment module 540 is used to compare the features of the grid point cloud at different times before and after, and judge whether there is a landslide according to the displacement of the center point of the grid point cloud.

[0133] The application also provides a landslide monitoring system based on data fusion, Figure 6 For the composition schematic diagram of the landslide monitoring system based on data fusion provided in another embodiment of the application, as Figure 6 shown, the landslide monitoring system 600 comprises a calibration subsystem 610, a monitoring subsystem 620 and a backend control display subsystem 630.

[0134] The calibration subsystem 610 is used for calibrating the camera and the laser radar in the monitoring subsystem; the monitoring subsystem 620 is used for collecting, data fusion and processing of the slope monitoring data, and performs the steps of the above data fusion based landslide monitoring method; the backend control display subsystem 630 is used for receiving the data of the monitoring subsystem and displaying the data, and is also used for sending control commands to the monitoring subsystem to control the collection, data fusion and processing of the monitoring data.

[0135] Figure 7 A schematic diagram of a data fusion based landslide monitoring system provided in another embodiment of the present application is shown, wherein 1 is a calibration system, 2 is a landslide monitoring photoelectric instrument, 3 is a backend control display system, 4 is a camera, 5 is a laser radar, 6 is a data processing board, and 7 is an audible and visual alarm

[0136] As shown in Figure 7 , the landslide monitoring system mainly consists of three parts, a calibration system 1 of a camera and a laser radar, a front end landslide monitoring photoelectric instrument 2 and a backend control display system 3. The calibration system 1 is responsible for calibrating the intrinsic parameters of the camera, distortion and the relative relationship between the camera and the laser radar. The landslide monitoring photoelectric instrument 2 is responsible for collecting, data fusion and processing of the slope data, sending of alarm signals and the like. The backend control display system 3 is responsible for sending control commands to the landslide monitoring photoelectric instrument and receiving the data of the landslide monitoring photoelectric instrument and displaying the data.

[0137] The calibration system 1 directly burns the obtained calibration file into the landslide monitoring photoelectric instrument 2, and the internal algorithm automatically reads the calibration information. The detection results of the landslide monitoring photoelectric instrument 2 can be sent to the backend control display system 3 through a network port or 4G / 5G signals.

[0138] Inside the landslide monitoring photoelectric instrument 2, there are a visible light camera 4, a laser radar 5, a data processing board 6 and an audible and visual alarm 7 and the like. The camera 4 is used for shooting color images of the slope, the laser radar 5 is used for three-dimensional imaging of the slope to obtain shape information of the slope. The data processing board 6 is used for receiving the data of the camera 4 and the laser radar 5 and performing calculation to obtain the features of the slope and the landslide results. If it is judged that a landslide occurs, the audible and visual alarm 7 is controlled to alarm and prompt.

[0139] The working process of the landslide monitoring system is as follows:

[0140] (1) System parameter calibration of the landslide monitoring photoelectric instrument.

[0141] The camera calibration is performed after the landslide monitoring photoelectric instrument is assembled, and the parameters that need to be calibrated include the intrinsic parameters of the camera, distortion, and the extrinsic parameters of the camera relative to the laser radar. The intrinsic parameters and distortion are calibrated using a checkerboard, shooting images of the calibration board at different positions and attitudes, and using Zhang Zhengyou calibration method, the intrinsic parameters and distortion of the camera can be calibrated.

[0142] The extrinsic calibration needs to acquire the data of the camera and the lidar at the same time. First, the lidar is used to acquire the point cloud in a certain time in a feature-rich static scene, and the integration time can be controlled at about 30 seconds. Then, the camera is used to shoot an image in the scene. The same-named points of the image and the point cloud are extracted using a feature extraction algorithm, and the extrinsic parameters of the camera are calculated using the PnP (Perspective-n-Point) algorithm. As long as the relative relationship between the camera and the lidar is not changed, it is not necessary to recalibrate in other scenes.

[0143] After the calibration is completed, in order to verify the good or bad of the calibration result, the optoelectronic instrument can be moved to another scene to acquire the camera image and the lidar point cloud, and then the point cloud is colored. If the coloring effect is the same as the actual scene, it is considered that the calibration is successful and accurate.

[0144] (2) Landslide monitoring

[0145] The monitoring process refers to the method of the above-mentioned embodiment Figures 1-4 , which will not be repeated here.

[0146] (3) Slope data fusion display

[0147] The back-end control display system 3 will receive the image, slope grid features and landslide area and other information sent by the landslide detection optoelectronic instrument 2, and the back-end will fuse and display these data.

[0148] The specific display method is as follows: first, generate M*N points in the OXY plane in the three-dimensional coordinate system, and the layout mode is consistent with the grid division in the landslide detection. Then read the distance from the grid center point in the point cloud feature to the plane as the height value of this point. In this way, M*N three-dimensional points can be obtained, and a grid surface can be formed by grouping these points, so that the surface shape of the slope can be displayed. The color image can be projected into the three-dimensional grid surface, so that the shape and color of the slope can be displayed at the same time. Further, the landslide area can be drawn in the three-dimensional coordinates.

[0149] Figure 8 is an indoor image collected by the optoelectronic instrument, Figure 9 is an effect diagram for the back-end fusion display of the indoor image, Figure 8 which can reflect the concave and convex of the object, Figure 9 in which the three-dimensional coordinates are length, width and height respectively, and the units are cm, Figure 9 as shown in the figure, the longitudinal coordinate represents the height, and the horizontal plane coordinates are length and width respectively.

[0150] applied in the embodiments of the present application, Figure 10 is a slope image collected by the landslide detection optoelectronic instrument in the embodiments of the present application, Figure 11 is an effect diagram for the back-end fusion display of the slope imageFigure 10 The slope images are then fused and displayed at the back end. (This is in contrast to the previous sentence about image fusion.) Figure 9 similar, Figure 11 The three-dimensional coordinates are length, width, and height, all in meters. Figure 11 As shown in the diagram, the vertical axis represents height, and the horizontal coordinates represent length and width, respectively. Figure 11 As shown, the scene depicts a sidewalk beside a highway and a nearby slope. The fused rendering correctly displays the undulations and depressions on the slope. Since the area near the sidewalk is outside the LiDAR's field of vision, its actual location and distance from the slope surface are unknown; therefore, its height is uniformly set to 0. The fused rendering can also utilize color to clearly indicate the level of danger, for example, displaying an orange warning effect. If a certain area has undergone deformation, that area will be highlighted in the fused rendering.

[0151] In summary, the slope monitoring system of this application provides a landslide monitoring photoelectric instrument to achieve full-area and all-weather monitoring of slopes. It also provides a landslide monitoring method for automatic detection and early warning of landslides. Furthermore, it provides a data fusion and display method to integrate slope features into a two-dimensional image for three-dimensional visualization. Based on this system, continuous slope monitoring is possible, covering the entire camera field of view, not just monitoring a few marked local locations. The landslide monitoring system can model the slope surface and fuse image data with the slope surface model to generate a visualized fusion result in a three-dimensional coordinate system. During landslide monitoring, calculations can be completed on the data processing board inside the landslide detection photoelectric instrument, eliminating the need to transmit raw data to the backend. It can also independently detect landslides and issue alarms in areas with low or no signal.

[0152] The following is for reference. Figure 12 It shows a schematic diagram of the structure of a computer system 400 suitable for implementing electronic devices according to embodiments of the present application. Figure 12 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0153] like Figure 12 As shown, the computer system 400 includes a central processing unit (CPU) 401, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 402 or programs loaded from storage section 408 into random access memory (RAM) 403. The RAM 403 also stores various programs and data required for the operation of the system 400. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0154] The following components are connected to the I / O interface 405: an input part 406 including a keyboard, a mouse, etc.; an output part 407 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage part 408 including a hard disk, etc.; and a communication part 409 including a network interface card such as a LAN card, a modem, etc. The communication part 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as necessary. A removable medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 410 as necessary, so that a computer program read out therefrom is installed in the storage part 408 as necessary.

[0155] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by the communication part 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit (CPU) 401, the above-described functions defined in the system of the present application are executed.

[0156] It should be noted that the storage medium shown in the present application can be a computer readable signal medium or a computer readable medium or any combination of the above two. The computer readable medium may, for example, be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of computer readable medium can include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present application, the computer readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present application, the computer readable signal medium can include a data signal carrying computer readable program code in a baseband or as a part of a carrier wave. Such a propagated data signal can take various forms, including but not limited to electromagnetic signals, optical signals or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable medium that can send, propagate or transmit the program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0157] The flowcharts and block diagrams in the drawings illustrate the possible implementation architectures, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment, or a part of code containing one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially concurrently, or they can be executed in reverse order, depending on the involved functions. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system for implementing the specified functions or operations, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0158] The units described in the embodiments of the present application can be implemented in the form of software or in the form of hardware. The described units can also be arranged in a processor, and in some cases, the names of the units do not constitute a limitation on the units themselves.

[0159] In another aspect, the present application also provides a storage medium, which can be included in the electronic device described in the above embodiments, or can exist independently without being assembled into the electronic device. The storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to include the following method steps:

[0160] Obtain the slope image and the point cloud data;

[0161] Project the point cloud data into the slope image;

[0162] Divide the slope image and the projected point cloud data into grids, and extract the features of the grid point cloud;

[0163] Compare the features of the grid point cloud at different time points, and determine whether a landslide occurs according to the displacement of the center point of the grid point cloud.

[0164] It should be understood that the above description of the specific embodiments of the present application is only for the purpose of illustrating the technical route and characteristics of the present application, and the purpose is to enable those skilled in the art to understand the content of the present application and to implement it, but the present application is not limited to the above specific embodiments. Any changes or modifications made within the scope of the claims of the present application should be covered within the protection scope of the present application.

Claims

1. A landslide monitoring method based on data fusion, characterized in that, include: Acquire slope images and point cloud data; Projecting point cloud data onto slope images; The slope image and the point cloud data projected onto the slope image are divided into grids, and the features of the grid point cloud are extracted. Compare the characteristics of the grid point cloud at different times, and determine whether a landslide has occurred based on the displacement of the center point of the grid point cloud. The features of the extracted grid point cloud include: For each grid segment, determine M*N grid blocks; Local and global features of point clouds within a grid block are extracted; the local features are the center coordinates of the point cloud within each grid block, and the global features are the distances from the center coordinates of each grid block to the plane. Global feature extraction further includes: Global plane fitting is performed on the center points of all grid blocks to obtain the plane equation of the entire slope. Calculate the distance from the center point of each grid block to the projection point on the plane. A positive distance indicates a protruding slope, and a negative distance indicates a concave slope. Calculate the vector pointing from the projection point to the center point of the grid block. If the inner product of the vector and the plane normal vector is greater than 0, the distance is positive; if the inner product is less than 0, the distance is negative.

2. The landslide monitoring method based on data fusion as described in claim 1, characterized in that, The acquisition of slope images and point cloud data includes: Point cloud data is acquired based on lidar over a certain integration time. The slope images are acquired using a camera, and the acquisition time of the slope images falls within the time period corresponding to the integration time of the point cloud data.

3. The landslide monitoring method based on data fusion as described in claim 2, characterized in that, Before projecting point cloud data onto the slope image, the following steps are also included: The camera's intrinsic parameters and distortion are calibrated; Acquire slope images from the camera and point cloud data from the lidar, and perform extrinsic parameter calibration; Change the scenario and verify the calibration results.

4. The landslide monitoring method based on data fusion as described in claim 1, characterized in that, The extraction of local features includes: Take any three points from the point cloud within the grid block to form an internal plane; Calculate the distance from all point clouds within the grid block to the plane, filter out the interior points corresponding to the internal plane, and determine the number of interior points; The loop is repeated a preset number of times W to obtain W internal planes and the corresponding number of internal points; Select the interior plane with the most interior points as the fitting plane, and record the interior point numbers of the fitting plane. Calculate the average coordinates of all interior points based on their interior point numbers to determine the location of the grid block; Points with a distance greater than a preset value from the fitted plane are removed, and the center point of the remaining point cloud within the grid block is calculated and used as the center point of the grid block.

5. The landslide monitoring method based on data fusion as described in claim 4, characterized in that, The comparison of the characteristics of the grid point cloud at different times before and after, and the determination of whether a landslide has occurred based on the displacement of the center point of the grid point cloud, includes: The displacement is calculated based on the distance from the center point of the grid block to the plane at different times. When the displacement exceeds a threshold, it is determined that the mesh block has deformed. Determine whether a landslide has occurred based on the degree of deformation.

6. A landslide monitoring device based on data fusion, characterized in that, include: The data acquisition module is used to acquire slope images and point cloud data; The point cloud projection module is used to project point cloud data onto the slope image; The feature extraction module is used to divide the slope image and the point cloud data projected onto the slope image into grids and extract the features of the grid point cloud. The landslide detection module is used to compare the characteristics of the grid point cloud at different times and determine whether a landslide has occurred based on the displacement of the center point of the grid point cloud. The features of the extracted grid point cloud include: For each grid segment, determine M*N grid blocks; Local and global features of point clouds within a grid block are extracted; the local features are the center coordinates of the point cloud within each grid block, and the global features are the distances from the center coordinates of each grid block to the plane. Global feature extraction further includes: Global plane fitting is performed on the center points of all grid blocks to obtain the plane equation of the entire slope. Calculate the distance from the center point of each grid block to the projection point on the plane. A positive distance indicates a protruding slope, and a negative distance indicates a concave slope. Calculate the vector pointing from the projection point to the center point of the grid block. If the inner product of the vector and the plane normal vector is greater than 0, the distance is positive; if the inner product is less than 0, the distance is negative.

7. A landslide monitoring system based on data fusion, characterized in that, include: Calibration subsystem, monitoring subsystem, and back-end control and display subsystem; The calibration subsystem is used to calibrate the cameras and lidar in the monitoring subsystem; The monitoring subsystem is used for the acquisition, fusion and processing of slope monitoring data, and performs the method described in any one of claims 1-5; The back-end control and display subsystem is used to receive data from the monitoring subsystem and display the data. It is also used to send control commands to the monitoring subsystem to control the acquisition, fusion and processing of monitoring data.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-5.

Images