Slope safety monitoring system based on multi-sensor fusion

Abstract

The invention discloses a slope safety monitoring system based on multi-sensor fusion. A detection image is obtained by using a synthetic aperture radar; sAR signal images continuously received by a radar twice are selected, and complex conjugate multiplication is carried out to obtain a complex interference pattern; extracting an interference phase of each pixel point in the complex interference pattern; dividing the input image into a plurality of grids; and for each grid, predicting a plurality of bounding boxes through an algorithm, carrying out classification and regression on each bounding box, deleting redundant bounding boxes by using non-maximum suppression, and judging whether the mountain has cracks or not according to output bounding box data. The intelligent landslide monitoring system has all-day and all-weather working capability, can monitor a mountain in real time, find mountain changes in advance, analyze slope displacement in real time through a radar data processing algorithm and an image data processing algorithm, can effectively identify potential landslide risks and send out early warning signals in advance, so that the intelligent landslide monitoring system can be used for monitoring the landslide in real time. Therefore, a precious time window is provided for taking preventive measures.

Description

Technical Field

[0001] The invention belongs to the technical field of mountain slope safety monitoring and early warning, and in particular relates to a slope safety monitoring system based on multi-sensor fusion. Background Art

[0002] In modern infrastructure construction, slope stability is one of the key factors to ensure project safety and environmental protection. With the acceleration of urbanization, a large number of construction projects involve complex terrains such as mountains, hills and river valleys, which makes slope safety monitoring particularly important.

[0003] In the prior art, for example, the technical solution described in the patent with publication number CN116612622A is: a safety monitoring and early warning system for complex high and steep slopes, including a monitoring data acquisition module, a single early warning module and a comprehensive early warning module. The monitoring data acquisition module is used to obtain monitoring data of multiple monitoring indicators for a target slope, the single early warning module determines the single early warning level of each monitoring indicator for the target slope based on the monitoring data of each monitoring indicator, and the comprehensive early warning module determines whether the single early warning levels of all monitoring indicators meet the red early warning condition, orange early warning condition, yellow early warning condition or green early warning condition.

[0004] However, traditional slope monitoring methods often rely on manual inspections and simple measurement tools, which is not only inefficient but also difficult to achieve real-time monitoring and early warning. Therefore, it is urgent to develop an efficient and accurate slope safety monitoring technology. Summary of the invention

[0005] The purpose of the present invention is to provide a slope safety monitoring system based on multi-sensor fusion to solve the problem that the slope monitoring method in the prior art mentioned in the background technology relies on manual inspection and simple measuring tools, is inefficient, and is difficult to achieve real-time monitoring and early warning.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A slope safety monitoring system based on multi-sensor fusion includes the following detection steps:

[0008] Step S1, using synthetic aperture radar to repeatedly move multiple times on the same track to obtain multiple detection images;

[0009] Step S2, selecting two images of SAR signals received by the radar twice in succession, and performing complex conjugate multiplication on each complex pixel point of the two images to obtain a complex interference pattern;

[0010] Step S3, extracting the interference phase of each pixel point in the complex interference pattern, and obtaining the deformation of the monitoring surface through phase unwrapping;

[0011] Step S4, collecting slope images by using a drone, and dividing the slope images collected by the drone into a plurality of grids;

[0012] Step S5, for each grid, predict multiple bounding boxes through an algorithm to obtain the coordinate value and confidence score of each bounding box; wherein the bounding box is used to frame the cracks on the mountain;

[0013] Step S6, classifying and regressing each bounding box, and selecting the bounding box with the highest confidence score by classifying and regressing each bounding box;

[0014] Step S7, using non-maximum suppression to delete redundant bounding boxes and outputting bounding box data;

[0015] Step S8, judging whether the mountain has issued an alarm based on the outputted bounding box data.

[0016] According to the above technical solution, in step S2, the calculation formula for obtaining the complex interference pattern by performing complex conjugate multiplication on each complex pixel point of the two images is:

[0017] (a+bi)*(a-bi)=a 2 -(bi) 2 =a 2 -b 2 i 2 =a 2 +b 2

[0018] In the formula, the size of the complex pixel of the image is a+bi, and the corresponding conjugate complex size is a-bi, and the result is the pixel value of the complex interference pattern.

[0019] According to the above technical solution, in step S3, the deformation amount of the monitoring surface is calculated by the following formula:

[0020]

[0021] In the formula, ΔR is the path difference of the signal, and λ is the wavelength of the electromagnetic wave emitted by the radar.

[0022] According to the above technical solution, in step S4, the slope image collected by the drone is divided into SxS grids, and for each grid, each bounding box includes 4 coordinate values ​​(x, y, w, h) and a confidence score.

[0023] According to the above technical solution, in step S5, predicting multiple bounding boxes through an algorithm is specifically as follows: first, training the neural network weights through data, and then using the trained neural network to predict the bounding box of the input data, and the neural network will output the coordinate value of the target bounding box.

[0024] According to the above technical solution, for each bounding box, a convolutional neural network is used to classify and regress it; wherein, the classification is to determine whether the category to which each bounding box belongs is a crack based on the feature map; the regression is to adjust the coordinate values ​​(x, y, w, h) of the position of each bounding box based on the feature map so that the bounding box frames the crack.

[0025] According to the above technical solution, the specific implementation process of classification and regression is: first, the neural network weights are trained through data, and then the trained neural network is used to predict the input data. The classification task means that the neural network will give a confidence score for each category of the target, and select the category with the largest confidence score as the category of the target; the regression task means that the neural network will output the coordinate value of the target bounding box.

[0026] According to the above technical solution, non-maximum suppression is used to delete redundant bounding boxes in all bounding boxes.

[0027] According to the above technical solution, non-maximum suppression is used to delete redundant bounding boxes as follows: a set H is constructed to store bounding boxes to be processed, which is initialized to contain all N bounding boxes, and then a set M is constructed to store optimal bounding boxes, which is initialized to an empty set; all bounding boxes in set H are sorted according to confidence scores, and the bounding box m with the highest score is selected and moved from set H to set M; then, the boxes in set H are traversed, and the intersection and union ratios are calculated with box m respectively. If the ratio is higher than the threshold, it is considered that this box overlaps with m, and this box is taken out of set H.

[0028] According to the above technical solution, non-maximum suppression is repeatedly used to delete redundant bounding boxes until the set H is empty, and the bounding boxes in the set M are the final output results.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] Compared with the prior art, the system in the present invention is a non-contact monitoring system and does not need to be installed at the disaster site. In addition, the present invention has the ability to work around the clock and in all weather conditions, and can monitor the mountain in real time, detect changes in the mountain in advance, and issue early warnings. The present invention uses radar data processing algorithms and image data processing algorithms to perform real-time analysis of slope displacement, which can effectively identify potential landslide risks and issue early warning signals, thereby providing a valuable time window for taking preventive measures. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a schematic diagram of the structure of the slope safety monitoring system of the present invention; Figure 2 A schematic diagram of a measurement method of the differential interferometry measurement technology of the present invention; Figure 3It is a monitoring flow chart of the slope safety monitoring system of the present invention; Figure 4 It is a schematic diagram of the slope detection result of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Embodiment 1

[0036] like Figure 1 As shown, a slope safety monitoring system based on multi-sensor fusion includes the following detection steps:

[0037] Step S1, using synthetic aperture radar to repeatedly move multiple times on the same track to obtain multiple detection images;

[0038] Step S2, selecting two images of SAR signals received by the radar twice in succession, and performing complex conjugate multiplication on each complex pixel point of the two images to obtain a complex interference pattern;

[0039] Step S3, extracting the interference phase of each pixel point in the complex interference pattern, and obtaining the deformation of the monitoring surface through phase unwrapping;

[0040] Step S4, collecting slope images by using a drone, and dividing the slope images collected by the drone into a plurality of grids;

[0041] Step S5, for each grid, predict multiple bounding boxes through an algorithm to obtain the coordinate value and confidence score of each bounding box; wherein the bounding box is used to frame the cracks on the mountain;

[0042] Step S6, classifying and regressing each bounding box, and selecting the bounding box with the highest confidence score by classifying and regressing each bounding box;

[0043] Step S7, using non-maximum suppression to delete redundant bounding boxes and outputting bounding box data;

[0044] Step S8, judging whether the mountain has issued an alarm based on the outputted bounding box data.

[0045] Compared with the prior art, the system in the present invention is a non-contact monitoring system and does not need to be installed at the disaster site. In addition, the present invention has the ability to work around the clock and in all weather conditions, and can monitor the mountain in real time, detect changes in the mountain in advance, and issue early warnings. The present invention uses radar data processing algorithms and image data processing algorithms to perform real-time analysis of slope displacement, which can effectively identify potential landslide risks and issue early warning signals, thereby providing a valuable time window for taking preventive measures.

[0046] Embodiment 2

[0047] This embodiment provides a specific implementation method.

[0048] First, the slope safety monitoring system analyzes and processes MIMO radar data. The slope monitoring radar uses synthetic aperture radar imaging technology and differential interferometry technology. By comparing the returned microwave information with multiple radar 2D / 3D images obtained under the same conditions, accurate displacement measurement of the surface is achieved. This remote sensing measurement method of surface displacement does not require the location of the target on the ground during field measurement, nor does it require any actual contact with the slope. The following are the technical principles used by the slope monitoring radar:

[0049] In the early days of radar development, the first to appear was the real aperture radar (RAR). According to the radar principle, since the resolution is proportional to the length of the radar antenna, in order to obtain a higher-resolution target image, the resolution needs to be increased by 1 times, and the physical size of the radar antenna needs to be increased by about 6-8 times. However, with the development of technology and actual needs, the resolution of the target image is getting higher and higher, and the detection distance of the target needs to be farther and farther, so the size (caliber) of the radar antenna needs to be continuously increased. However, due to the limitations of the use conditions and the limitations of the radar transportation carrier, the antenna caliber cannot be increased indefinitely, and the development and application of the real aperture radar RAR is limited. Later, it was gradually replaced by the synthetic aperture radar SAR (Synthetic Aperture Radar: SAR). The principle of synthetic aperture radar is to make the antenna that originally only rotates but does not move move in a certain direction. The moving antenna becomes a large virtual antenna (equivalent to the antenna in four-dimensional space) within the entire time range, thereby synthesizing a large-sized radar antenna, thereby improving the detection accuracy. Synthetic aperture radar uses the movement of a small real antenna to equivalently form a long antenna, so that an equivalent large antenna aperture (called synthetic aperture) can be obtained in the direction of movement. In this way, the radar's resolution of the target will be improved. As an active microwave sensor, synthetic aperture radar has the characteristics of realizing all-day and all-weather ground observation without being restricted by lighting and weather conditions.

[0050] According to the principle of synthetic aperture imaging technology, assuming that the bandwidth of the radar system transmitting signal is B and the propagation speed of electromagnetic waves in vacuum is c, the range resolution is:

[0051]

[0052] Where B is the bandwidth of the radar system transmitting signal, c is the propagation speed of electromagnetic waves in vacuum, and δr is the range resolution. When the bandwidth is 300MHz, the maximum range resolution that the system can achieve is δr = 0.5mm. If the wavelength of the radar transmitting electromagnetic wave is λ, and the maximum distance of the radar sliding is L, then the azimuth resolution is:

[0053]

[0054] Where δθ is the azimuth resolution; λ is the wavelength of the electromagnetic wave emitted by the radar; and L is the maximum distance the radar slides.

[0055] SAR images are single-view complex images, and each pixel consists of a complex number, including the amplitude and phase of the echo. The echo amplitude reflects the intensity of microwave reflection from the ground surface, and the phase contains the ranging information from the radar to the ground surface. By comparing the phase difference of the signal reflected from the target at different times, the displacement change of the target can be accurately measured. This is the core content of differential interferometry technology.

[0056] That is, the synthetic aperture radar moves repeatedly on the same track, and the distance between the radar and the monitoring surface is D. The distance between a point P on the monitoring surface and the radar is represented by R. Assuming that the displacement in the slant range direction after the deformation of the monitoring surface is R, the SAR signal s1 image and s2 image received by the radar twice before and after the deformation of the monitoring surface.

[0057] The complex conjugate multiplication of each complex pixel of the two images gives a complex interference pattern. The phase of the interference pattern depends on the path difference ΔR of the signal. The interference phase of each pixel in the interference pattern is extracted, and the deformation of the monitoring surface is obtained through phase unwrapping, such as Figure 2 shown.

[0058] As shown in the following formula, the deformation of the point where the change occurs on the monitoring surface is

[0059]

[0060] In the formula, ΔR is the path difference of the signal, and λ is the wavelength of the electromagnetic wave emitted by the radar.

[0061] By processing radar data, it is possible to locate the locations where large deformations and displacements occur at a coarse granularity. Then the drone is instructed to go to the scene to take images for further processing and analysis.

[0062] Next, the slope safety monitoring system analyzes and processes the drone image data. The present invention uses a convolutional neural network-based target detection model to detect cracks and further monitor on-site safety. The present invention performs crack detection based on the mainstream visual detection model YOLO. The YOLO algorithm is a fast and efficient target detection algorithm proposed in 2015. Compared with traditional target detection methods, the YOLO algorithm does not require multiple scans of the image, but directly detects and classifies the entire image through a single neural network model, and outputs the position and category probability of each target box.

[0063] The workflow of the YOLO algorithm can be divided into the following steps:

[0064] 1. Divide the input image into a grid.

[0065] 2. For each grid, predict multiple bounding boxes.

[0066] 3. For each bounding box, predict its category probability and location information.

[0067] 4. Use non-maximum suppression (NMS) to remove redundant bounding boxes.

[0068] The following is a detailed description of the workflow of the YOLO algorithm.

[0069] 1. Divide the input image into a grid:

[0070] First, the input image is divided into an SxS grid. For each grid, the algorithm predicts multiple bounding boxes, usually 5, each of which includes 4 coordinate values ​​(x, y, w, h) and a confidence score (the confidence score is used to judge the accuracy of the detection box, and the value is between 0-1, the closer to 1, the more accurate it is).

[0071] 2. Predict multiple bounding boxes:

[0072] For each bounding box, a convolutional neural network (CNN) is used to classify and regress it (classification is to judge the category of each bounding box based on the feature map, which refers to whether it is a crack in this case. Regression is to adjust the position of each bounding box (i.e. x, y, w, h) based on the feature map so that it just frames the crack). Specifically, in the Yolo v3 version, DarkNet-53 is used as the backbone network to extract features, and classification and regression are performed on the feature map.

[0073] 3. Predict category probability and location information:

[0074] For each bounding box, predict its category probability and location information (using the trained model to perform the above classification and regression calculations is prediction, where classification will determine the category probability of whether it is a crack, and regression will calculate the location information of the bounding box). In the YOLO algorithm, the softmax function is used to calculate the probability of each category, and a linear transformation is used to obtain the coordinates of the bounding box.

[0075] 4. Use non-maximum suppression (NMS):

[0076] Finally, non-maximum suppression (NMS) is used to remove redundant bounding boxes in all bounding boxes. Specifically, for each type of object, the bounding box with the highest confidence score is first selected, then its overlapping area with other bounding boxes is calculated, and the bounding boxes whose IoU (Intersection-over-Union) is greater than the threshold are deleted.

[0077] Specifically, first build a set H to store the bounding boxes to be processed, initialized to contain all N boxes, and then build a set M to store the optimal boxes, initialized to an empty set; then sort all the boxes in set H according to the confidence score, select the box m with the highest score, and move it from set H to set M; then traverse the boxes in set H, and calculate the intersection and union ratio (i.e., the area of ​​the intersection of two boxes divided by the area occupied by the previous two boxes combined) with box m respectively. If it is higher than a certain threshold (usually 0 to 0.5), it is considered that this box overlaps with m, and this box is taken out of set H. Then, return to the first step to iterate until set H is empty. The box in set M is the final output result.

[0078] When the system detects the existence of cracks, it infers that the mountain has undergone significant deformation and displacement, so the system sends an early warning signal to the staff, thus providing a valuable time window for taking preventive measures. Figure 3 As shown, the radar data provides the system with a coarse-grained location of the deformation, and the image data provides the system with a fine-grained location of the deformation (e.g. Figure 4 Finally, the system generates an alarm.

[0079] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0080] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A slope safety monitoring system based on multi-sensor fusion, characterized by: The following detection steps are included: Step S1, using synthetic aperture radar to repeatedly move multiple times on the same track to obtain multiple detection images; Step S2, selecting two images of SAR signals received by the radar twice in succession, and performing complex conjugate multiplication on each complex pixel point of the two images to obtain a complex interference pattern; Step S3, extracting the interference phase of each pixel point in the complex interference pattern, and obtaining the deformation of the monitoring surface through phase unwrapping; Step S4, collecting slope images by using a drone, and dividing the slope images collected by the drone into a plurality of grids; Step S5, for each grid, predict multiple bounding boxes through an algorithm to obtain the coordinate value and confidence score of each bounding box; wherein the bounding box is used to frame the cracks on the mountain; Step S6, classifying and regressing each bounding box, and selecting the bounding box with the highest confidence score by classifying and regressing each bounding box; Step S7, using non-maximum suppression to delete redundant bounding boxes and outputting bounding box data; Step S8, judging whether the mountain has given an alarm based on the outputted bounding box data.

2. The slope safety monitoring system based on multi-sensor fusion according to claim 1 is characterized by: In step S2, complex conjugate multiplication is performed on each complex pixel of the two images to obtain the complex interference pattern calculation formula: (a+bi)*(a-bi)=a 2 -(bi) 2 =a 2 -b 2 i 2 =a 2 +b 2 In the formula, the size of the complex pixel of the image is a+bi, and the corresponding conjugate complex size is a-bi, and the result is the pixel value of the complex interference pattern.

3. The slope safety monitoring system based on multi-sensor fusion according to claim 2 is characterized in that: In step S3, the deformation of the monitoring surface is calculated by the following formula: In the formula, ΔR is the path difference of the signal, and λ is the wavelength of the electromagnetic wave emitted by the radar.

4. The slope safety monitoring system based on multi-sensor fusion according to claim 3 is characterized by: In step S4, the slope image collected by the UAV is divided into SxS grids. For each grid, each bounding box includes 4 coordinate values ​​(x, y, w, h) and a confidence score.

5. The slope safety monitoring system based on multi-sensor fusion according to claim 4 is characterized in that: In step S5, the algorithm predicts multiple bounding boxes as follows: first, the neural network weights are trained using data, and then the trained neural network is used to predict the bounding boxes of the input data. The neural network outputs the coordinate values ​​of the target bounding boxes.

6. A slope safety monitoring system based on multi-sensor fusion according to claim 5, characterized in that: For each bounding box, a convolutional neural network is used to classify and regress it. The classification is to determine whether each bounding box belongs to a crack based on the feature map; the regression is to adjust the coordinate values ​​(x, y, w, h) of the position of each bounding box based on the feature map so that the bounding box frames the crack.

7. The slope safety monitoring system based on multi-sensor fusion according to claim 6 is characterized by: The specific implementation process of classification and regression is: first, the neural network weights are trained through data, and then the trained neural network is used to predict the input data. The classification task means that the neural network will give a confidence score for each category of the target, and select the category with the largest confidence score as the category of the target; the regression task means that the neural network will output the coordinate value of the target bounding box.

8. The slope safety monitoring system based on multi-sensor fusion according to claim 7 is characterized by: Among all bounding boxes, non-maximum suppression is used to remove redundant bounding boxes.

9. The slope safety monitoring system based on multi-sensor fusion according to claim 8 is characterized by: The specific method of using non-maximum suppression to delete redundant bounding boxes is as follows: construct a set H to store the bounding boxes to be processed, initialized to contain all N bounding boxes, and then construct a set M to store the optimal bounding boxes, initialized to an empty set; sort all the bounding boxes in set H according to the confidence score, select the bounding box m with the highest score, and move it from set H to set M; then, traverse the boxes in set H, and calculate the intersection and union ratio with box m respectively. If it is higher than the threshold, it is considered that this box overlaps with m, and this box is taken out of set H.

10. A slope safety monitoring system based on multi-sensor fusion according to claim 9, characterized in that: Repeatedly use non-maximum suppression to delete redundant bounding boxes until the set H is empty and the bounding boxes in the set M are the final output results.

Images