Debris Flow Early Warning Method, Device, System and Medium Based on Image and Radar

Abstract

The present invention relates to the technical field of geological disaster early warning, and in particular to a debris flow early warning method, device, system and medium based on images and radar. The debris flow early warning method based on images and radar of the present invention includes: obtaining a first reference video stream of a mountain body captured by a first imaging unit within a first time period and a second reference video stream of the mountain body captured by a second imaging unit; obtaining a first video stream of the mountain body captured by the first imaging unit within a second time period and a second video stream of the mountain body captured by the second imaging unit; obtaining the sliding distance of the mountain body and the target area monitored by radar according to the image differences among the first reference video stream, the first video stream, the second reference video stream and the second video stream; controlling the radar to monitor the target area, and sending a debris flow early warning signal according to the sliding distance of the mountain body and / or the monitoring result. The present invention can avoid the interference caused by the direct contact between the debris flow monitoring device and the monitored mountain body area.

Description

Technical Field

[0001] The present invention relates to the technical field of geological disaster early warning, and particularly to a debris flow early warning method, device, system and medium based on images and radar. Background Art

[0002] Debris flow is a geological disaster that frequently occurs in mountainous areas and is characterized by explosiveness and strong destructiveness. In order to avoid a large amount of property and life losses, it is necessary to monitor debris flow disasters and give early warnings in a timely manner. Currently, devices for monitoring and warning debris flows include mud flow meters, mud level monitors, etc. However, these instruments need to be installed on the slopes where debris flows are likely to occur and are in contact with the monitored area, so they are easily interfered by the contact area, resulting in inaccurate early warnings. Since the monitoring range of a single instrument is small, when the monitored area is large, a large number of instruments are required, so the cost is high. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a debris flow early warning method, device, system and medium based on images and radar, which are used to solve the technical problem of low accuracy of existing debris flow early warning technologies.

[0004] The technical solution adopted by the present invention is as follows:

[0005] In the first aspect, the present invention provides a debris flow early warning method based on images and radar, and the method includes the following steps:

[0006] A debris flow early warning method based on images and radar, the method includes the following steps:

[0007] Obtain a first reference video stream obtained by a first imaging unit of a binocular camera shooting a mountain body and a second reference video stream obtained by a second imaging unit of the binocular camera shooting the mountain body within a first time period;

[0008] Obtain a first video stream obtained by the first imaging unit of the binocular camera shooting the mountain body and a second video stream obtained by the second imaging unit of the binocular camera shooting the mountain body within a second time period after the first time period;

[0009] Obtain the landslide distance of the mountain body and the target area monitored by the radar according to the image differences among the first reference video stream, the first video stream, the second reference video stream and the second video stream;

[0010] Control the radar to monitor the target area, and send a debris flow early warning signal according to the landslide distance and / or the monitoring result.

[0011] Preferably, obtaining the landslide distance of the mountain body and the target area monitored by the radar according to the image differences among the first reference video stream, the first video stream, the second reference video stream and the second video stream further includes the following steps.

[0012] Analyze whether the change of the mountain body meets the preset conditions according to the first reference video stream and the first video stream;

[0013] If so, obtain the image difference between the first reference video stream and the first video stream;

[0014] Obtain the landslide distance of the mountain body and the target area monitored by the radar according to the image difference, the first reference video stream, the first video stream, the second reference video stream and the second video stream.

[0015] Preferably, the step of analyzing whether the change of the mountain body meets the preset conditions according to the first reference video stream and the first video stream further includes the following steps:

[0016] Obtain the average image of the first reference video stream as the first average image;

[0017] Obtain the average image of the first video stream as the second average image;

[0018] Obtain the cross-correlation coefficient between the first average image and the second average image and the first threshold.

[0019] If the cross-correlation coefficient is greater than the first threshold, the change of the mountain body meets the preset conditions, otherwise it does not meet the preset conditions.

[0020] Preferably, the step of obtaining the image difference between the first reference video stream and the first video stream if so further includes the following steps:

[0021] Obtain the first grayscale image converted from the first average image and the second grayscale image converted from the second average image;

[0022] Obtain the first image set for the n-layer pyramid image constructed for the first grayscale image;

[0023] Obtain the second image set for the n-layer pyramid image constructed for the second grayscale image;

[0024] Subtract the first image set and the second image set to obtain a difference image set;

[0025] Perform low-pass filtering and amplification processing on each image in the difference image set;

[0026] Overlay the images after low-pass filtering and amplification processing together to obtain a reference image with the same resolution as the first average image;

[0027] Obtain several pixel points with pixel values greater than the set threshold in the reference image as the difference pixel point set.

[0028] Preferably, obtaining the landslide distance and the target area monitored by the radar according to the image difference, the first reference video stream, the first video stream, the second reference video stream, and the second video stream further includes the following steps:

[0029] Obtain the three-dimensional reconstruction image of the first grayscale image from the first reference video stream and the second reference video stream as the first three-dimensional image;

[0030] Obtain the three-dimensional reconstruction image of the second grayscale image from the first video stream and the second video stream as the second three-dimensional image;

[0031] Obtain the depth value set corresponding to the differential pixel point set from the first three-dimensional image and the differential pixel point set as the first depth value set (za1, za2, …, za(m - 1), zam);

[0032] Obtain the depth value set corresponding to the differential pixel point set from the second three-dimensional image and the differential pixel point set as the second depth value set (zb1, zb2, …, zb(m - 1), zbm);

[0033] Calculate the landslide distance S according to the first depth value set and the second depth value set, where S = (zb1 - za1 + zb2 - za2 + zb(m - 1) - za(m - 1) + zbm - zam) / m, where m is a positive integer greater than 1.

[0034] Preferably, obtaining the depth value set corresponding to the differential pixel point set from the first three-dimensional image and the differential pixel point set as the first depth value set (za1, za2, …, za(m - 1), zam) further includes the following steps:

[0035] Obtain a number of uniform target pixels in the first grayscale image according to the differential pixel point set;

[0036] Obtain the corresponding first target area centered on each uniform target pixel in the first grayscale image;

[0037] Obtain the corresponding second target area of each first target area in the first three-dimensional image;

[0038] For each first target area, obtain the depth value of each pixel in the first target area according to the corresponding second target area;

[0039] For each first target area, obtain the three-dimensional coordinates (xi, yi, zi) of each pixel in the first target area, where the xi coordinate and the yi coordinate are respectively the abscissa and ordinate of the pixel in the first grayscale image, and the zi coordinate is the depth value of the pixel.

[0040] For each first target area, surface fitting is performed on all pixel points in the first target area according to the three-dimensional coordinates of the pixel points to obtain a surface equation;

[0041] The depth values of each target pixel point are obtained according to the abscissa, ordinate and corresponding surface equation of the target pixel point in the first grayscale image, and the set of depth values of all target pixel points is used as the first depth value set.

[0042] Preferably, the steps of obtaining the landslide distance and the target area monitored by the radar according to the image difference, the first reference video stream, the first video stream, the second reference video stream and the second video stream further include the following steps:

[0043] The three-dimensional coordinates of each target pixel point are obtained, and the three-dimensional coordinates of the target pixel point include the abscissa, ordinate and depth value of the target pixel point in the second grayscale image;

[0044] The relative position between the landslide area and the binocular camera is obtained according to the three-dimensional coordinates of each target pixel point;

[0045] The relative position between the radar and the binocular camera is obtained;

[0046] The target area monitored by the radar is obtained according to the relative position between the landslide area and the binocular camera and the relative position between the radar and the binocular camera.

[0047] In a second aspect, the present invention further provides a debris flow warning device based on video images and radar, and the device includes:

[0048] A debris flow warning device based on video images and radar, and the device includes:

[0049] A reference video stream acquisition module, which is used to acquire a first reference video stream obtained by the first imaging unit of the binocular camera shooting the mountain body and a second reference video stream obtained by the second imaging unit of the binocular camera shooting the mountain body within the first time period;

[0050] A video stream acquisition module, which is used to acquire a first video stream obtained by the first imaging unit of the binocular camera shooting the mountain body and a second video stream obtained by the second imaging unit of the binocular camera shooting the mountain body within the second time period after the first time period;

[0051] A video stream analysis module, which is used to obtain the landslide distance and the target area monitored by the radar according to the image difference between the first reference video stream, the first video stream, the second reference video stream and the second video stream;

[0052] A radar monitoring module, which is used to control a radar to monitor the target area and send a debris flow early warning signal according to the monitoring result.

[0053] In a third aspect, the present invention also provides a debris flow early warning system based on video images and radar, which is characterized by including: a binocular camera, a radar, at least one processor, at least one memory, and computer program instructions stored in the memory. The binocular camera and the radar are respectively electrically connected to the processor. When the computer program instructions are executed by the processor, the method described in the first aspect is implemented.

[0054] In a fourth aspect, the present invention also provides a storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0055] Beneficial effects: The debris flow early warning method, device, system and medium based on images and radar of the present invention analyze video streams of two time periods to obtain minute image differences between the two time periods. The aforementioned image differences can reflect the minute sliding conditions of the photographed mountain before a debris flow breaks out. The present invention calculates the sliding distance of the mountain based on the image differences and finds the area where the mountain slides, and then uses a radar to closely monitor this area, so that the potential areas where debris flows may break out can be monitored in a targeted manner, without the need to deploy a large number of high-precision radars to cover all areas of the mountain. Therefore, it has the characteristics of low cost, low energy consumption, and high precision. Since both the binocular camera and the radar can monitor the mountain in a non-contact manner, it is not easily affected by the mountain itself being monitored, which can further improve the accuracy of early warning. Description of the Drawings

[0056] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required to be used in the embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings, and all of these are within the protection scope of the present invention.

[0057] Figure 1 It is a schematic flowchart of the debris flow early warning method based on images and radar of the present invention;

[0058] Figure 2 It is a schematic flowchart of the method for obtaining the sliding distance of the mountain of the present invention;

[0059] Figure 3 It is a schematic flowchart of the method for analyzing the changes of the mountain of the present invention;

[0060] Figure 4Flowchart schematic diagram of the method for obtaining the difference image of the present invention;

[0061] Figure 5 Flowchart schematic diagram of the method for calculating the landslide distance of the present invention;

[0062] Figure 6 Flow process schematic diagram of the method for obtaining the first depth value set of the present invention;

[0063] Figure 7 Flow process schematic diagram of the method for obtaining the target area monitored by radar of the present invention;

[0064] Figure 8 Structural schematic diagram of the debris flow early warning device based on video image and radar of the present invention;

[0065] Figure 9 Structural block diagram of the debris flow early warning based on image and radar of the present invention;

[0066] Figure 10 Schematic diagram of dividing the first grayscale image into a number of uniform rectangular regions of the present invention;

[0067] Figure 11 Schematic diagram of obtaining the abnormal pixel point closest to the center position of the rectangular region of the present invention. Detailed implementation manners

[0068] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It should be noted that, in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or sequence between these entities or operations. In the description of the present invention, it should be understood that the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation of the present invention. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, the elements defined by the statement "including..." do not exclude the existence of additional identical elements in the process, method, article or device including the said elements. If there is no conflict, the embodiments of the present invention and the various features in the embodiments may be combined with each other, and all are within the protection scope of the present invention.

[0069] Embodiment 1

[0070] As Figure 1 shown, this embodiment provides a debris flow warning method based on images and radar. The method includes the following steps:

[0071] S1: Obtain a first reference video stream C1 captured by the first imaging unit of the binocular camera for the mountain body within the first time period and a second reference video stream C2 captured by the second imaging unit of the binocular camera for the mountain body.

[0072] There are two imaging units in the binocular camera, and the two imaging units are in different positions, so that the monitored mountain body can be photographed from two different angles. The baseline length of the binocular camera is L, and the resolution is W*H. To avoid the influence of accidental interference factors such as wind or falling stones, this step collects a video stream for a period of time, and the collected video stream includes multiple frames of images stored in chronological order. The length of the first time period can be set according to experience.

[0073] S2: Obtain a first video stream C11 obtained by the first camera unit of the binocular camera shooting the mountain body and a second video stream C22 obtained by the second camera unit of the binocular camera shooting the mountain body within a second time period after the first time period;

[0074] In this step, the binocular camera is used to shoot the mountain body again after a certain interval of time to obtain a video stream within a period of time. The collected video stream includes multiple frames of images stored in chronological order. The interval time and the length of the first time period can be set according to experience. As an optional but advantageous implementation manner, the first time period and the second time period are equal.

[0075] S3: Obtain the landslide distance of the mountain body and the target area monitored by the radar according to the image differences among the first reference video stream, the first video stream, the second reference video stream, and the second video stream;

[0076] Before a debris flow breaks out, the mountain body often shows slight sliding. This step uses the analysis of the video streams in two time periods to obtain the slight image differences between the two time periods. The aforementioned image differences can reflect the slight sliding conditions of the photographed mountain body. Therefore, this step can calculate the landslide distance of the mountain body and find the area where the landslide occurs, and then use the radar to closely monitor this area, which is conducive to early warning before the sudden outbreak of the debris flow.

[0077] S4: Control the radar to monitor the target area, and send a debris flow warning signal according to the landslide distance and / or the monitoring result.

[0078] In this embodiment, the binocular camera can be used to continuously shoot the mountain body. After finding the area where the mountain body shows slight sliding, control the radar to conduct targeted key monitoring on this area. When it is detected that the shape of the mountain body in this area changes to a certain extent within a short period of time, a warning signal is generated and sent out. This avoids arranging a large number of high-precision radars to monitor all mountain areas, thereby saving energy consumption and reducing costs while ensuring the accuracy and timeliness of the early warning. Among them, the radar can use an x-band radar to monitor the movement of stones within the target area. When the monitored movement speed of the stones exceeds the preset speed, a warning signal is sent. This embodiment can also send a warning signal when the calculated landslide distance of the mountain body exceeds the preset distance. In addition, this embodiment can also calculate the landslide speed of the mountain body according to the landslide distance of the mountain body and the interval time T between the first time period and the second time period. Let the aforementioned landslide distance be S, then the landslide speed v = T / S, and a warning signal can also be sent when the landslide speed exceeds the allowable speed.

[0079] As Figure 2As shown, as an optional but advantageous implementation, in this embodiment, step S3: obtaining the landslide distance and the target area monitored by the radar according to the image differences among the first reference video stream, the first video stream, the second reference video stream, and the second video stream further includes the following steps:

[0080] S31: Analyze whether the change of the mountain body satisfies a preset condition according to the first reference video stream and the first video stream;

[0081] This step determines that the mountain body has undergone a slight landslide before the debris flow breaks out. If the landslide scale is large, it indicates that the debris flow may have broken out. Since this embodiment is mainly used for early warning before the debris flow breaks out, it is mainly applicable to the analysis and processing under the condition of slight landslide of the mountain body. In order to improve the accuracy of the analysis, this step first analyzes the degree of change of the mountain body and screens out the cases where the degree of change of the mountain body meets the requirements for analysis and processing.

[0082] As Figure 3 shown, as an optional but advantageous implementation, in this embodiment, S31: Analyze whether the change of the mountain body satisfies a preset condition according to the first reference video stream and the first video stream further includes the following steps:

[0083] S311: Obtain the average image of the first reference video stream as the first average image;

[0084] This step can use the multi-image averaging method to process multiple frames of images of the first reference video stream to obtain the average image to reduce the noise caused by external interference factors such as falling rocks and wind.

[0085] S312: Obtain the average image of the first video stream as the second average image;

[0086] This step can use the multi-image averaging method to process multiple frames of images of the first video stream to obtain the average image to reduce the noise caused by external interference factors such as falling rocks and wind.

[0087] S313: Obtain the cross-correlation coefficient and the first threshold of the first average image and the second average image.

[0088] The first threshold is the maximum value that the cross-correlation coefficient cannot exceed, and this value can be determined according to experience.

[0089] S314: If the cross-correlation coefficient is greater than the first threshold, the change of the mountain body satisfies the preset condition; otherwise, it does not satisfy the preset condition.

[0090] For example, if the mutual correlation coefficient is e and the first threshold is E, then if e > E, it indicates that the landslide belongs to the category of minor landslides, and the changes in the mountain body meet the preset conditions; otherwise, it indicates that the landslide does not belong to the category of minor landslides, and the changes in the mountain body do not meet the aforementioned preset conditions. As an optional but advantageous implementation manner, E = 0.95 herein.

[0091] S32: If so, obtain the image difference between the first reference video stream and the first video stream;

[0092] In this step, when the landslide satisfies minor landslides, the image difference between the video streams captured at two time intervals is found, where the image difference refers to the pixel points of the images that have changed significantly.

[0093] As Figure 4 shown, as an optional but advantageous implementation manner, in this embodiment, the step S32: If so, obtain the image difference between the first reference video stream and the first video stream further includes the following steps:

[0094] S321: Obtain the first grayscale image converted from the first average image and the second grayscale image converted from the second average image;

[0095] S322: Obtain the first image set for the n-layer pyramid image constructed for the first grayscale image;

[0096] where n is a positive integer greater than 1. As an optional but advantageous implementation manner, n = 5. The n-layer pyramid image constructed from the first grayscale image has a total of n images, and the set of these n images is the first image set.

[0097] S323: Obtain the second image set for the n-layer pyramid image constructed for the second grayscale image;

[0098] The n-layer pyramid image constructed from the second grayscale image has a total of n images, and the set of these n images is the second image set.

[0099] The n-layer pyramid image is composed of a series of images. The bottommost image has the largest size, and the topmost image has the smallest size.

[0100] S324: Subtract the second image set from the first image set to obtain the difference image set;

[0101] In this step, the first layer image in the first image set is subtracted from the first layer in the first image set to obtain the difference image of the first layer. The second layer image in the first image set is subtracted from the second layer in the first image set to obtain the difference image of the second layer, ……, the nth layer image in the first image set is subtracted from the nth layer in the first image set to obtain the difference image of the nth layer. Then, all the difference images from the first layer to the nth layer are combined to form a difference image set. Subtracting two images means subtracting the pixel values of the corresponding pixel points (the pixel points with the same horizontal and vertical coordinates in the two images) and then taking the absolute value, and this absolute value is used as the pixel value of the corresponding pixel point (the pixel point with the same horizontal and vertical coordinates) in the difference image.

[0102] S325: Perform low-pass filtering and magnification processing on each image in the difference image set;

[0103] In this step, each image in the difference set can be processed separately. Low-pass filtering can be performed first, and then magnification.

[0104] S326: Stack the images after low-pass filtering and magnification processing together to obtain a reference image with the same resolution as the first average image;

[0105] After the n images in the difference image set have all been subjected to low-pass filtering and magnification processing, the n images are restored to their original sizes before the component pyramid image, and then the n processed images are stacked together to restore to the original resolution before constructing the pyramid image.

[0106] S327: Obtain a number of pixel points in the reference image whose pixel values are greater than the set threshold as a set of difference pixel points.

[0107] The set threshold can be determined based on experience. As an optional implementation, the set threshold can be any real number between 15 and 45. For the pixel points in the reference image whose pixel values are greater than the set threshold, the horizontal and vertical coordinates of the pixel point in the reference image can be obtained, and the pixel points with the same horizontal and vertical coordinates in the first grayscale image and the second grayscale image are the difference pixel points in the first grayscale image and the difference pixel points in the second grayscale image respectively.

[0108] S33: Obtain the landslide distance and the target area monitored by the radar based on the image difference, the first reference video stream, the first video stream, the second reference video stream, and the second video stream.

[0109] As Figure 5 shown, the method for obtaining the landslide distance mainly includes the following steps:

[0110] S331: Obtain the three-dimensional reconstruction image of the first grayscale image as the first three-dimensional image according to the first reference video stream and the second reference video stream;

[0111] Since this embodiment uses a binocular camera to simultaneously capture the mountain from different angles, the first reference video stream and the second reference video stream captured by the two camera units in the binocular camera can be used to establish the three-dimensional image of the captured mountain. To avoid noise interference, the grayscale image obtained by converting the average image of the first reference video stream and the grayscale image obtained by converting the average image of the second reference video stream can be used to establish the three-dimensional image, and the method of establishing the three-dimensional image using the images captured by the binocular camera can adopt the existing technology.

[0112] S332: Obtain the three-dimensional reconstruction image of the second grayscale image as the second three-dimensional image according to the first video stream and the second video stream;

[0113] Similarly, this embodiment uses the two video streams captured by the binocular camera in the second time period to establish the three-dimensional image, and the method is the same as the previous step, which will not be elaborated here.

[0114] S333: Obtain the depth value set corresponding to the set of differential pixel points as the first depth value set (za1, za2,..., za(m - 1), zam) according to the first three-dimensional image and the set of differential pixel points;

[0115] In this step, the depth value of each differential pixel in the set of differential pixel points in the first three-dimensional image is obtained using the first three-dimensional image, and these depth values are combined into the first depth value set. One element in the set represents the depth value corresponding to a differential pixel. For example, za1 represents the depth value of the 1st differential pixel, za2 represents the depth value of the 2nd differential pixel, za(m - 1) represents the depth value of the (m - 1)th differential pixel, and zam represents the depth value of the mth differential pixel.

[0116] As Figure 6 shown, as a preferred embodiment, in this embodiment, S333: Obtain the depth value set corresponding to the set of differential pixel points as the first depth value set (za1, za2,..., za(m - 1), zam) further includes the following steps:

[0117] S3331: Obtain several uniform target pixels in the first grayscale image according to the set of differential pixel points;

[0118] S3332: Obtain the corresponding first target regions centered on each of the uniform target pixels in the first grayscale image;

[0119] In the first grayscale image, a region with a width of w and a height of h is selected centered on each target pixel point, where the width is the length in the x direction of the first grayscale image, and the height is the length in the y direction of the first grayscale image;

[0120] S3333: Obtain the corresponding second target regions for each first target region in the first three-dimensional image;

[0121] In the first three-dimensional image, also centered on the target pixel point, a region with a width of w and a height of h is selected, and this region is the second target region corresponding to the first target region in the previous step. Where the width is the length in the x direction of the first three-dimensional image, and the height is the length in the y direction of the first three-dimensional image;

[0122] S3334: For each first target region, obtain the depth value of each pixel point in the first target region according to the corresponding second target region;

[0123] Input the two-dimensional coordinates of all pixel points in the first target region into the three-dimensional image, and obtain the third coordinate corresponding to the two-dimensional coordinates in the three-dimensional image. The third coordinate is the depth value of the pixel point.

[0124] For example, the coordinates of the i-th pixel in the first target region are (xi, yi), where xi is the abscissa and yi is the ordinate. That is, the i-th pixel is located in the xi-th column and yi-th row of the first grayscale image. And the three-dimensional coordinates of the pixel point with abscissa xi and ordinate yi in the first three-dimensional image are (xi, yi, zi), then zi is the depth value of the i-th pixel in the first target region. Since each difference pixel point corresponds to a first target region, the above operations can be performed on each first target region one by one.

[0125] S3335: For each first target region, obtain the three-dimensional coordinates (xi, yi, zi) of each pixel point in the first target region, where the xi coordinate and yi coordinate are respectively the abscissa and ordinate of the pixel point in the first grayscale image, and the zi coordinate is the depth value of the pixel point.

[0126] S3336: For each first target region, perform surface fitting on all pixel points in the first target region according to the three-dimensional coordinates of the pixel points to obtain a surface equation;

[0127] In this step, a surface is performed on all pixel points in the first target region according to the three-dimensional coordinates of all pixel points in the first target region. Since there are multiple first target regions, the above operations can be performed on these first target regions one by one, so as to obtain the surface equation corresponding to each first target region.

[0128] S3337: Obtain the depth values of each target pixel point according to the abscissa, ordinate, and corresponding surface equation of each target pixel point in the first grayscale image, and use the set of depth values of all target pixel points as the first depth value set.

[0129] In this step, substitute the abscissa and ordinate of the target pixel point in the first grayscale image into the surface equation corresponding to the first target region where the target pixel point is located to obtain the depth value of the target pixel point, and use the depth values of all obtained target pixel points as the first depth value set.

[0130] S334: Obtain the depth value set corresponding to the set of difference pixel points as the second depth value set (zb1, zb2,..., zb(m - 1), zbm) according to the second three-dimensional image and the set of difference pixel points;

[0131] Similarly, in this step, use the second three-dimensional image to obtain the depth value of each difference pixel point in the set of difference pixel points in the second three-dimensional image, and combine these depth values into the second depth value set. An element in the set represents the depth value corresponding to a difference pixel point. For example, zb1 represents the depth value of the 1st difference pixel point, zb2 represents the depth value of the 2nd difference pixel point, zb(m - 1) represents the depth value of the (m - 1)th difference pixel point, and zbm represents the depth value of the mth difference pixel point.

[0132] As a preferred implementation manner, in this embodiment, the S334: Obtain the depth value set corresponding to the set of difference pixel points as the second depth value set (zb1, zb2,..., zb(m - 1), zbm) further includes the following steps:

[0133] S3341: Obtain several uniform target pixel points in the second grayscale image according to the set of difference pixel points;

[0134] S3342: Obtain the corresponding first target regions centered on each uniform target pixel point in the second grayscale image; In the second grayscale image, select a region with a width of w and a height of h centered on each target pixel point, where the width is the length in the x direction of the second grayscale image, and the height is the length in the y direction of the second grayscale image;

[0135] S3343: Obtain the corresponding second target regions of each first target region in the second three-dimensional image;

[0136] In the second three-dimensional image, also centered on the target pixel point, select a region with a width of w and a height of h, and this region is the second target region corresponding to the first target region in the previous step. Where the width is the length in the x direction of the second three-dimensional image, and the height is the length in the y direction of the second three-dimensional image;

[0137] S3344: For each first target region, obtain the depth value of each pixel point in the first target region according to the corresponding second target region; input the two-dimensional coordinates of all pixel points in the first target region into the three-dimensional image, and obtain the third coordinate corresponding to the two-dimensional coordinates in the three-dimensional image, and the third coordinate is the depth value of the pixel point.

[0138] For example, the coordinates of the i-th pixel in the first target region are (xi, yi), where xi is the abscissa and yi is the ordinate. That is, the i-th pixel is located in the xi-th column and yi-th row of the second grayscale image. And the three-dimensional coordinates of the pixel point with abscissa xi and ordinate yi in the first three-dimensional image are (xi, yi, zi), then zi is the depth value of the i-th pixel in the first target region. Since each differential pixel point corresponds to a first target region, the foregoing operations can be performed on the first target regions one by one.

[0139] S3345: For each first target region, obtain the three-dimensional coordinates (xi, yi, zi) of each pixel point in the first target region, where the xi coordinate and the yi coordinate are the abscissa and ordinate of the pixel point in the second grayscale image respectively, and the zi coordinate is the depth value of the pixel point.

[0140] S3346: For each first target region, perform surface fitting on all pixel points in the first target region according to the three-dimensional coordinates of the pixel points to obtain a surface equation; in this step, surface fitting is performed on all pixel points in the first target region according to the three-dimensional coordinates of all pixel points in the first target region. Since there are multiple first target regions, the foregoing operations can be performed on these first target regions one by one, so as to obtain the surface equation corresponding to each first target region.

[0141] S3347: Obtain the depth values of each target pixel point according to the abscissa, ordinate and the corresponding surface equation of each target pixel point in the first grayscale image, and use the set of depth values of all target pixel points as the second depth value set. In this step, substitute the abscissa and ordinate of the target pixel point in the first grayscale image into the surface equation corresponding to the first target region where the target pixel point is located to obtain the depth value of the target pixel point, and use all the obtained depth values of the target pixel points as the second depth value set.

[0142] As Figure 10 shown, the foregoing uniform target pixel points refer to the differential pixel points closest to the center position of each region after the first grayscale image or the second grayscale image is evenly divided into multiple regions. For this, the first grayscale image or the second grayscale image can be evenly divided into k rectangular regions with the same length and width first, and then the center coordinates of each rectangular region are obtained. Figure 10The small circles therein indicate the positions of the central coordinates of each area, and the large hollow circles indicate abnormal pixel points. As Figure 11 shown, then find a differential pixel point closest to the central coordinate within the rectangular area as the uniform target pixel point. That is, search can be carried out centered on the central position of the rectangular area, and find the differential pixel point closest to the center from all the pixels in the differential pixel point set as the uniform target pixel point of this area. Figure 11 The solid circles in it indicate the uniform target pixel points selected from the differential pixel points. Since the resolutions of the reference image, the first grayscale image, and the second grayscale image are the same, the abscissa and ordinate of the differential pixel point in the reference image can be used as the abscissa and ordinate of the differential pixel point in the first grayscale image and the second grayscale image (that is, the differential pixel point can be represented by the pixel points of the reference image, the first grayscale image, and the second grayscale image, and the abscissa and ordinate of the same differential pixel point are the same in the foregoing three images), so as to obtain the differential pixel point closest to the divided distance area in the first grayscale image or the second grayscale image as the target pixel point. The foregoing operations are respectively performed on each divided rectangular area to obtain the same number of target pixel points as the number of divided rectangular areas.

[0143] In addition to obtaining the depth value of the target pixel point in the foregoing manner, a radar point cloud map of the mountain surface can also be generated by radar, and then the target pixel point is registered with the radar point cloud map, and the depth value of the target pixel point is obtained through the position of the target pixel point in the radar point cloud map after registration.

[0144] S335: Calculate the mountain sliding distance S according to the first depth value set and the second depth value set, where S=(zb1 - za1 + zb2 - za2 + zb(m - 1) - za(m - 1) + zbm - zam) / m, where m is a positive integer greater than 1. As described in this embodiment, S33: Obtaining the mountain sliding distance and the target area monitored by radar according to the image difference, the first reference video stream, the first video stream, the second reference video stream, and the second video stream further includes the following steps:

[0145] S336: Obtain the three-dimensional coordinates of each target pixel point, and the three-dimensional coordinates of the target pixel point include the abscissa, ordinate, and depth value of the target pixel point in the second grayscale image;

[0146] S337: Obtain the relative position between the landslide area and the binocular camera according to the three-dimensional coordinates of each target pixel point; after the binocular camera is installed at the monitoring location, calibrate the parameters of the binocular camera, and use the binocular camera to reconstruct a three-dimensional image. Obtain the three-dimensional coordinates of each target pixel point in the camera coordinates of the binocular camera by using the two-dimensional coordinates of the target pixel point and the reconstructed three-dimensional image, that is, substitute the two-dimensional coordinates of the target pixel point into the three-dimensional image to obtain the three-dimensional coordinates of the target pixel point. Take the mountain area corresponding to the target pixel point as the landslide area. The relative position between the landslide area corresponding to the target pixel point and the binocular camera can be found through the aforementioned three-dimensional coordinates.

[0147] S338: Obtain the relative position between the radar and the binocular camera; after the radar and the binocular camera are both installed at the designated location, the relative position relationship between the two is determined.

[0148] S339: Obtain the target area monitored by the radar according to the relative position between the landslide area and the binocular camera and the relative position between the radar and the binocular camera. In this step, the landslide area can be transferred from the camera coordinate system of the binocular camera to the coordinate system of the radar by means of coordinate transformation, so as to obtain the coordinates of the landslide area in the radar coordinate system.

[0149] Embodiment 2

[0150] Please refer to Figure 8 , this embodiment provides a debris flow early warning device based on video images and radar, and the device includes:

[0151] A reference video stream acquisition module, which is used to acquire a first reference video stream obtained by the first imaging unit of the binocular camera photographing the mountain body and a second reference video stream obtained by the second imaging unit of the binocular camera photographing the mountain body within a first time period;

[0152] A video stream acquisition module, which is used to acquire a first video stream obtained by the first imaging unit of the binocular camera photographing the mountain body and a second video stream obtained by the second imaging unit of the binocular camera photographing the mountain body within a second time period after the first time period;

[0153] A video stream analysis module, which is used to obtain the landslide distance and the target area monitored by the radar according to the image differences among the first reference video stream, the first video stream, the second reference video stream, and the second video stream;

[0154] A radar monitoring module, which is used to control the radar to monitor the target area and send a debris flow early warning signal according to the monitoring result. The travel direction and pedestrian position acquisition module further includes:

[0155] The video stream analysis module further includes: a condition checking sub-module, which is used to check whether the change of the mountain body satisfies a preset condition according to the first reference video stream and the first video stream; an image difference obtaining sub-module, which is used to obtain the image difference between the first reference video stream and the first video stream if the condition is satisfied; a sliding distance and target area obtaining sub-module, which is used to obtain the mountain body sliding distance and the target area monitored by the radar according to the image difference, the first reference video stream, the first video stream, the second reference video stream, and the second video stream.

[0156] Embodiment 3

[0157] In addition, combined with Figure 9 the debris flow warning method based on images and radar in the foregoing embodiments of the present invention described can be implemented by the smart pole of this embodiment. Figure 9 Fig. shows a schematic structural diagram of debris flow warning based on video images and radar provided by an embodiment of the present invention.

[0158] The smart pole of this embodiment may include a processing circuit 401, a binocular camera 404, a radar 405, and a memory 402 storing computer program instructions.

[0159] Specifically, the above-mentioned processing circuit 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0160] The memory 402 may include a mass memory for data or instructions. By way of example and not limitation, the memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disc, a magneto-optical disc, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In a suitable case, the memory 402 may include a removable or non-removable (or fixed) medium. In a suitable case, the memory 402 may be internal or external to the data processing device. In a specific embodiment, the memory 402 is a non-volatile solid-state memory. In a specific embodiment, the memory 402 includes a read-only memory (ROM). In a suitable case, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0161] The processor 401 reads and executes the computer program instructions stored in the memory 402 to implement any one of the regional random intelligent pole data addressing methods in the above embodiments.

[0162] In one example, the intelligent pole of this embodiment may further include a communication interface 403 and a bus 410. Among them, as Figure 9 shown, the processing circuit 401, the memory 402, the communication interface 403, the binocular camera 404, and the radar 405 are connected through the bus 410 and complete communication with each other.

[0163] The communication interface 403 is mainly used to implement communication between each module, device, unit, and / or device in the embodiments of the present invention.

[0164] The bus 410 includes hardware, software, or both, and couples the various components used for the intelligent pole together. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses or a combination of two or more of these. In a suitable case, the bus 410 may include one or more buses. Although the embodiments of the present invention describe and illustrate a specific bus, the present invention contemplates any suitable bus or interconnect.

[0165] Embodiment 4

[0166] In addition, in combination with the debris flow early warning method based on images and radar in the above embodiments, the embodiments of the present invention can be implemented by providing a computer-readable storage medium. Computer program instructions are stored on the computer-readable storage medium; when the computer program instructions are executed by a processor, any one of the debris flow early warning methods based on images and radar in the above embodiments is implemented.

[0167] The above is a detailed introduction to the debris flow early warning method, device, equipment, and storage medium provided by the embodiments of the present invention.

[0168] It should be clear that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present invention.

[0169] The functional blocks shown in the above-described structural block diagrams can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a functional card, and so on. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted via a data signal carried in a carrier wave on a transmission medium or a communication link. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical discs, hard disks, fiber optic media, radio frequency (RF) links, and so on. The code segment can be downloaded via a computer network such as the Internet, an intranet, and so on.

[0170] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be executed in the order mentioned in the embodiments, or different from the order in the embodiments, or several steps can be executed simultaneously.

[0171] As described above, the above is only the specific implementation manner of the present invention. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the above-described systems, modules, and units can refer to the corresponding processes in the foregoing method embodiments and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention can easily think of various equivalent modifications or substitutions, and these modifications or substitutions should all be covered by the protection scope of the present invention.

Claims

1. A debris flow early warning method based on images and radar, the method comprising the following steps: Obtain a first reference video stream captured by a first imaging unit of a binocular camera of a mountain body and a second reference video stream captured by a second imaging unit of the mountain body within a first time period; Obtain a first video stream captured by the first imaging unit of the binocular camera of the mountain body and a second video stream captured by the second imaging unit of the mountain body within a second time period after the first time period; Obtain the landslide distance of the mountain body and the target area monitored by the radar according to the image differences among the first reference video stream, the first video stream, the second reference video stream, and the second video stream; Control the radar to monitor the target area, and send a debris flow early warning signal according to the landslide distance and / or the monitoring result; The step of obtaining the landslide distance of the mountain body and the target area monitored by the radar according to the image differences among the first reference video stream, the first video stream, the second reference video stream, and the second video stream further comprises the following steps: Analyze whether the change of the mountain body satisfies a preset condition according to the first reference video stream and the first video stream; If so, obtain the image difference between the first reference video stream and the first video stream; Obtain the landslide distance of the mountain body and the target area monitored by the radar according to the image difference, the first reference video stream, the first video stream, the second reference video stream, and the second video stream, comprising the following steps: Obtain the three-dimensional reconstruction image of the first grayscale image as the first three-dimensional image according to the first reference video stream and the second reference video stream; Obtain the three-dimensional reconstruction image of the second grayscale image as the second three-dimensional image according to the first video stream and the second video stream; Obtain the corresponding depth value set as the first depth value set (za1, za2, …, za(m-1), zam) corresponding to the difference pixel point set according to the first three-dimensional image and the difference pixel point set; Obtain the corresponding depth value set as the second depth value set (zb1, zb2, …, zb(m-1), zbm) corresponding to the difference pixel point set according to the second three-dimensional image and the difference pixel point set; Calculate the landslide distance S of the mountain body according to the first depth value set and the second depth value set, where S = (zb1 - za1 + zb2 - za2 + zb(m-1) - za(m-1) + zbm - zam) / m, where m is a positive integer greater than 1; The step of controlling the radar to monitor the target area and sending a debris flow early warning signal according to the landslide distance and / or the monitoring result comprises the following steps: Obtain the three-dimensional coordinates of each target pixel point, where the three-dimensional coordinates of the target pixel point include the abscissa, ordinate, and depth value of the target pixel point in the second grayscale image; Obtain the relative position between the landslide area of the mountain body and the binocular camera according to the three-dimensional coordinates of each target pixel point; Obtain the relative position between the radar and the binocular camera; Obtain the target area monitored by the radar according to the relative position between the landslide area of the mountain body and the binocular camera and the relative position between the radar and the binocular camera.

2. The debris flow warning method based on images and radar according to claim 1, wherein The step of analyzing whether the change of the mountain body satisfies a preset condition according to the first reference video stream and the first video stream further comprises the following steps: Obtain the average image of the first reference video stream as the first average image; Obtain the average image of the first video stream as the second average image; Obtain the cross-correlation coefficient between the first average image and the second average image and a first threshold; If the cross-correlation coefficient is greater than the first threshold, the change of the mountain satisfies the preset condition, otherwise it does not satisfy the preset condition.

3. The debris flow warning method based on images and radar according to claim 1, wherein, The step of obtaining the image difference between the first reference video stream and the first video stream if yes further includes the following steps: Obtain a first grayscale image converted from the first average image and a second grayscale image converted from the second average image; Obtain a first image set for the n-layer pyramid image constructed for the first grayscale image; Obtain a second image set for the n-layer pyramid image constructed for the second grayscale image; Perform a subtraction process on the first image set and the second image set to obtain a difference image set; Perform low-pass filtering and magnification processing on each image in the difference image set; Overlay the images after low-pass filtering and magnification processing together to obtain a reference image with the same resolution as the first average image; Obtain a number of pixel points with pixel values greater than the set threshold in the reference image as a set of difference pixel points.

4. The debris flow warning method based on images and radar according to claim 1, characterized in that, The step of obtaining a set of depth values corresponding to the set of difference pixel points as a first set of depth values (za1, za2,..., za(m-1), zam) according to the first three-dimensional image and the set of difference pixel points further includes the following steps: Obtain a number of uniform target pixel points in the first grayscale image according to the set of difference pixel points; Obtain corresponding first target regions centered on each of the uniform target pixel points in the first grayscale image; Obtain second target regions corresponding to each of the first target regions in the first three-dimensional image; For each first target region, obtain the depth value of each pixel point in the first target region according to the corresponding second target region; For each first target region, obtain the three-dimensional coordinates (xi, yi, zi) of each pixel point in the first target region, where the xi coordinate and the yi coordinate are respectively the abscissa and ordinate of the pixel point in the first grayscale image, and the zi coordinate is the depth value of the pixel point; For each first target region, perform surface fitting on all pixel points in the first target region according to the three-dimensional coordinates of the pixel points to obtain a surface equation; Obtain the depth values of each target pixel point according to the abscissa, ordinate of each target pixel point in the first grayscale image and the corresponding surface equation, and the set of depth values of all target pixel points is used as the first set of depth values.

5. A debris flow warning device based on video images and radar, characterized in that Apply the method according to any one of claims 1 to 4, the device includes: A reference video stream acquisition module, the video stream acquisition module is used to acquire a first reference video stream obtained by the first imaging unit of the binocular camera shooting the mountain body and a second reference video stream obtained by the second imaging unit of the binocular camera shooting the mountain body within a first time period; A video stream acquisition module, the video stream acquisition module is used to acquire a first video stream obtained by the first imaging unit of the binocular camera shooting the mountain body and a second video stream obtained by the second imaging unit of the binocular camera shooting the mountain body within a second time period after the first time period; A video stream analysis module, which is used to obtain the landslide distance and the target area monitored by the radar according to the image differences among the first reference video stream, the first video stream, the second reference video stream, and the second video stream; A radar monitoring module, which is used to control the radar to monitor the target area and send a debris flow early warning signal according to the monitoring results.

6. A debris flow warning system based on video images and radar, characterized in that Comprising: A binocular camera, a radar, at least one processor, at least one memory, and computer program instructions stored in the memory. The binocular camera and the radar are respectively electrically connected to the processor. When the computer program instructions are executed by the processor, the method described in any one of claims 1-4 is implemented.

7. A storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, the method described in any one of claims 1-4 is implemented.

Images