Non-contact debris flow early warning method, device and system based on image difference
By combining binocular cameras and radar, and using image difference analysis to calculate the landslide distance, the problem of low accuracy and high cost of existing debris flow early warning technologies has been solved, realizing high-precision, low-energy-consumption, non-contact debris flow early warning.
Patent Information
- Application Number
- CN202511151970.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2022-11-12
- Publication Date
- 2025-11-11
AI Technical Summary
Existing debris flow early warning technologies have low accuracy and high cost because they need to be installed in debris flow-prone areas and are subject to interference, and require a large number of instruments for small monitoring ranges.
A non-contact early warning method based on image differences is adopted. The method uses a binocular camera to capture video streams of the mountain from different angles, calculates the mountain sliding distance through image difference analysis, and uses radar to monitor the target area and send early warning signals.
It improves the accuracy of early warning and reduces costs. By using non-contact monitoring, it reduces interference with the mountain and achieves a high-precision, low-energy-consumption early warning effect.
Smart Images
Figure CN120932382A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of geological disaster early warning technology, and in particular to a non-contact debris flow early warning method, device, and system based on image differences. Background Technology
[0002] Debris flows are a type of geological disaster that frequently occurs in mountainous areas, characterized by their explosiveness and destructive power. To avoid significant property and loss of life, it is essential to monitor debris flow hazards and provide timely warnings. Currently, devices for monitoring and issuing early warnings for debris flows include mud flow meters and mud level monitors. However, these instruments need to be installed on slopes prone to debris flows, in contact with the monitored area, making them susceptible to interference from the contact area and leading to inaccurate warnings. Furthermore, because the monitoring range of a single instrument is relatively small, a large number of instruments are required when the monitored area is extensive, resulting in high costs. Summary of the Invention
[0003] In view of this, embodiments of the present invention provide a non-contact debris flow early warning method, device, and system based on image differences, to solve the technical problem of low accuracy in existing debris flow early warning technologies.
[0004] The technical solution adopted in this invention is: In a first aspect, the present invention provides a non-contact debris flow early warning method based on image differences, the method comprising the following steps: Acquire the first reference video stream obtained by the first camera unit of the binocular camera capturing the mountain and the second reference video stream obtained by the second camera unit capturing the mountain within the first time period; Acquire the first video stream obtained by the first camera unit of the binocular camera capturing the mountain and the second video stream obtained by the second camera unit capturing the mountain during the second time period after the first time period; The landslide distance and the target area monitored by radar are obtained based on the image differences between the first reference video stream, the second reference video stream, and the second video stream. The control radar monitors the target area and sends debris flow early warning signals based on the landslide distance and / or monitoring results; The binocular camera has two camera units located in different positions, which capture images of the monitored mountain from two different angles.
[0005] Preferably, the control radar monitors the target area and sends a debris flow early warning signal based on the landslide distance and / or monitoring results, including generating and sending an early warning signal when a certain scale of change in the shape of the mountain is detected.
[0006] Preferably, the step of obtaining the landslide distance and the target area monitored by radar based on the image differences between the first reference video stream, the second reference video stream, and the second video stream further includes the following steps: Based on the first baseline video stream and the first video stream, analyze whether the changes in the mountain meet the preset conditions; If so, obtain the image difference between the first reference video stream and the first video stream; The mountain sliding distance and the target area monitored by radar are obtained based on the image differences, the first reference video stream, the first video stream, the second reference video stream, and the second video stream.
[0007] The step of analyzing whether the changes in the mountain meet the preset conditions based on the first reference video stream and the first video stream also includes the following steps: The average image of the first reference video stream is obtained as the first average image; The average image of the first video stream is used as the second average image; Obtain the cross-correlation coefficient of the first average image and the second average image, and the first threshold; If the cross-correlation coefficient is greater than the first threshold, the change in the mountain meets the preset condition; otherwise, the preset condition is not met. If so, obtaining the image difference between the first reference video stream and the first video stream further includes the following steps: Obtain a first grayscale image obtained by converting a first average image and a second grayscale image obtained by converting a second average image; The first image set is obtained by constructing an n-layer pyramid image based on the first grayscale image; The second image set is obtained by constructing an n-layer pyramid image based on the second grayscale image; Subtracting the first image set from the second image set yields the difference image set. Each image in the difference image set is low-pass filtered and magnified; The images that have undergone low-pass filtering and magnification are superimposed to obtain a reference image with the same resolution as the first average image; Obtain a set of pixels in the reference image whose pixel values are greater than a set threshold as the set of difference pixels.
[0008] Preferably, n is 5.
[0009] Preferably, the threshold value is any real number between 15 and 45.
[0010] Preferably, obtaining the landslide distance and the target area monitored by radar based on the image differences, the first reference video stream, the first video stream, the second reference video stream, and the second video stream further includes the following steps: The three-dimensional reconstructed image of the first grayscale image is obtained from the first reference video stream and the second reference video stream as the first three-dimensional image; The three-dimensional reconstructed image of the second grayscale image is obtained from the first video stream and the second video stream as the second three-dimensional image; Based on the first 3D image and the set of differing pixels, obtain the set of depth values corresponding to the set of differing pixels as the first depth value set (za1, za2, ..., za(m-1), zam); Based on the second 3D image and the set of difference pixels, obtain the set of depth values corresponding to the set of difference pixels as the second depth value set (zb1, zb2, ..., zb(m-1), zbm); The landslide distance S is calculated based on the first set of depth values and the second set of depth values, where S = (zb1-za1+zb2-za2+zb(m-1)-za(m-1)+zbm-zam) / m, and m is a positive integer greater than 1; The step of obtaining the set of depth values corresponding to the set of difference pixels as the first depth value set (za1, za2, ..., za(m-1), zam) based on the first three-dimensional image and the set of difference pixels further includes the following steps: Obtain several uniform target pixels from the first grayscale image based on the set of differing pixels; In the first grayscale image, the corresponding first target region is obtained with each uniform target pixel as the center; Obtain the second target region corresponding to each first target region in the first three-dimensional image; For each first target region, the depth value of each pixel in the first target region is obtained according to the corresponding second target region; For each first target region, obtain the three-dimensional coordinates (xi, yi, zi) of each pixel in the first target region, where the xi coordinate and the yi coordinate are the horizontal and vertical coordinates of the pixel in the first grayscale image, respectively, and the zi coordinate is the depth value of the pixel. For each first target region, a surface equation is obtained by performing surface fitting on all pixels in the first target region based on the three-dimensional coordinates of the pixel. The depth value of each target pixel is obtained based on its x-coordinate, y-coordinate and the corresponding surface equation in the first grayscale image. The set of depth values of all target pixels is used as the first depth value set.
[0011] Preferably, obtaining a plurality of uniform target pixels in the first grayscale image based on the set of differing pixels includes: Divide the first grayscale image or the second grayscale image into k rectangular regions of equal length and width; Obtain the center coordinates of each rectangular region; The search is performed with the center of the rectangular region as the center. The pixel closest to the center is found among all pixels in the set of difference pixels and is taken as the uniform target pixel of the region.
[0012] Preferably, the step of obtaining the corresponding first target region in the first grayscale image with each uniform target pixel as the center includes selecting a region with a width of w and a height of h in the first grayscale image with each target pixel as the center as the first target region, wherein the width is the length in the x direction of the first grayscale image and the height is the length in the y direction of the first grayscale image; The step of obtaining the second target region corresponding to each first target region in the first three-dimensional image includes selecting a region with a width of w and a height of h centered on the target pixel in the first three-dimensional image as the second target region corresponding to the first target region in the previous step, wherein the width is the length in the x direction of the first three-dimensional image and the height is the length in the y direction of the first three-dimensional image.
[0013] In a second aspect, the present invention also provides a debris flow early warning device based on video images and radar, the device comprising: A debris flow early warning device based on video images and radar, the device comprising: A reference video stream acquisition module is used to acquire a first reference video stream obtained by the first camera unit of the binocular camera capturing the mountain and a second reference video stream obtained by the second camera unit capturing the mountain within a first time period. The video stream acquisition module is used to acquire the first video stream obtained by the first camera unit of the binocular camera capturing the mountain and the second video stream obtained by the second camera unit capturing the mountain during a second time period after the first time period. A video stream analysis module is used to obtain the landslide distance and the target area monitored by radar based on the image differences between the first reference video stream, the second reference video stream, and the second video stream. A radar monitoring module is used to control the radar to monitor the target area and send a debris flow early warning signal based on the monitoring results.
[0014] Thirdly, the present invention also provides a debris flow early warning system based on video images and radar, comprising: a binocular camera, a radar, at least one processor, at least one memory, and computer program instructions stored in the memory, wherein the binocular camera and the radar are electrically connected to the processor, and the method described in the first aspect is implemented when the computer program instructions are executed by the processor.
[0015] Beneficial Effects: The non-contact debris flow early warning method, device, and system based on image differences of the present invention utilize the analysis of video streams from two time periods to obtain minute image differences between the two time periods. These image differences can reflect the minute sliding of the captured mountain before a debris flow eruption. The present invention calculates the distance of the mountain slide and locates the area where the slide occurred based on the image differences, and then uses radar to closely monitor this area. This allows for targeted monitoring of potential debris flow areas without the need to deploy a large number of high-precision radars to cover the entire mountain area. Therefore, it features low cost, low energy consumption, and high accuracy. Since both binocular cameras and radar can monitor the mountain in a non-contact manner, they are less affected by the mountain itself being monitored, further improving the accuracy of the early warning. Attached Figure Description
[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments of the present invention will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, and these are all within the protection scope of the present invention.
[0017] Figure 1 This is a flowchart illustrating the debris flow early warning method based on imagery and radar according to the present invention. Figure 2 This is a flowchart illustrating the method for obtaining the mountain sliding distance according to the present invention; Figure 3 This is a flowchart illustrating the method for analyzing mountain changes according to the present invention; Figure 4 This is a flowchart illustrating the method for acquiring difference images according to the present invention; Figure 5 This is a flowchart illustrating the method for calculating the sliding distance of a mountain in this invention; Figure 6 This is a flowchart illustrating the method for obtaining the first set of depth values according to the present invention. Figure 7 This is a flowchart illustrating the method for acquiring the target area monitored by radar according to the present invention. Figure 8 This is a schematic diagram of the debris flow early warning device based on video images and radar according to the present invention; Figure 9 This is a structural block diagram of the debris flow early warning system based on imagery and radar according to the present invention. Figure 10 This is a schematic diagram illustrating the process of dividing a first grayscale image into several uniform rectangular regions according to the present invention. Figure 11This is a schematic diagram illustrating the acquisition of the abnormal pixel closest to the center of the rectangular region according to the present invention. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. In the description of the present invention, it should be understood that the terms "center," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Where there is no conflict, the various features in the embodiments and examples of this invention can be combined with each other, all of which are within the scope of protection of this invention.
[0019] Example 1 like Figure 1 As shown, this embodiment provides a debris flow early warning method based on imagery and radar. The method includes the following steps: S1: Acquire the first reference video stream C1 obtained by the first camera unit of the binocular camera capturing the mountain and the second reference video stream C2 obtained by the second camera unit capturing the mountain during the first time period; The binocular camera has two camera units positioned at different locations, allowing for the capture of images of the monitored mountain from two different angles. The binocular camera has a baseline length of L and a resolution of W*H. To avoid interference from accidental factors such as wind or falling rocks, this step involves acquiring a video stream over a specific period. This acquired video stream includes multiple frames stored in chronological order. The length of the first time period can be set empirically.
[0020] S2: Obtain the first video stream C11 obtained by the first camera unit of the binocular camera capturing the mountain and the second video stream C22 obtained by the second camera unit capturing the mountain during the second time period after the first time period; This step involves using a binocular camera to capture images of the mountain at regular intervals, obtaining a video stream over a given period. The acquired video stream includes multiple frames stored in chronological order. The intervals and the length of the first time period can be set empirically. As an optional but advantageous implementation, the first and second time periods are equal.
[0021] S3: Obtain the landslide distance and the target area monitored by radar based on the image differences between the first reference video stream, the second reference video stream, and the second video stream; Before a mudslide, minor landslides often occur. This step utilizes the analysis of video streams from two time periods to capture minute image differences between them. These differences reflect subtle landslides captured on camera. Therefore, this step can calculate the distance of the landslide and locate the affected area based on these image differences. This area is then closely monitored using radar, providing early warning before a sudden mudslide erupts.
[0022] S4: Control the radar to monitor the target area and send a debris flow warning signal based on the landslide distance and / or monitoring results.
[0023] This embodiment utilizes a binocular camera to continuously photograph the mountainside. Once an area exhibiting minor landslides is identified, the radar is controlled to focus on monitoring that area. When a significant change in the mountain's shape is detected within a short period, a warning signal is generated and transmitted. This avoids deploying numerous high-precision radars to monitor the entire mountain area, thus saving energy and reducing costs while ensuring accurate and timely warnings. The radar can be an X-band radar to monitor the movement of rocks within the target area. A warning signal is sent when the detected rock movement speed exceeds a preset speed. Alternatively, this embodiment can send a warning signal when the calculated landslide distance exceeds a preset distance. Furthermore, this embodiment can calculate the landslide speed based on the landslide distance and the interval T between the first and second time periods. Let the landslide distance be S, then the landslide speed v = T / S. A warning signal can also be sent when the landslide speed exceeds an allowable speed.
[0024] like Figure 2As shown, as an optional but advantageous implementation, in this embodiment, step S3: obtaining the landslide distance and the target area monitored by radar based on the image differences between the first reference video stream, the second reference video stream, and the second video stream, further includes the following steps: S31: Analyze whether the changes in the mountain meet the preset conditions based on the first reference video stream and the first video stream; This step determines whether a small landslide occurred before a debris flow occurred. If the landslide is large, it indicates that a debris flow may have already occurred. Since this embodiment is mainly used for early warning before a debris flow occurs, it is mainly applicable to the analysis and processing of small landslides. In order to provide the accuracy of the analysis, this step first analyzes the degree of change of the mountain and selects the cases where the degree of change of the mountain meets the requirements for analysis and processing.
[0025] like Figure 3 As shown, as an optional but advantageous implementation, in this embodiment, S31: analyzing whether the changes in the mountain meet the preset conditions based on the first reference video stream and the first video stream further includes the following steps: S311: Obtain the average image of the first reference video stream as the first average image; This step can use a multi-image averaging method to process multiple frames of the first reference video stream to obtain an average image, thereby reducing noise caused by external interference factors such as falling rocks and wind.
[0026] S312: Obtain the average image of the first video stream as the second average image; This step can use the multi-image averaging method to process multiple frames of the first video stream to obtain an average image, thereby reducing noise caused by external interference factors such as falling rocks and wind.
[0027] S313: Obtain the cross-correlation coefficient of the first average image and the second average image, and the first threshold.
[0028] The first threshold is the maximum value that the cross-correlation coefficient cannot exceed, and this value can be determined based on experience.
[0029] S314: If the cross-correlation coefficient is greater than the first threshold, the change of the mountain meets the preset condition; otherwise, the preset condition is not met.
[0030] For example, if the cross-correlation coefficient is e and the first threshold is E, then if e > E, it indicates that the landslide belongs to the category of minor landslides and the change in the mountain meets the preset conditions; otherwise, it indicates that the landslide does not belong to the category of minor landslides and the change in the mountain does not meet the aforementioned preset conditions. As an optional but advantageous implementation method, E = 0.95.
[0031] S32: If yes, then obtain the image difference between the first reference video stream and the first video stream; This step identifies image differences between video streams captured at two different times, assuming the landslide is minor. Image differences refer to pixels in the image that have changed significantly.
[0032] like Figure 4 As shown, as an optional but advantageous implementation, in this embodiment, step S32: if so, obtaining the image difference between the first reference video stream and the first video stream further includes the following steps: S321: Obtain the first grayscale image obtained by converting the first average image and the second grayscale image obtained by converting the second average image; S322: The first image set is obtained by constructing an n-layer pyramid image based on the first grayscale image; Where n is a positive integer greater than 1, and as an optional but advantageous implementation, n is 5. The n-layer pyramid image constructed from the first grayscale image has a total of n images, and the set of these n images is the first image set.
[0033] S323: The second image set is obtained by constructing an n-layer pyramid image based on the second grayscale image; The n-layer pyramid image constructed from the second grayscale image has a total of n images, and the set of these n images is the second image set.
[0034] The n-layer pyramid image is composed of a series of images, with the bottom image being the largest and the top image being the smallest. S324: Subtract the first image set from the second image set to obtain the difference image set; This step involves subtracting the first layer image from the first image set to obtain the difference image of the first layer, subtracting the second layer image from the first image set to obtain the difference image of the second layer, and so on, until the nth layer image from the first image set is obtained. All difference images from layers 1 to n are then combined into a difference image set. Subtracting two images means subtracting the pixel values of corresponding pixels (pixels with the same horizontal and vertical coordinates in both images) and taking the absolute value. This absolute value is used as the pixel value of the corresponding pixels (pixels with the same horizontal and vertical coordinates) in the difference image.
[0035] S325: Perform low-pass filtering and magnification on each image in the differential image set; This step can process each image in the difference set separately, first by performing low-pass filtering and then by magnification.
[0036] S326: The images after low-pass filtering and magnification are superimposed to obtain a reference image with the same resolution as the first average image; After low-pass filtering and magnification of all n images in the difference image set, the n images are restored to their original size before the pyramid image was constructed. Then, these n processed images are superimposed to restore the original resolution before the pyramid image was constructed.
[0037] S327: Obtain a set of several pixels in the reference image whose pixel values are greater than a set threshold as the set of difference pixels.
[0038] The threshold value can be determined empirically. As an optional implementation, the threshold value can be any real number between 15 and 45. For pixels in the reference image whose pixel values are greater than the threshold value, the horizontal and vertical coordinates of the pixel in the reference image can be obtained. Pixels in the first grayscale image and the second grayscale image with the same horizontal and vertical coordinates are respectively the difference pixels in the first grayscale image and the difference pixels in the second grayscale image. S33: Obtain the mountain sliding distance and the target area monitored by radar based on the image differences, the first reference video stream, the first video stream, the second reference video stream, and the second video stream.
[0039] like Figure 5 As shown, the method for obtaining the mountain sliding distance mainly includes the following steps: S331: Obtain the three-dimensional reconstructed image of the first grayscale image based on the first reference video stream and the second reference video stream as the first three-dimensional image; Since this embodiment uses a binocular camera to simultaneously capture images of the mountain from different angles, a three-dimensional image of the mountain can be created using the first and second reference video streams captured by the two camera units of the binocular camera. To avoid noise interference, a grayscale image obtained by converting the average image of the first and second reference video streams can be used to create the three-dimensional image. The method for creating a three-dimensional image using images captured by the binocular camera can employ existing technology.
[0040] S332: Obtain the three-dimensional reconstructed image of the second grayscale image based on the first video stream and the second video stream as the second three-dimensional image; Similarly, in this embodiment, two video streams captured by a binocular camera in the second time period are used to create a three-dimensional image. The method is the same as the previous step and will not be repeated here.
[0041] S333: Based on the first three-dimensional image and the set of difference pixels, obtain the set of depth values corresponding to the set of difference pixels as the first depth value set (za1, za2, ..., za(m-1), zam); This step uses the first 3D image to obtain the depth value of each difference pixel in the difference pixel set in the first 3D image, and combines these depth values into a first depth value set. An element in the set represents the depth value corresponding to a difference pixel. For example, za1 represents the depth value of the first difference pixel, za2 represents the depth value of the second difference pixel, za(m-1) represents the depth value of the (m-1)th difference pixel, and zam represents the depth value of the mth difference pixel.
[0042] like Figure 6 As shown, in a preferred embodiment, S333 in this embodiment, which involves obtaining a set of depth values corresponding to the set of difference pixels as a first depth value set (za1, za2, ..., za(m-1), zam), further includes the following steps: S3331: Obtain several uniform target pixels in the first grayscale image based on the set of difference pixels; S3332: In the first grayscale image, obtain the corresponding first target region centered on each uniform target pixel; In the first grayscale image, a region with a width of w and a height of h is selected centered on each target pixel, where the width is the length in the x-direction of the first grayscale image and the height is the length in the y-direction of the first grayscale image; S3333: Obtain the second target region corresponding to each first target region in the first three-dimensional image; In the first 3D image, a region with a width of w and a height of h is selected, centered on the target pixel. This region is the second target region corresponding to the first target region in the previous step. The width is the length in the x-direction of the first 3D image, and the height is the length in the y-direction of the first 3D image. S3334: For each first target region, obtain the depth value of each pixel in the first target region according to the corresponding second target region; Input the two-dimensional coordinates of all pixels in the first target region into the three-dimensional image, and obtain the third coordinate corresponding to the two-dimensional coordinates in the three-dimensional image. The third coordinate is the depth value of the pixel.
[0043] For example, the coordinates of the i-th pixel in the first target region are (xi, yi), where xi is the horizontal coordinate and yi is the vertical coordinate. That is, the i-th pixel is located in the xi-th column and yi-th row of the first grayscale image. In the first 3D image, the 3D coordinates of the pixel with horizontal coordinate xi and vertical coordinate yi are (xi, yi, zi), where zi is the depth value of the i-th pixel in the first target region. Since each difference pixel corresponds to a first target region, the aforementioned operation can be performed on each of the first target regions one by one.
[0044] S3335: For each first target region, obtain the three-dimensional coordinates (xi, yi, zi) of each pixel in the first target region, where the xi coordinate and yi coordinate are the horizontal and vertical coordinates of the pixel in the first grayscale image, respectively, and the zi coordinate is the depth value of the pixel.
[0045] S3336: For each first target region, perform surface fitting on all pixels in the first target region based on the three-dimensional coordinates of the pixel to obtain the surface equation. This step involves creating a surface for all pixels in the first target region based on their 3D coordinates. Since there are multiple first target regions, the aforementioned operation can be performed on each of these regions individually to obtain the surface equation corresponding to each first target region.
[0046] S3337: Obtain the depth value of each target pixel based on its horizontal and vertical coordinates in the first grayscale image and the corresponding surface equation. The set of depth values of all target pixels is used as the first depth value set.
[0047] This step substitutes the x-coordinate and y-coordinate of the target pixel in the first grayscale image into the surface equation corresponding to the first target region where the target pixel is located, to obtain the depth value of the target pixel, and uses all the obtained depth values of the target pixels as the first depth value set.
[0048] S334: Based on the second three-dimensional image and the set of difference pixels, obtain the set of depth values corresponding to the set of difference pixels as the second depth value set (zb1, zb2, ..., zb(m-1), zbm); Similarly, this step uses the second 3D image to obtain the depth value of each difference pixel in the difference pixel set in the second 3D image, and combines these depth values into a second depth value set. An element in the set represents the depth value corresponding to a difference pixel. For example, zb1 represents the depth value of the first difference pixel, zb2 represents the depth value of the second difference pixel, zb(m-1) represents the depth value of the (m-1)th difference pixel, and zbm represents the depth value of the mth difference pixel.
[0049] As a preferred embodiment, in this embodiment, S334: obtaining the set of depth values corresponding to the set of difference pixels as the second depth value set (zb1, zb2, ..., zb(m-1), zbm) based on the second three-dimensional image and the set of difference pixels further includes the following steps: S3341: Obtain several uniform target pixels in the second grayscale image based on the set of difference pixels; S3342: In the second grayscale image, obtain the corresponding first target region centered on each uniform target pixel; in the second grayscale image, select a region with width w and height h centered on each target pixel, where the width is the length in the x direction of the second grayscale image and the height is the length in the y direction of the second grayscale image. S3343: Obtain the second target region corresponding to each first target region in the second three-dimensional image; In the second 3D image, a region with a width of w and a height of h is selected, centered on the target pixel. This region is the second target region corresponding to the first target region in the previous step. The width is the length in the x-direction of the second 3D image, and the height is the length in the y-direction of the second 3D image. S3344: For each first target region, obtain the depth value of each pixel in the first target region according to the corresponding second target region; input the two-dimensional coordinates of all pixels in the first target region into the three-dimensional image, and obtain the third coordinate corresponding to the two-dimensional coordinates in the three-dimensional image. The third coordinate is the depth value of the pixel.
[0050] For example, the coordinates of the i-th pixel in the first target region are (xi, yi), where xi is the horizontal coordinate and yi is the vertical coordinate. That is, the i-th pixel is located in the xi-th column and yi-th row of the second grayscale image. In the first 3D image, the 3D coordinates of the pixel with horizontal coordinate xi and vertical coordinate yi are (xi, yi, zi), where zi is the depth value of the i-th pixel in the first target region. Since each difference pixel corresponds to a first target region, the aforementioned operation can be performed on each of the first target regions one by one.
[0051] S3345: For each first target region, obtain the three-dimensional coordinates (xi, yi, zi) of each pixel in the first target region, where the xi coordinate and yi coordinate are the horizontal and vertical coordinates of the pixel in the second grayscale image, respectively, and the zi coordinate is the depth value of the pixel.
[0052] S3346: For each first target region, perform surface fitting on all pixels in the first target region based on the three-dimensional coordinates of the pixels to obtain the surface equation; this step involves performing surface fitting on all pixels in the first target region based on the three-dimensional coordinates of all pixels in the first target region. Since there are multiple first target regions, the aforementioned operation can be performed on each of these first target regions one by one to obtain the surface equation corresponding to each first target region.
[0053] S3347: Obtain the depth value of each target pixel based on its x-coordinate, y-coordinate, and the corresponding surface equation in the first grayscale image. The set of depth values for all target pixels is then used as the second depth value set. This step substitutes the x-coordinate and y-coordinate of the target pixel in the first grayscale image into the surface equation corresponding to the first target region where the target pixel is located to obtain its depth value. All obtained depth values of the target pixels are then used as the second depth value set.
[0054] like Figure 10 As shown, the aforementioned uniform target pixel refers to the difference pixel closest to the center of each region after the first grayscale image or the second grayscale image is divided into multiple regions. This can be achieved by first uniformly dividing the first grayscale image or the second grayscale image into k rectangular regions of equal length and width, and then obtaining the center coordinates of each rectangular region. Figure 10 The small circles in the diagram represent the center coordinates of each region, while the large hollow circles represent abnormal pixels. For example... Figure 11 As shown, the closest difference pixel to the center coordinate within the rectangular area is then selected as the uniform target pixel. In other words, the search can be performed with the center of the rectangular area as the center, and the closest difference pixel to the center can be found from all pixels in the set of difference pixels as the uniform target pixel for that area. Figure 11 The solid circle represents a uniform target pixel selected from the difference pixels. Since the reference image, the first grayscale image, and the second grayscale image have the same resolution, the x and y coordinates of the difference pixels in the reference image can be used as the x and y coordinates of the difference pixels in the first and second grayscale images (i.e., difference pixels can be represented by pixels in the reference image, the first grayscale image, and the second grayscale image, and the same difference pixel has the same x and y coordinates in the aforementioned three images), to obtain the difference pixel in the first or second grayscale image that is closest to the divided distance region as the target pixel. The aforementioned operation is performed on each divided rectangular region to obtain the same number of target pixels as the divided rectangular region.
[0055] In addition to obtaining the depth value of the target pixel using the aforementioned method, a radar point cloud map of the mountain surface can also be generated using radar. Then, the target pixel is registered with the radar point cloud map, and the depth value of the target pixel is obtained by the position of the registered target pixel in the radar point cloud map.
[0056] S335: Calculate the landslide distance S based on the first depth value set and the second depth value set, where S = (zb1 - za1 + zb2 - za2 + zb(m-1) - za(m-1) + zbm - zam) / m, where m is a positive integer greater than 1. In this embodiment, S33: Obtaining the landslide distance and the target area monitored by radar based on the image differences, the first reference video stream, the first video stream, the second reference video stream, and the second video stream further includes the following steps: S336: Obtain the three-dimensional coordinates of each target pixel, wherein the three-dimensional coordinates of the target pixel include the horizontal coordinate, vertical coordinate and depth value of the target pixel in the second grayscale image; S337: The relative position between the landslide area and the binocular camera is obtained based on the 3D coordinates of each target pixel. After the binocular camera is installed at the monitoring location, its parameters are calibrated, and a 3D image is reconstructed using the binocular camera. The 3D coordinates of each target pixel in the camera coordinate system are obtained using the 2D coordinates of the target pixels and the reconstructed 3D image. That is, the 2D coordinates of the target pixels are substituted into the 3D image to obtain the 3D coordinates of the target pixels. The landslide area corresponding to the target pixel is taken as the landslide area. The relative position between the landslide area corresponding to the target pixel and the binocular camera can be found using the aforementioned 3D coordinates.
[0057] S338: Obtain the relative position between the radar and the binocular camera; after both the radar and the binocular camera are installed at the designated location, the relative positional relationship between the two is determined.
[0058] S339: Obtain the target area monitored by the radar based on the relative position between the landslide area and the binocular camera, and the relative position between the radar and the binocular camera. This step can be achieved by transforming the landslide area from the camera coordinate system of the binocular camera to the radar coordinate system, thereby obtaining the coordinates of the landslide area in the radar coordinate system.
[0059] Example 2 Please see Figure 8 This embodiment provides a debris flow early warning device based on video images and radar. The device includes: A reference video stream acquisition module is used to acquire a first reference video stream obtained by the first camera unit of the binocular camera capturing the mountain and a second reference video stream obtained by the second camera unit capturing the mountain within a first time period. The video stream acquisition module is used to acquire the first video stream obtained by the first camera unit of the binocular camera capturing the mountain and the second video stream obtained by the second camera unit capturing the mountain during a second time period after the first time period. A video stream analysis module is used to obtain the landslide distance and the target area monitored by radar based on the image differences between the first reference video stream, the second reference video stream, and the second video stream. A radar monitoring module is included, which controls the radar to monitor the target area and sends a debris flow early warning signal based on the monitoring results. The module for obtaining the direction of travel and pedestrian location also includes: The video stream analysis module further includes: a condition verification submodule, which is used to analyze whether the changes in the mountain meet preset conditions based on the first reference video stream and the first video stream; an image difference acquisition submodule, which is used to acquire the image difference between the first reference video stream and the first video stream if the conditions are met; and a sliding distance and target area acquisition submodule, which is used to acquire the mountain sliding distance and the radar-monitored target area based on the image difference, the first reference video stream, the first video stream, the second reference video stream, and the second video stream.
[0060] Example 3 In addition, combined Figure 9 The debris flow early warning method based on imagery and radar described in the foregoing embodiments of the present invention can be implemented by the smart pole of this embodiment. Figure 9 A schematic diagram of a debris flow early warning system based on video images and radar provided in an embodiment of the present invention is shown.
[0061] The smart pole in this embodiment may include a processing circuit 401, a binocular camera 404, a radar 405, and a memory 402 storing computer program instructions.
[0062] Specifically, the processing circuit 401 described above may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.
[0063] Memory 402 may include mass storage for data or instructions. For example, and not limitingly, memory 402 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 402 may include removable or non-removable (or fixed) media. Where appropriate, memory 402 may be internal or external to a data processing device. In a particular embodiment, memory 402 is a non-volatile solid-state memory. In a particular embodiment, memory 402 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.
[0064] The processor 401 reads and executes computer program instructions stored in the memory 402 to implement any of the regional random smart pole data addressing methods in the above embodiments.
[0065] In one example, the smart pole of this embodiment may also include a communication interface 403 and a bus 410. Wherein, as... Figure 9 As shown, the processing circuit 401, memory 402, communication interface 403, binocular camera 404, and radar 405 are connected through bus 410 and complete mutual communication.
[0066] The communication interface 403 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of the present invention.
[0067] Bus 410 includes hardware, software, or both, that couples the various components used for the smart pole together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 410 may include one or more buses. While specific buses are described and illustrated in embodiments of the invention, the invention contemplates any suitable bus or interconnect.
[0068] Example 4 Furthermore, in conjunction with the image- and radar-based debris flow early warning methods described in the above embodiments, this invention can be implemented using a computer-readable storage medium. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the image- and radar-based debris flow early warning methods described in the above embodiments.
[0069] The above is a detailed description of the debris flow early warning method, device, equipment and storage medium based on image and radar provided in the embodiments of the present invention.
[0070] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.
[0071] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.
[0072] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.
[0073] The above description is merely a specific embodiment of the present invention. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the protection scope of the present invention.
Claims
1. A non-contact debris flow early warning method based on image differences, the method comprising the following steps: Acquire the first reference video stream obtained by the first camera unit of the binocular camera capturing the mountain and the second reference video stream obtained by the second camera unit capturing the mountain within the first time period; Acquire the first video stream obtained by the first camera unit of the binocular camera capturing the mountain and the second video stream obtained by the second camera unit capturing the mountain during the second time period after the first time period; The landslide distance and the target area monitored by radar are obtained based on the image differences between the first reference video stream, the second reference video stream, and the second video stream. The control radar monitors the target area and sends debris flow early warning signals based on the landslide distance and / or monitoring results; The binocular camera has two camera units located in different positions, which capture images of the monitored mountain from two different angles.
2. The non-contact debris flow early warning method based on image differences according to claim 1, characterized in that, The control radar monitors the target area and sends debris flow early warning signals based on the landslide distance and / or monitoring results, including generating and sending early warning signals when a certain scale of change in the shape of the mountain is detected.
3. The non-contact debris flow early warning method based on image differences according to claim 1, characterized in that, The process of obtaining the landslide distance and the target area monitored by radar based on the image differences between the first reference video stream, the second reference video stream, and the second video stream further includes the following steps: Based on the first baseline video stream and the first video stream, analyze whether the changes in the mountain meet the preset conditions; If so, obtain the image difference between the first reference video stream and the first video stream; The mountain sliding distance and the target area monitored by radar are obtained based on the image differences, the first reference video stream, the first video stream, the second reference video stream, and the second video stream. The step of analyzing whether the changes in the mountain meet the preset conditions based on the first reference video stream and the first video stream also includes the following steps: The average image of the first reference video stream is obtained as the first average image; The average image of the first video stream is used as the second average image; Obtain the cross-correlation coefficient of the first average image and the second average image, and the first threshold; If the cross-correlation coefficient is greater than the first threshold, the change in the mountain meets the preset condition; otherwise, the preset condition is not met. If so, obtaining the image difference between the first reference video stream and the first video stream further includes the following steps: Obtain a first grayscale image obtained by converting a first average image and a second grayscale image obtained by converting a second average image; The first image set is obtained by constructing an n-layer pyramid image based on the first grayscale image; The second image set is obtained by constructing an n-layer pyramid image based on the second grayscale image; Subtracting the first image set from the second image set yields the difference image set. Each image in the difference image set is low-pass filtered and magnified; The images that have undergone low-pass filtering and magnification are superimposed to obtain a reference image with the same resolution as the first average image; Obtain a set of pixels in the reference image whose pixel values are greater than a set threshold as the set of difference pixels.
4. The non-contact debris flow early warning method based on image differences according to claim 3, characterized in that, The value of n is 5.
5. The non-contact debris flow early warning method based on image differences according to claim 4, characterized in that, The threshold value is any real number between 15 and 45.
6. The non-contact debris flow early warning method based on image differences according to claim 3, characterized in that, Obtaining the landslide distance and the target area monitored by radar based on the image differences, the first reference video stream, the first video stream, the second reference video stream, and the second video stream further includes the following steps: The three-dimensional reconstructed image of the first grayscale image is obtained from the first reference video stream and the second reference video stream as the first three-dimensional image; The three-dimensional reconstructed image of the second grayscale image is obtained from the first video stream and the second video stream as the second three-dimensional image; Based on the first 3D image and the set of differing pixels, obtain the set of depth values corresponding to the set of differing pixels as the first depth value set (za1, za2, ..., za(m-1), zam); Based on the second 3D image and the set of difference pixels, obtain the set of depth values corresponding to the set of difference pixels as the second depth value set (zb1, zb2, ..., zb(m-1), zbm); The landslide distance S is calculated based on the first set of depth values and the second set of depth values, where S = (zb1-za1+zb2-za2+zb(m-1)-za(m-1)+zbm-zam) / m, and m is a positive integer greater than 1; The step of obtaining the set of depth values corresponding to the set of difference pixels as the first depth value set (za1, za2, ..., za(m-1), zam) based on the first three-dimensional image and the set of difference pixels further includes the following steps: Obtain several uniform target pixels from the first grayscale image based on the set of differing pixels; In the first grayscale image, the corresponding first target region is obtained with each uniform target pixel as the center; Obtain the second target region corresponding to each first target region in the first three-dimensional image; For each first target region, the depth value of each pixel in the first target region is obtained according to the corresponding second target region; For each first target region, obtain the three-dimensional coordinates (xi, yi, zi) of each pixel in the first target region, where the xi coordinate and the yi coordinate are the horizontal and vertical coordinates of the pixel in the first grayscale image, respectively, and the zi coordinate is the depth value of the pixel. For each first target region, a surface equation is obtained by performing surface fitting on all pixels in the first target region based on the three-dimensional coordinates of the pixel. The depth value of each target pixel is obtained based on its x-coordinate, y-coordinate and the corresponding surface equation in the first grayscale image. The set of depth values of all target pixels is used as the first depth value set.
7. The non-contact debris flow early warning method based on image differences according to claim 6, characterized in that, The step of obtaining several uniform target pixels in the first grayscale image based on the set of difference pixels includes: Divide the first grayscale image or the second grayscale image into k rectangular regions of equal length and width; Obtain the center coordinates of each rectangular region; The search is performed with the center of the rectangular region as the center. The pixel closest to the center is found among all pixels in the set of difference pixels and is taken as the uniform target pixel of the region.
8. The non-contact debris flow early warning method based on image differences according to claim 6, characterized in that, The step of obtaining the corresponding first target region in the first grayscale image with each uniform target pixel as the center includes selecting a region with a width of w and a height of h as the center in the first grayscale image with each target pixel as the center, wherein the width is the length in the x direction of the first grayscale image and the height is the length in the y direction of the first grayscale image; The step of obtaining the second target region corresponding to each first target region in the first three-dimensional image includes selecting a region with a width of w and a height of h centered on the target pixel in the first three-dimensional image as the second target region corresponding to the first target region in the previous step, wherein the width is the length in the x direction of the first three-dimensional image and the height is the length in the y direction of the first three-dimensional image.
9. A debris flow early warning device based on video images and radar, characterized in that, The device includes: A reference video stream acquisition module is used to acquire a first reference video stream obtained by the first camera unit of the binocular camera capturing the mountain and a second reference video stream obtained by the second camera unit capturing the mountain within a first time period. The video stream acquisition module is used to acquire the first video stream obtained by the first camera unit of the binocular camera capturing the mountain and the second video stream obtained by the second camera unit capturing the mountain during a second time period after the first time period. A video stream analysis module is used to obtain the landslide distance and the target area monitored by radar based on the image differences between the first reference video stream, the second reference video stream, and the second video stream. A radar monitoring module is used to control the radar to monitor the target area and send a debris flow early warning signal based on the monitoring results.
10. A debris flow early warning system based on video images and radar, characterized in that, include: A binocular camera, a radar, and at least one processor, at least one memory, and computer program instructions stored in the memory, wherein the binocular camera and the radar are electrically connected to the processor, and the computer program instructions are executed by the processor to implement the method as described in any one of claims 1-8.