Debris flow monitoring method, system and equipment, storage medium and product

By combining the target detection model and dense optical flow algorithm to calculate the properties of the mudslide, the problems of high false alarm rate and difficulty in judging scale of traditional monitoring methods are solved, and more accurate mudslide monitoring and early warning are achieved.

CN119006841AInactive Publication Date: 2024-11-22HUNAN NOVASKY ELECTRONICS TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411085216.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional mudslide monitoring methods are prone to false alarms and cannot accurately judge the scale of mudslides, especially based on radar and video monitoring methods, there are problems of insufficient resolution and high false alarm rates.

Method used

Using vision-based mudslide monitoring method, combined with the target detection model and dense optical flow algorithm, the flow direction, flow velocity, overflow width and solid particle size of mudslide flow are calculated, and the mudslide flow is realized. Scale judgment and early warning.

Benefits of technology

It reduces false positives, improves the accuracy of mudslide monitoring, can accurately detect target types and moving areas, make full use of the advantages of the target detection model and optical flow information extraction algorithm, and achieves accurate judgment and early warning of mudslide scale.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006841A_ABST
    Figure CN119006841A_ABST
Patent Text Reader

Abstract

The invention discloses a debris flow monitoring method, system and device, a storage medium and a product, and the monitoring method comprises the steps: carrying out the debris flow target detection of each frame of to-be-monitored region image, and obtaining a target detection frame; dense optical flow extraction is carried out on two adjacent frames of to-be-monitored area images to obtain a debris flow movement area; wherein the debris flow motion area comprises a plurality of optical flow feature points; when the target detection frame is not overlapped with the debris flow movement area, there is no debris flow risk; and when the target detection frame is overlapped with the debris flow movement area, determining that a debris flow risk exists, and calculating debris flow attributes according to the debris flow movement area. According to the method, the target detection model and the optical flow information extraction algorithm are combined, and the problem of misinformation of debris flow monitoring is effectively reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a debris flow monitoring method, system, equipment, storage medium and product. Background Art

[0002] Traditional debris flow monitoring methods are carried out through geological surveys, meteorological monitoring, manual inspections, etc., which consume a lot of manpower and material resources. In recent years, with the development of science and technology, debris flow monitoring based on sensors such as radar and cameras has gradually become one of the main monitoring and early warning methods.

[0003] At present, although the radar-based debris flow monitoring method can meet the usage requirements of some scenarios, it has great limitations because of the limited resolution of the radar and cannot provide intuitive information such as color and texture, resulting in the inability to intuitively monitor the debris flow site area. At the same time, the radar is easily affected by moving targets such as on-site personnel, vehicles, and animals, and cannot accurately locate the target type, which can easily cause false alarms.

[0004] Most of the video-based debris flow monitoring methods use background modeling to separate the target foreground and determine whether a debris flow exists, thereby providing debris flow warnings. Although this method can accurately identify the moving area in the video, it is impossible to determine whether the moving area is caused by a debris flow or other targets, and it is easily affected by changes in lighting and other moving targets, resulting in false alarms. It has great limitations and is difficult to correctly monitor the occurrence of debris flows. At the same time, most of these methods are limited to whether a debris flow exists, and there is no calculation of the debris flow properties, so it is impossible to monitor the debris flow properties and scale. This type of method causes a large number of false alarms in scenes such as rain and small amounts of water flow.

[0005] For example, the Chinese patent application with publication number CN117152463A and titled "Debris flow tap monitoring method based on video analysis" discloses that the actual movement area of ​​the debris flow is calculated and segmented by image fusion, and whether it is a tap is determined based on the change in the optical flow area that breaks into the pre-set ROI area. Although this method can calculate the tap flow rate of the debris flow through segmentation and optical flow extraction methods, this method cannot determine the type of debris flow, and this method does not calculate the flow direction, flow width, solid particle size and other attributes of the debris flow, so it cannot accurately determine the scale of the debris flow. Summary of the invention

[0006] The purpose of the present invention is to provide a debris flow monitoring method, system, equipment, storage medium and product to solve the problem that traditional monitoring technology is prone to false alarms and cannot accurately determine the scale of debris flows.

[0007] The present invention solves the above technical problems through the following technical solutions: a debris flow monitoring method, comprising:

[0008] Perform debris flow target detection on each frame of the image of the area to be monitored to obtain a target detection frame;

[0009] Performing dense optical flow extraction on two adjacent frames of images of the area to be monitored to obtain a debris flow movement area; wherein the debris flow movement area includes a plurality of optical flow feature points;

[0010] When the target detection frame does not overlap with the debris flow movement area, there is no debris flow risk;

[0011] When the target detection frame overlaps with the debris flow movement area, there is a debris flow risk, and the debris flow attributes are calculated according to the debris flow movement area.

[0012] Furthermore, debris flow target detection is performed on each frame of the image of the area to be monitored, including:

[0013] Calling a target detection model, and using the target detection model to perform debris flow target detection on each frame of the image of the area to be monitored;

[0014] Among them, the target detection model adopts a deep learning neural network, which includes an input layer, a backbone network, a connection network and a detection head output layer; the input layer is used to preprocess the image of the monitored area; the backbone network is used to extract features of the preprocessed image to obtain feature maps of different scales; the connection network is used to fuse and enhance features of feature maps of different scales to obtain the final feature map; the detection head output layer is used to generate a target detection frame according to the final feature map.

[0015] Furthermore, the connection network includes a feature pyramid network, a path aggregation network, a convolution layer and a pooling layer; the feature pyramid network is used to fuse feature maps of different scales to obtain a fused feature map; the path aggregation network is used to perform multi-scale feature enhancement on the fused feature map; the convolution layer and the pooling layer are used to perform feature processing and adjustment on the feature map after feature enhancement so that the size and number of channels of the feature map meet the requirements of the output layer of the detection head.

[0016] Furthermore, the Farneback dense optical flow algorithm is used to extract dense optical flow from two adjacent frames of images of the monitored area to obtain the debris flow movement area.

[0017] Further, the debris flow attributes include the flow direction and flow velocity of the debris flow; wherein calculating the flow direction and flow velocity of the debris flow includes:

[0018] Construct a transformation matrix between the world coordinate system and the pixel coordinate system of the image of the area to be monitored;

[0019] Determine the pixel coordinates of each optical flow feature point in the debris flow motion area in the corresponding two frames of images of the area to be monitored;

[0020] According to the conversion matrix, the pixel coordinates of each optical flow feature point in the corresponding two frames of the monitored area image are converted into the world coordinate system to obtain two physical position coordinates of each optical flow feature point;

[0021] Define the 0° direction in the image of the area to be monitored, and calculate the angle between the line connecting the two physical position coordinates of each optical flow feature point and the 0° direction;

[0022] Calculate the average value of all angles to get the direction of the debris flow;

[0023] Calculate the distance between the two physical position coordinates of each optical flow feature point, and calculate the speed of each optical flow feature point according to the distance and the frame extraction interval time;

[0024] Calculate the average velocity of all optical flow feature points to get the velocity of the debris flow.

[0025] Furthermore, the construction of a conversion matrix between the world coordinate system and the pixel coordinate system of the image of the area to be monitored includes:

[0026] Select any point P1 in the image of the area to be monitored and obtain the pixel coordinates of point P1;

[0027] Find the point W1 corresponding to the point P1 in the world coordinate system, and measure the physical distance between the point W1 and the visual sensor that collects the image of the area to be monitored in the horizontal direction and the radial direction to obtain the physical position coordinates of the point W1;

[0028] A set of calibration points is formed by point P1 and point W1;

[0029] Repeat the steps of obtaining a set of calibration points to obtain multiple sets of calibration points;

[0030] The conversion matrix between the world coordinate system and the pixel coordinate system of the image of the monitored area is calculated based on multiple sets of calibration points.

[0031] Furthermore, the debris flow attribute also includes the flow width of the debris flow; wherein calculating the flow width of the debris flow includes:

[0032] The direction perpendicular to the flow direction of the debris flow is taken as the reference direction;

[0033] The Euclidean distance between the two optical flow feature points with the largest distance in the reference direction of the debris flow movement area is calculated to obtain the maximum flow width of the debris flow.

[0034] Furthermore, the debris flow attribute also includes a particle size of solid matter in the debris flow; wherein calculating the particle size of solid matter in the debris flow includes:

[0035] Performing binary segmentation on the debris flow movement area to obtain a binary image;

[0036] Performing contour extraction on the binary image to obtain the contours of objects in the debris flow movement area;

[0037] Calculate the distance between the two optical flow feature points with the largest distance in each object contour in the world coordinate system to obtain the particle size of the object contour;

[0038] The average particle size of all object contours is calculated to obtain the particle size of solid objects in the debris flow.

[0039] Furthermore, the monitoring method also includes providing different levels of early warning according to the properties of the debris flow, which specifically includes:

[0040] If each debris flow attribute is less than the corresponding first threshold, no debris flow warning is issued;

[0041] If the corresponding first threshold value ≤ at least one debris flow attribute < the corresponding second threshold value, a debris flow mild warning is issued;

[0042] If the corresponding second threshold value ≤ at least one debris flow attribute < the corresponding third threshold value, a moderate debris flow warning is issued;

[0043] If the corresponding third threshold is ≤ at least one debris flow attribute, a severe debris flow warning is issued.

[0044] Based on the same concept, the present invention also provides a debris flow monitoring system, comprising:

[0045] The target detection unit is used to detect debris flow targets for each frame of the image of the area to be monitored to obtain a target detection frame;

[0046] A motion region extraction unit is used to perform dense optical flow extraction on two adjacent frames of images of the area to be monitored to obtain a debris flow motion region; wherein the debris flow motion region includes a plurality of optical flow feature points;

[0047] A judging unit, used for judging whether the target detection frame overlaps with the debris flow movement area;

[0048] The attribute calculation unit is used to calculate the debris flow attribute according to the debris flow movement area when the target detection frame overlaps with the debris flow movement area.

[0049] Based on the same concept, the present invention also provides an electronic device, including a memory, a processor, and a computer program / instruction stored in the memory, wherein the processor executes the computer program / instruction to implement the debris flow monitoring method as described above.

[0050] Based on the same concept, the present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the debris flow monitoring method as described above is implemented.

[0051] Based on the same concept, the present invention also provides a computer program product, including a computer program / instruction, which implements the debris flow monitoring method as described above when executed by a processor.

[0052] Beneficial Effects

[0053] Compared with the prior art, the advantages of the present invention are:

[0054] The present invention adopts a vision-based debris flow monitoring method, which can intuitively process and analyze the information in the image, reducing the false alarm phenomenon; the present invention combines the target detection model with the optical flow information extraction algorithm, can accurately detect the target type and extract the debris flow movement area, and fully utilizes the complementary advantages of the target detection model and the optical flow information extraction algorithm, further effectively reducing the false alarm problem of debris flow monitoring. The present invention can calculate the debris flow attributes according to the debris flow movement area, and then realize the judgment and early warning of the debris flow scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only one embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 is a flow chart of a debris flow monitoring method according to an embodiment of the present invention;

[0057] Figure 2 is a flow chart for determining whether a target detection frame overlaps with a debris flow motion area in an embodiment of the present invention;

[0058] Figure 3 It is a flow chart for determining the warning level in an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following is a clear and complete description of the technical solutions in the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0061] Example 1

[0062] like Figure 1 As shown, a debris flow monitoring method provided by an embodiment of the present invention comprises the following steps:

[0063] Step S1: Acquire an image of the area to be monitored.

[0064] A visual sensor (such as a high-definition camera) is used to collect a video stream of the area to be monitored, and the video stream is subjected to frame extraction processing to obtain multiple consecutive frames of images of the area to be monitored.

[0065] Step S2: Perform debris flow target detection on each frame of the image of the area to be monitored to obtain a target detection frame.

[0066] In a specific embodiment of the present invention, debris flow target detection is performed on each frame of the image of the area to be monitored, including calling a target detection model and using the target detection model to perform debris flow target detection on each frame of the image of the area to be monitored.

[0067] In this embodiment, the target detection model adopts a deep learning neural network, and the deep learning neural network is specifically a YOLOv8 model. The YOLOv8 model includes an input layer, a backbone network, a connection network, and a detection head output layer.

[0068] The input layer is used to preprocess the image of the monitored area, including image resizing and normalization. Image resizing is to use an interpolation algorithm (such as a bilinear interpolation algorithm) to resize the image of the monitored area to a fixed size required by the target detection model, such as 640×640; normalization is to perform a normalization operation on the resized image so that each pixel value in the image is normalized to the range of [0,1] or [-1,1], usually by dividing the pixel value by 255.

[0069] The backbone network is used to extract features from the preprocessed image and obtain feature maps of different scales. The backbone network includes a series of convolutional layers, cross-stage partial networks, residual blocks, spatial pooling pyramids, feature aggregation pyramids and other modules, which are used to extract features from the preprocessed image, such as color, texture, edge information, etc.

[0070] The connection network is used to connect the backbone network and the output layer of the detection head. Its main function is to fuse and enhance the feature maps of different scales so as to provide richer and more effective information for the final target detection. In this embodiment, the connection network includes a feature pyramid network (FPN), a path aggregation network (PANet), a convolution layer and a pooling layer; the feature pyramid network fuses features from different scales through top-down paths and lateral connections; the path aggregation network further enhances multi-scale features, aggregates features of different levels through bottom-up paths, and the convolution layer and the pooling layer are used for feature processing and adjustment to ensure that the size and number of channels of the final feature map meet the requirements of the subsequent detection head output layer.

[0071] The output layer of the detection head is mainly responsible for generating the final detection results, including the debris flow target detection box and its position, size and category. The output layer of the detection head uses the anchor box mechanism. The anchor box is a predefined bounding box with different scales and aspect ratios, which is used to initially predict the position and size of the target; then by learning the adjustment parameters of the predicted anchor box, the target detection model can locate and describe the target more accurately. In order to improve the detection ability of targets of different sizes, the output layer of the detection head usually makes predictions at multiple scales. By performing detection on feature maps of different scales, it can effectively handle the detection tasks of large and small targets.

[0072] The target detection model has been trained before being called. The specific training process includes: building a sample data set; building a target detection model; using the sample data set to train the target detection model to obtain a trained target detection model for ready calling.

[0073] Each sample in the sample dataset includes a debris flow image and its type label. The type label is debris flow, which is obtained by manually marking the debris flow image. In order to improve the generalization ability and performance of the target detection model, the debris flow images are preprocessed before constructing the sample dataset. The preprocessing includes image resizing, normalization, and data enhancement. Data enhancement specifically includes: randomly rotating the image, adjusting image parameters (such as brightness, contrast, saturation, etc.), and increasing the number and diversity of images.

[0074] Step S3: Perform dense optical flow extraction on two adjacent frames of images of the area to be monitored to obtain the debris flow movement area.

[0075] In a specific implementation of the present invention, the Farneback dense optical flow algorithm is used to extract dense optical flow from two adjacent frames of images of the area to be monitored. The main principle is to model the image in a local area through polynomial expansion, and use the built model to estimate the optical flow feature points. The specific implementation process includes:

[0076] A pyramid of different resolutions is constructed based on two adjacent frames of images of the area to be monitored. For each layer of the pyramid, a polynomial model is used to fit the image block and the polynomial coefficients are calculated. Starting from the coarsest layer of the pyramid, the initial optical flow estimation is used. In the current layer, the polynomial model is used to perform local area matching and calculate the motion vector of the local area image block. The initial optical flow is then iteratively optimized and updated by minimizing the error function. Finally, the optical flow estimation result of the current layer is propagated to a finer level for further optimization.

[0077] Set multiple optical flow feature points, extract optical flow from them, and after completing the optical flow extraction, record the motion area of ​​each optical flow feature point for multiple frames, and compare the pixel coordinates of the optical flow feature point before and after the frame extraction interval (for example, 1 second), record the pixel coordinates of each extracted optical flow feature point in two adjacent frames of the image of the area to be monitored, and take the circumscribed polygon of all optical flow feature points to obtain the debris flow motion area. A debris flow motion area corresponds to two adjacent frames of the image of the area to be monitored, so each optical flow feature point in the debris flow motion area has pixel coordinates in the corresponding two frames of the image of the area to be monitored, so one optical flow feature point corresponds to two pixel coordinates.

[0078] Step S4: Determine whether the target detection frame overlaps with the debris flow movement area.

[0079] The target detection model only detects a single frame of image, but it has a higher ability to extract and identify image features, while the optical flow algorithm processes multiple frames of image and is more sensitive to motion areas. Therefore, combining the advantages of the two methods, it is necessary to make an overlap judgment between the debris flow target detection frame obtained by the target detection model and the debris flow motion area extracted by the optical flow algorithm to obtain a debris flow motion area with a higher confidence level.

[0080] like Figure 2 As shown, judging whether the target detection frame overlaps with the debris flow movement area includes:

[0081] When the target detection frame does not overlap with the debris flow movement area (polygon), it indicates that there is no debris flow risk and no debris flow risk report is made;

[0082] When the target detection frame overlaps with the debris flow movement area, there is a debris flow risk, and the debris flow attributes are calculated based on the debris flow movement area.

[0083] There may be the following situations where the target detection frame does not overlap with the debris flow movement area:

[0084] (a) There is a target detection frame (all soil areas in the image of the monitored area are identified as target detection frames), and there is no debris flow movement area. In this case, it is considered that the target detection model detects the debris flow, but the debris flow does not move and there is no risk;

[0085] (b) There is no target detection frame, but there is a debris flow motion area. In this case, it is considered that there is no debris flow target, and other target motion forms the optical flow;

[0086] (c) There is a target detection frame and a debris flow motion area, but there is no overlapping area between the two. In this case, it is considered that the debris flow is not moving or other targets cause the optical flow to change, so no report is required.

[0087] The target detection model solves the problem that the optical flow information extraction algorithm cannot accurately determine the target type that causes the image pixel change, and the optical flow information extraction algorithm solves the problem that the target detection model can only extract the region of interest for a single frame image and cannot monitor moving targets. The present invention combines the target detection model with the optical flow information extraction algorithm, fully utilizes the complementarity of the two methods, ensures the accuracy of debris flow monitoring, and effectively reduces the false alarm of debris flow.

[0088] In a specific embodiment of the present invention, the debris flow attributes include the flow direction and flow velocity of the debris flow, and calculating the flow direction and flow velocity of the debris flow according to the debris flow movement area includes:

[0089] Step A1: Construct a transformation matrix between the world coordinate system and the pixel coordinate system of the image of the area to be monitored.

[0090] Debris flow properties are usually measured by physical distance. For example, the velocity of a debris flow is usually measured in meters per second, and the flow width of a debris flow is usually measured in meters. Visual information is collected by visual sensors, which can provide information such as pixels and colors of an image, but cannot measure distance information in an image. Therefore, it is necessary to establish a connection between the pixel coordinate position in the image and the actual physical distance, that is, to map the physical distance information in the world coordinate system to the image pixel coordinate system. In a specific embodiment of the present invention, a conversion matrix between the world coordinate system and the pixel coordinate system of the image of the monitored area is constructed, including:

[0091] Step A1.1: Select any point P1 in the image of the area to be monitored and obtain the pixel coordinates of the point P1;

[0092] Step A1.2: Find point W1 corresponding to point P1 in the world coordinate system, and measure the physical distance between point W1 and the visual sensor that collects the image of the monitored area in the horizontal direction and radial direction to obtain the physical position coordinates of point W1;

[0093] Step A1.3: A set of calibration points is formed by point P1 and point W1;

[0094] Step A1.4: Repeat steps A1.1 to A1.3 to obtain multiple sets of calibration points;

[0095] Step A1.5: Calculate the transformation matrix between the world coordinate system and the pixel coordinate system of the image of the monitored area based on multiple sets of calibration points. In this embodiment, multiple sets of calibration points are used as inputs of the PnP (Perspective-n-Point) problem to calculate the transformation matrix between the world coordinate system and the pixel coordinate system of the image of the monitored area.

[0096] According to the transformation matrix, the pixel distance between any two pixels in the image can be converted into the actual physical distance, laying the foundation for the calculation of debris flow properties.

[0097] Step A2: Determine the pixel coordinates of each optical flow feature point in the debris flow movement area in the corresponding two frames of images of the area to be monitored.

[0098] The debris flow movement area is obtained from two adjacent frames of images of the area to be monitored. Therefore, each optical flow feature point in the debris flow movement area has a pixel coordinate position in the two frames of images of the area to be monitored, thereby obtaining the pixel coordinates of each optical flow feature point in the corresponding two frames of images of the area to be monitored.

[0099] Step A3: According to the conversion matrix, the pixel coordinates of each optical flow feature point in the corresponding two frames of the monitored area image are converted into the world coordinate system to obtain two physical position coordinates of each optical flow feature point.

[0100] Step A4: define the 0° direction in the image of the area to be monitored, and calculate the angle between the line between the two physical position coordinates of each optical flow feature point and the 0° direction.

[0101] In this embodiment, the center point of the image of the monitored area is taken as the origin, the top is taken as the 0° direction, and the angle between the line between the two physical position coordinates of each optical flow feature point and the 0° direction is the movement direction of the optical flow feature point.

[0102] Step A5: Calculate the average value of all angles to obtain the direction of the debris flow.

[0103] The average of the running directions of all optical flow feature points in the debris flow movement area is taken as the overall flow direction of the debris flow. In order to simplify the calculation, the optical flow feature points whose distance between the two physical position coordinates is less than the distance threshold are not included in the calculation of the debris flow direction.

[0104] Step A6: Calculate the distance between the two physical position coordinates of each optical flow feature point, and calculate the speed of each optical flow feature point based on the distance and the frame extraction interval time.

[0105] Assuming that the two physical position coordinates of the optical flow feature point are (x1, y1) and (x2, y2), the calculation formula of the distance d is: The calculation formula of the speed v is: v=d / t, where t represents the frame extraction interval time, that is, the time interval between two adjacent frames of the image of the area to be monitored.

[0106] Step A7: Calculate the average value of the velocities of all optical flow feature points to obtain the velocity of the debris flow.

[0107] In a specific embodiment of the present invention, the debris flow attribute also includes the flow width of the debris flow. Since the debris flow may flow from all directions, before calculating the flow width, it is necessary to first calculate the flow direction of the debris flow. The calculation process of the debris flow direction is shown in steps A1 to A5; then the direction perpendicular to the flow direction of the debris flow is taken as the reference direction, where the reference direction is equal to the sum of the flow direction of the debris flow and 90°; finally, the width of the debris flow movement area in the reference direction is the flow width of the debris flow, that is, the Euclidean distance between the two optical flow feature points with the largest distance in the reference direction of the debris flow movement area is calculated, and the Euclidean distance is the maximum flow width of the debris flow.

[0108] In a specific embodiment of the present invention, the debris flow attribute further includes the particle size of solid matter in the debris flow; wherein calculating the particle size of solid matter in the debris flow includes:

[0109] Step B1: Binarize and segment the debris flow movement area to obtain a binary image, that is, divide the image into foreground and background;

[0110] Step B2: extracting the contour of the binary image to obtain the contour of the object in the debris flow movement area;

[0111] Step B3: According to the transformation matrix, the pixel coordinates of the two optical flow feature points with the largest distance in each object contour are first converted to the world coordinate system, and then the distance between the two optical flow feature points with the largest distance in each object contour is calculated in the world coordinate system, and the distance is the particle size of the object contour;

[0112] Step B4: Calculate the average particle size of all object outlines in the debris flow movement area to obtain the particle size of solid objects in the debris flow.

[0113] In step B3, the pixel coordinates of the two optical flow feature points with the largest distance in each object contour correspond to the pixel coordinates in the same frame of the image of the area to be monitored. Exemplarily, if the pixel coordinates of one of the two optical flow feature points with the largest distance are derived from the previous frame of the image of the area to be monitored, then the pixel coordinates of the other optical flow feature point are also derived from the previous frame of the image of the area to be monitored; if the pixel coordinates of one of the two optical flow feature points with the largest distance are derived from the next frame of the image of the area to be monitored, then the pixel coordinates of the other optical flow feature point are also derived from the next frame of the image of the area to be monitored.

[0114] Step S5: Issue different levels of warnings based on the properties of the debris flow.

[0115] Since debris flows occur at different scales, various debris flow properties can be used to measure the scale of debris flows and then issue different levels of warnings.

[0116] In a specific embodiment of the present invention, Figure 3 As shown in the figure, different levels of early warning are issued according to the properties of debris flow, including:

[0117] If each debris flow attribute is less than the corresponding first threshold, no debris flow warning is issued;

[0118] If the corresponding first threshold value ≤ at least one debris flow attribute < the corresponding second threshold value, a debris flow mild warning is issued;

[0119] If the corresponding second threshold value ≤ at least one debris flow attribute < the corresponding third threshold value, a moderate debris flow warning is issued;

[0120] If the corresponding third threshold is ≤ at least one debris flow attribute, a severe debris flow warning is issued.

[0121] Debris flow properties include flow velocity, flow width and solid particle size.

[0122] For example, if the flow velocity, flow width, and solid particle size of the debris flow are all smaller than the corresponding first thresholds, no debris flow warning is issued;

[0123] If the first threshold of flow velocity ≤ flow velocity < the second threshold of flow velocity, the flow width < the first threshold of flow width, and the solid particle size < the first threshold of solid particle size, a slight debris flow warning is issued;

[0124] If the second threshold of flow velocity ≤ flow velocity < the third threshold of flow velocity, the flow width < the second threshold of flow width, and the solid particle size < the second threshold of solid particle size, then a moderate debris flow warning is issued;

[0125] If the third threshold value of flow velocity ≤ flow velocity, the flow width < the third threshold value of flow width, and the solid particle size < the third threshold value of solid particle size, a severe debris flow warning is issued.

[0126] The present invention can accurately calculate various debris flow properties and assist the debris flow monitoring system in early warning. Different early warning levels are reported through different early warning thresholds of various debris flow properties, reducing alarms caused by rain and small amounts of water flow, and effectively improving the intelligence of the debris flow monitoring system.

[0127] Example 2

[0128] A debris flow monitoring system provided by an embodiment of the present invention includes a target detection unit, a motion region extraction unit, a judgment unit, an attribute calculation unit and an early warning unit.

[0129] The target detection unit is used to perform debris flow target detection on each frame of the image of the monitored area to obtain a target detection frame; the motion area extraction unit is used to perform dense optical flow extraction on two adjacent frames of the image of the monitored area to obtain a debris flow motion area; wherein the debris flow motion area includes multiple optical flow feature points; the judgment unit is used to judge whether the target detection frame overlaps with the debris flow motion area; the attribute calculation unit is used to calculate the debris flow attributes according to the debris flow motion area when the target detection frame overlaps with the debris flow motion area; the early warning unit is used to issue different levels of debris flow early warnings according to the debris flow attributes.

[0130] In some embodiments, the debris flow monitoring system may be combined with features of the debris flow monitoring method of any one of the first embodiments, and vice versa, which will not be described in detail herein.

[0131] Example 3

[0132] An embodiment of the present invention further provides an electronic device, which includes: a memory, a processor, and a computer program / instructions stored in the memory, and the processor executes the computer program / instructions to implement the debris flow monitoring method in the embodiment of the present application.

[0133] Although not shown, the electronic device includes a processor, which can perform various appropriate operations and processes according to the program and / or data stored in the read-only memory (ROM) or the program and / or data loaded from the storage portion into the random access memory (RAM). The processor can be a multi-core processor, or it can include multiple processors. In some embodiments, the processor can include a general main processor and one or more special coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM, various programs and data required for device operation are also stored. The processor, ROM and RAM are connected to each other via a bus. The input / output (I / O) interface is also connected to the bus.

[0134] The processor and memory are used together to execute the program / instructions stored in the memory. When the program / instructions are executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0135] Although not shown, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the debris flow monitoring method in the embodiment of the present application is implemented.

[0136] Storage media in embodiments of the present invention include permanent and non-permanent, removable and non-removable items that can be implemented by any method or technology to store information. Examples of storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0137] Readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0138] Although not shown, an embodiment of the present invention further provides a computer program product, including: a computer program / instruction, which, when executed by a processor, implements the debris flow monitoring method in the embodiment of the present application.

[0140] What is disclosed above is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or modifications within the technical scope disclosed in the present invention, which should be covered within the protection scope of the present invention.

Claims

1. A debris flow monitoring method, characterized in that: The monitoring method comprises: Perform debris flow target detection on each frame of the image of the area to be monitored to obtain a target detection frame; Performing dense optical flow extraction on two adjacent frames of images of the area to be monitored to obtain a debris flow movement area; wherein the debris flow movement area includes a plurality of optical flow feature points; When the target detection frame does not overlap with the debris flow movement area, there is no debris flow risk; When the target detection frame overlaps with the debris flow movement area, there is a debris flow risk, and the debris flow attributes are calculated according to the debris flow movement area.

2. The debris flow monitoring method according to claim 1, characterized in that: Perform debris flow target detection on each frame of the image of the monitored area, including: Calling a target detection model, and using the target detection model to perform debris flow target detection on each frame of the image of the area to be monitored; Among them, the target detection model adopts a deep learning neural network, which includes an input layer, a backbone network, a connection network and a detection head output layer; the input layer is used to preprocess the image of the monitored area; the backbone network is used to extract features of the preprocessed image to obtain feature maps of different scales; the connection network is used to fuse and enhance features of feature maps of different scales to obtain the final feature map; the detection head output layer is used to generate a target detection frame according to the final feature map.

3. The debris flow monitoring method according to claim 2, characterized in that: The connection network includes a feature pyramid network, a path aggregation network, a convolutional layer and a pooling layer; The feature pyramid network is used to fuse feature maps of different scales to obtain a fused feature map; the path aggregation network is used to perform multi-scale feature enhancement on the fused feature map; the convolution layer and the pooling layer are used to perform feature processing and adjustment on the feature map after feature enhancement, so that the size and number of channels of the feature map meet the requirements of the output layer of the detection head.

4. The debris flow monitoring method according to claim 1, characterized in that: The Farneback dense optical flow algorithm is used to extract the dense optical flow of two adjacent frames of images of the monitored area to obtain the debris flow movement area.

5. The debris flow monitoring method according to claim 1, characterized in that: The debris flow attributes include the flow direction and flow velocity of the debris flow; wherein calculating the flow direction and flow velocity of the debris flow includes: Construct a transformation matrix between the world coordinate system and the pixel coordinate system of the image of the area to be monitored; Determine the pixel coordinates of each optical flow feature point in the debris flow motion area in the corresponding two frames of images of the area to be monitored; According to the conversion matrix, the pixel coordinates of each optical flow feature point in the corresponding two frames of the monitored area image are converted into the world coordinate system to obtain two physical position coordinates of each optical flow feature point; Define the 0° direction in the image of the area to be monitored, and calculate the angle between the line connecting the two physical position coordinates of each optical flow feature point and the 0° direction; Calculate the average value of all angles to get the direction of the debris flow; Calculate the distance between the two physical position coordinates of each optical flow feature point, and calculate the speed of each optical flow feature point according to the distance and the frame extraction interval time; Calculate the average velocity of all optical flow feature points to get the velocity of the debris flow.

6. The debris flow monitoring method according to claim 5, characterized in that: The step of constructing a conversion matrix between the world coordinate system and the pixel coordinate system of the image in the area to be monitored includes: Select any point P1 in the image of the area to be monitored and obtain the pixel coordinates of point P1; Find the point W1 corresponding to the point P1 in the world coordinate system, and measure the physical distance between the point W1 and the visual sensor that collects the image of the area to be monitored in the horizontal direction and the radial direction to obtain the physical position coordinates of the point W1; A set of calibration points is formed by point P1 and point W1; Repeat the steps of obtaining a set of calibration points to obtain multiple sets of calibration points; The conversion matrix between the world coordinate system and the pixel coordinate system of the image of the monitored area is calculated based on multiple sets of calibration points.

7. The debris flow monitoring method according to claim 5, characterized in that: The debris flow attribute also includes the flow width of the debris flow; wherein calculating the flow width of the debris flow includes: The direction perpendicular to the flow direction of the debris flow is taken as the reference direction; The Euclidean distance between the two optical flow feature points with the largest distance in the reference direction of the debris flow movement area is calculated to obtain the maximum flow width of the debris flow.

8. The debris flow monitoring method according to claim 1, characterized in that: The debris flow attribute includes the particle size of solid matter in the debris flow; wherein calculating the particle size of solid matter in the debris flow includes: Performing binary segmentation on the debris flow movement area to obtain a binary image; Performing contour extraction on the binary image to obtain the contours of objects in the debris flow movement area; Calculate the distance between the two optical flow feature points with the largest distance in each object contour in the world coordinate system to obtain the particle size of the object contour; The average particle size of all object contours is calculated to obtain the particle size of solid objects in the debris flow.

9. The debris flow monitoring method according to any one of claims 1 to 8, characterized in that: The monitoring method also includes providing different levels of early warning according to the properties of the debris flow, which specifically includes: If each debris flow attribute is less than the corresponding first threshold, no debris flow warning is issued; If the corresponding first threshold value ≤ at least one debris flow attribute < the corresponding second threshold value, a debris flow mild warning is issued; If the corresponding second threshold value ≤ at least one debris flow attribute < the corresponding third threshold value, a moderate debris flow warning is issued; If the corresponding third threshold is ≤ at least one debris flow attribute, a severe debris flow warning is issued.

10. A debris flow monitoring system, characterized in that: include: The target detection unit is used to detect debris flow targets for each frame of the image of the area to be monitored to obtain a target detection frame; A motion region extraction unit is used to perform dense optical flow extraction on two adjacent frames of images of the area to be monitored to obtain a debris flow motion region; wherein the debris flow motion region includes a plurality of optical flow feature points; A judging unit, used for judging whether the target detection frame overlaps with the debris flow movement area; The attribute calculation unit is used to calculate the debris flow attribute according to the debris flow movement area when the target detection frame overlaps with the debris flow movement area.

11. An electronic device comprising a memory, a processor, and a computer program / instruction stored in the memory, characterized in that: The processor executes the computer program / instructions to implement the debris flow monitoring method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the debris flow monitoring method according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the debris flow monitoring method according to any one of claims 1 to 9 is implemented.

Citation Information

Patent Citations

  • Method for measuring two-dimensional flow velocity field on surface of fluid

    CN104297516A

  • Low water flow velocity detection method and electronic equipment

    CN115690154A

  • River surface flow velocity calculation method and system based on optical flow measurement and calculation

    CN115761563A

  • Debris flow faucet monitoring method based on video analysis

    CN117152463A

  • Moving target detection method, device and equipment and computer readable storage medium

    CN117612059A