Rockfall Detection Method, Device, Equipment, Storage Medium and Product

Abstract

The present application discloses a rockfall detection method, device, equipment, storage medium and product, relating to the technical field of machine vision. The disclosed rockfall detection method includes: acquiring a monitoring image of a rockfall on a roadside slope; performing static background removal processing on the monitoring image to obtain a background reconstruction image, and performing region annotation on dynamic targets in the monitoring image to obtain the dynamic regions of the dynamic targets; inputting the background reconstruction image into a preset YOLO algorithm to obtain a first detection result of the dynamic targets, and inputting the dynamic regions into a preset classification network to obtain a second detection result of the dynamic targets; determining the landing point of the rockfall based on the first detection result and the second detection result, that is, by improving the detection rate of the rockfall and the accuracy of the landing point detection of the rockfall, the efficiency of the rockfall detection is improved.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and particularly to a rockfall detection method, device, equipment, storage medium and product. Background Art

[0002] Rockfall detection on the side slopes of roads has characteristics such as complex and changeable backgrounds, high similarity between the target object and the background, difficult data collection, and a large amount of dynamic interference data.

[0003] Currently, the rockfall detection method can segment the monitored image through an image segmentation algorithm, then eliminate the dynamic interference in the image through a deep learning algorithm, and finally detect the rockfall by superimposing an object detection algorithm. However, the implementation of the image segmentation algorithm and the superimposed object detection algorithm consumes a large amount of hardware resources and the processing process is complex, resulting in low efficiency in rockfall detection.

[0004] The above content is only used to assist in understanding the technical solution of this application, and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a rockfall detection method, aiming to solve the technical problem of low efficiency in rockfall detection.

[0006] To achieve the above purpose, this application proposes a rockfall detection method, which includes:

[0007] Obtain a monitoring image of the rockfall on the side slope of the road;

[0008] Perform static background removal processing on the monitoring image to obtain a background reconstruction map, and perform area annotation on the dynamic objects in the monitoring image to obtain the dynamic areas of the dynamic objects;

[0009] Input the background reconstruction map into a preset YOLO algorithm to obtain a first detection result of the dynamic object, and input the dynamic area into a preset classification network to obtain a second detection result of the dynamic object;

[0010] Based on the first detection result and the second detection result, determine the landing point of the rockfall.

[0011] In one embodiment, the step of determining the landing point of the rockfall based on the first detection result and the second detection result includes:

[0012] Based on a preset Kalman filter, fuse the first detection result and the second detection result to obtain a fused detection result;

[0013] Based on the detection frame marking the rockfall in the fused detection result, predict the landing point path of the rockfall.

[0014] Determine the landing point of the falling rock based on the landing path.

[0015] In one embodiment, the step of fusing the first detection result and the second detection result based on a preset Kalman filter to obtain a fused detection result includes:

[0016] Screen the first detection data of the falling rock from the first detection result, and screen the second detection data of the falling rock from the second detection result;

[0017] Based on a preset Kalman filter, perform weighted fusion processing on the first detection data and the second detection data to obtain a fused detection result of the falling rock.

[0018] In one embodiment, the first detection data includes the first confidence level of detecting the falling rock and the target coordinate frame of the falling rock in the background reconstruction map, and the second detection data includes the second confidence level of detecting the falling rock. The step of performing weighted fusion processing on the first detection data and the second detection data based on a preset Kalman filter to obtain a fused detection result of the falling rock includes:

[0019] Based on the first confidence level and the second confidence level, perform Kalman filter weighted processing on the target coordinate frame and the target area to obtain a weighted coordinate frame and a weighted area, where the target area is the dynamic area corresponding to the falling rock;

[0020] Fuse the weighted coordinate frame and the weighted area to obtain a fused detection result that completely describes the trajectory of the falling rock.

[0021] In one embodiment, the step of performing static background removal processing on the monitoring screen to obtain a background reconstruction map includes:

[0022] Crop the effective screen that clearly shows the falling rock in a preset number of frames from the monitoring screen;

[0023] Perform background reconstruction on the effective screen to remove the static background in the effective screen, and obtain a background reconstruction map composed of regions based on dynamic targets.

[0024] In one embodiment, the step of performing background reconstruction on the effective screen to remove the static background in the effective screen, and obtain a background reconstruction map composed of regions based on dynamic targets includes:

[0025] Perform discrete noise removal processing on the effective screen to obtain a feature-enhanced screen;

[0026] Identify static targets from the feature-enhanced screen;

[0027] Remove the static target to obtain a background reconstruction map composed of regions of dynamic targets.

[0028] In addition, to achieve the above object, the present application also proposes a rockfall detection device, which includes:

[0029] An acquisition module for acquiring a monitoring image of a rockfall on a roadside slope;

[0030] A processing module for performing static background removal processing on the monitoring image to obtain a background reconstruction map, and performing region annotation on dynamic targets in the monitoring image to obtain dynamic regions of the dynamic targets;

[0031] A detection module for inputting the background reconstruction map into a preset YOLO algorithm to obtain a first detection result of the dynamic target, and inputting the dynamic region into a preset classification network to obtain a second detection result of the dynamic target;

[0032] A determination module for determining the landing point of the rockfall based on the first detection result and the second detection result.

[0033] In addition, to achieve the above object, the present application also proposes a rockfall detection device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the rockfall detection method as described above.

[0034] In addition, to achieve the above object, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, it implements the steps of the rockfall detection method as described above.

[0035] In addition, to achieve the above object, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the rockfall detection method as described above.

[0036] One or more technical solutions proposed by the present application have at least the following technical effects:

[0037] Since the falling rocks to be detected are dynamic targets, the monitoring images of the falling rocks on the side slope of the road contain both dynamic targets and static backgrounds. To reduce the computing resources occupied by the static background during detection, after obtaining the monitoring images of the falling rocks on the side slope of the road, the static background of the monitoring images is removed to obtain a background reconstruction image. Since the YOLO algorithm will affect the accuracy of the detection results when dealing with a small amount of data, to make up for this accuracy, it is necessary to label the dynamic targets in the monitoring images to obtain the dynamic areas of the dynamic targets. When using the YOLO algorithm to detect the background reconstruction image, the preset classification network is used to detect the dynamic area at the same time, and the first detection result obtained by using the YOLO algorithm and the second detection result obtained by using the preset classification network are obtained respectively. Then, according to the first detection result and the second detection result, the landing point of the falling rock is determined to make up for the accuracy of the first detection result with the second detection result, and then the landing point of the falling rock is accurately detected. Since the YOLO algorithm and the preset classification network process a small amount of data, the detection rate of the falling rocks can be improved. Therefore, by improving the detection rate of the falling rocks and the accuracy of the detection of the landing point of the falling rocks, the detection efficiency of the falling rocks is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The accompanying drawings herein are incorporated into and constitute a part of this specification, showing embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application.

[0039] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0040] Figure 1 It is a schematic flowchart provided for the first embodiment of the falling rock detection method of the present application;

[0041] Figure 2 It is a schematic flowchart provided for the second embodiment of the falling rock detection method of the present application;

[0042] Figure 3 It is a schematic flowchart provided for the third embodiment of the falling rock detection method of the present application;

[0043] Figure 4 It is a schematic flowchart of the brief process of the falling rock detection method provided for the embodiments of the present application;

[0044] Figure 5 It is a schematic diagram of the module structure of the falling rock detection device for the embodiments of the present application;

[0045] Figure 6It is a schematic diagram of the device structure of the hardware operating environment involved in the rockfall detection method in the embodiments of the present application.

[0046] The realization of the purpose, functional characteristics and advantages of the present application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. Specific embodiments

[0047] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0048] To better understand the technical solutions of the present application, the following will be described in detail in conjunction with the accompanying drawings of the specification and specific embodiments.

[0049] The main solution of the embodiments of the present application is: the rockfall monitoring platform obtains the monitoring image of the rockfall on the roadside slope; performs static background removal processing on the monitoring image to obtain a background reconstruction map, and performs area annotation on the dynamic targets in the monitoring image to obtain the dynamic areas of the dynamic targets; inputs the background reconstruction map into a preset YOLO algorithm to obtain a first detection result of the dynamic target, and inputs the dynamic area into a preset classification network to obtain a second detection result of the dynamic target; based on the first detection result and the second detection result, determine the landing point of the rockfall.

[0050] In this embodiment, for the convenience of description, the rockfall detection platform is used as the execution subject for elaboration below.

[0051] Since the prior art can segment the monitored image through an image segmentation algorithm, then eliminate the dynamic interference in the image through a deep learning algorithm, and finally realize the detection of rockfalls by superimposing a target detection algorithm. However, the implementation of the image segmentation algorithm and the superimposed target detection algorithm consumes a large amount of hardware resources and the processing process is complex, resulting in low efficiency of rockfall detection.

[0052] The present application provides a solution. Since the falling rocks to be detected are dynamic targets, the monitoring images of the falling rocks on the roadside slope contain both dynamic targets and static backgrounds. To reduce the computing resources occupied by the static background during detection, after obtaining the monitoring images of the falling rocks on the roadside slope, the static background in the monitoring images is removed to obtain a background reconstruction map. Since the YOLO algorithm will affect the accuracy of the detection results when dealing with a small amount of data, to compensate for this accuracy, it is necessary to label the dynamic targets in the monitoring images to obtain the dynamic regions of the dynamic targets. When using the YOLO algorithm to detect the background reconstruction map, the preset classification network is simultaneously used to detect the dynamic regions, and the first detection result obtained by using the YOLO algorithm and the second detection result obtained by using the preset classification network are respectively obtained. Then, according to the first detection result and the second detection result, the landing point of the falling rock is determined to use the second detection result to compensate for the accuracy of the first detection result, thereby accurately detecting the landing point of the falling rock. Since the YOLO algorithm and the preset classification network process a small amount of data, the detection rate of the falling rocks can be improved. Therefore, by improving the detection rate of the falling rocks and the accuracy of the detection of the landing point of the falling rocks, the detection efficiency of the falling rocks is improved.

[0053] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a falling rock monitoring platform, etc. that can implement the above functions. Hereinafter, taking the falling rock monitoring platform as an example, this embodiment and the following embodiments will be described.

[0054] Based on this, the embodiment of the present application provides a falling rock detection method, referring to Figure 1 , Figure 1 which is a schematic flowchart of the first embodiment of the falling rock detection method of the present application.

[0055] In this embodiment, the falling rock detection method includes steps S10 to S40:

[0056] Step S10, obtain the monitoring images of the falling rocks on the roadside slope;

[0057] It should be noted that the roadside slope is a sloping terrain structure with an inclination angle on both sides of the road, and it is a slope area where geological disasters such as falling rocks may occur on its surface. A falling rock is a rock or stone that loosens and falls from the roadside slope. The monitoring images are single-frame or multi-frame images in the video stream continuously collected by the optical imaging device installed at the slope monitoring point, and contain static background elements and dynamic targets.

[0058] In a specific implementation, a high-dynamic range (HDR) camera can be used to collect videos at a frame rate of more than 30fps and transmit them to the rockfall monitoring platform in real time using the RTSP (Real-Time Streaming Protocol) protocol. Since the RTSP protocol supports multispectral imaging (visible light + infrared), clear monitoring images of rockfalls can be collected even in adverse weather conditions such as rain and fog.

[0059] Step S20: Perform static background removal processing on the monitoring image to obtain a background reconstruction image, and perform regional annotation on the dynamic targets in the monitoring image to obtain the dynamic regions of the dynamic targets.

[0060] It should be noted that the static background is an object whose position does not change in the monitoring image. For example, mountains, vegetation, or roads, etc. Static background removal processing is an image processing technique used to remove the fixed background part from the monitoring image to highlight dynamic targets. The background reconstruction image is the image obtained after static background removal processing, which only retains the regions of dynamic targets. Dynamic targets are objects whose positions change relative to the background in the monitoring image. For example, rockfalls, moving vehicles, or animals, etc. The dynamic region is the image region occupied by the dynamic target in the monitoring image, usually marked with a coordinate frame.

[0061] It can be understood that through static background removal processing, background noise can be effectively removed, dynamic targets can be highlighted, reducing the amount of data that needs to be processed when the subsequent model processes the background reconstruction image, reducing the complexity of subsequent rockfall detection, increasing the rate of rockfall detection, and further improving the efficiency of rockfall detection by increasing the detection rate.

[0062] It can be understood that although the use of deep learning models can improve the accuracy of rockfall detection, this accuracy is based on a large amount of data. When the data input to the model is less, it will affect the detection accuracy. Therefore, to ensure the accuracy of rockfall detection, the regions of dynamic targets are marked from the monitoring image. For example, the dynamic targets in the monitoring image are marked with coordinate frames for subsequent use of the marked dynamic targets to supplement the results of rockfall detection by the deep learning model.

[0063] Step S30: Input the background reconstruction image into a preset YOLO algorithm to obtain the first detection result for the dynamic target, and input the dynamic region into a preset classification network to obtain the second detection result for the dynamic target.

[0064] It should be noted that the YOLO algorithm is a real-time object detection algorithm that can quickly identify objects in an image and output their position and category information. The first detection result is the detection information of dynamic objects obtained after processing the background reconstruction image by the YOLO algorithm, including the coordinate box, confidence level, and category of the object. The preset classification network can be a series of networks such as ResNet, which can accurately identify the dynamic area marked in the background reconstruction image. The second detection result is the classification information of dynamic objects obtained after processing the dynamic area by the classification network, including the category and confidence level of the object.

[0065] It can be understood that the background reconstruction image is input into the preset YOLO algorithm model, and its fast detection ability is used to obtain the position information (such as the object coordinate box) and confidence level of the dynamic object. However, due to the small number of dynamic objects in the background reconstruction image, when the YOLO algorithm detects falling rocks based on the background reconstruction image, it may cause frame loss, that is, the detection box (dynamic area) jumps. To make up for the decrease in detection accuracy caused by the jump of the dynamic area when the YOLO algorithm detects falling rocks, based on the dynamic area, the preset classification network is used to detect falling rocks, so that the detection result of the preset classification network for falling rocks is fused with the detection result of the YOLO algorithm for falling rocks, thereby making the prediction of the detection box smooth, making up for the problem caused by the jump of the detection box, and then realizing the accurate detection of falling rocks.

[0066] Step S40: Based on the first detection result and the second detection result, determine the landing point of the falling rock.

[0067] It should be noted that the landing point is the position where the falling rock may finally reach.

[0068] It can be understood that through Kalman filtering or other data fusion algorithms, the detection results of the YOLO algorithm and the classification network are weighted and fused, comprehensively considering the confidence level and object position information, to obtain a more accurate fusion detection result, thereby improving the accuracy and reliability of the landing point prediction.

[0069] This embodiment provides a rockfall detection method. Since the rockfall to be detected is a dynamic target, the monitoring image of the rockfall on the roadside slope collected contains both dynamic targets and static backgrounds. To reduce the computational resources occupied by the static background during detection, after obtaining the monitoring image of the rockfall on the roadside slope, the monitoring image is processed to remove the static background to obtain a background reconstruction image. Since the YOLO algorithm will affect the accuracy of the detection result when dealing with a small amount of data, to make up for this accuracy, it is necessary to label the dynamic targets in the monitoring image to obtain the dynamic regions of the dynamic targets. When using the YOLO algorithm to detect the background reconstruction image, the preset classification network is used to detect the dynamic regions at the same time, and the first detection result obtained by using the YOLO algorithm and the second detection result obtained by using the preset classification network are obtained respectively. Then, according to the first detection result and the second detection result, the landing point of the rockfall is determined to make up for the accuracy of the first detection result by using the second detection result, and then the landing point of the rockfall is accurately detected. Since the YOLO algorithm and the preset classification network process a small amount of data, the detection rate of the rockfall can be improved. Therefore, by improving the detection rate of the rockfall and the accuracy of the landing point detection of the rockfall, the detection efficiency of the rockfall is improved.

[0070] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar content as that in the above-mentioned first embodiment can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 2 , step S40 further includes steps S41 to S43:

[0071] Step S41, based on the preset Kalman filter, fuse the first detection result and the second detection result to obtain a fused detection result;

[0072] Step S42, based on the detection frame marking the rockfall in the fused detection result, predict the landing point path of the rockfall;

[0073] Step S43, based on the landing point path, determine the landing point of the rockfall.

[0074] It should be noted that the Kalman filter is an efficient self-recursive filter that can estimate the dynamic state of the system from a series of noisy measurements. In this embodiment, the Kalman filter is a pre-configured Kalman filter algorithm used to fuse and optimize the detection results to improve the accuracy and stability of target detection. The fused detection result refers to the comprehensive detection result obtained by weighted fusion of the first detection result and the second detection result through the Kalman filter; this result combines the detection results of multiple different models and can more accurately describe the state of the dynamic target. The detection frame refers to the rectangular frame that marks the target position in the image. The landing point path refers to the movement trajectory of the rockfall from the current position to the final landing point.

[0075] It is understandable that by fusing the detection results of the YOLO algorithm and the classification network and using Kalman filtering to optimize the data, the accuracy and stability of detection can be improved. Further, by analyzing the detection boxes marking the falling rocks in the fused detection results, predicting their movement trajectories (landing paths), and finally determining the landing points of the falling rocks, that is, combining the advantages of image processing, deep learning, and kinematic analysis, the landing points of the falling rocks can be predicted quickly and accurately in complex environments.

[0076] It is understandable that since the inference performance of deep learning models depends highly on the amount of data, it is very common for detection boxes to jump, fluctuate greatly, and be unstable under the premise of insufficient data. To solve this problem, a method of parallel multi-detection algorithms is adopted, and finally the Kalman filtering method is used to fuse the prediction results, which can improve the accuracy of the overall detection algorithm and further improve the reliability of the tracking path.

[0077] In the specific implementation, the position information (such as coordinate boxes) and confidence levels of the targets are extracted from the first detection results; the classification confidence levels are extracted from the second detection results; based on the dynamic model and observation model of Kalman filtering, the above data are weighted and fused to obtain the fused detection results; then the center point coordinates of the detection boxes marking the falling rocks in the fused detection results are extracted; according to the changes in the center point coordinates of the detection boxes in consecutive frames, the movement speed and acceleration of the falling rocks are calculated; based on kinematic formulas, the movement trajectories of the falling rocks, that is, the landing paths, are predicted; according to the movement trajectories of the landing paths, combined with the initial speed, acceleration, and movement direction of the falling rocks, the movement equations of the falling rocks are calculated; through the movement equations, the projected positions (landing points) of the falling rocks on the ground are deduced; the detection error factors (such as air resistance, terrain influence, etc.) can be further analyzed, and then the landing points are corrected according to the error factors to accurately obtain the landing points of the falling rock detection.

[0078] Furthermore, step S41 further includes:

[0079] Screen the first detection data of the falling rocks from the first detection results, and screen the second detection data of the falling rocks from the second detection results;

[0080] Based on the preset Kalman filtering, perform weighted fusion processing on the first detection data and the second detection data to obtain the fused detection results of the falling rocks.

[0081] It should be noted that the first detection data is the key information related to the rockfall screened from the first detection result, mainly including the target coordinate box (i.e., the detection box) of the rockfall in the background reconstruction image and the corresponding first confidence level. The second detection data refers to the key information related to the rockfall screened from the second detection result, mainly including the second confidence level for the rockfall detection. The weighted fusion process comprehensively processes the first detection data and the second detection data according to the confidence levels or other weight factors of the detection data to obtain a more accurate fusion detection result.

[0082] It can be understood that by screening the effective data related to the rockfall from the first detection result and the second detection result, and using the preset Kalman filtering algorithm to perform weighted fusion processing on these data, an accurate rockfall detection result is finally obtained. The whole process combines the advantages of object detection and classification. Through the optimization process of Kalman filtering, the reliability of each detection method is fully utilized, and at the same time, the influence of noise and error is suppressed.

[0083] It can be understood that the recursive nature of Kalman filtering can effectively process the noise and uncertainty in the detection data, ensuring the stability of the detection result. Even in a complex environment, it can maintain a high detection performance.

[0084] It can be understood that by screening the effective data related to the rockfall from the first detection result and the second detection result, and then using the preset Kalman filtering algorithm to process the effective data, the amount of data processing is reduced, and the fusion rate of the first detection data and the second detection data is improved. Therefore, by improving the accuracy and detection rate of the detection result, the efficiency of predicting the landing point of the rockfall is improved.

[0085] Furthermore, the step of performing weighted fusion processing on the first detection data and the second detection data based on the preset Kalman filtering to obtain a fusion detection result for the rockfall includes:

[0086] Based on the first confidence level and the second confidence level, perform Kalman filtering weighted processing on the target coordinate box and the target area to obtain a weighted coordinate box and a weighted area, where the target area is the dynamic area corresponding to the rockfall;

[0087] Fuse the weighted coordinate box and the weighted area to obtain a fusion detection result that completely describes the trajectory of the rockfall.

[0088] It should be noted that the first confidence level is a reliability index of the detection result output by the YOLO algorithm when detecting the rockfall target. The second confidence level is a reliability index of whether the target is a rockfall output by the classification network when classifying the dynamic target area. The target coordinate box is a rectangular box output by the target detection algorithm, which is used to identify the position of the rockfall in the image. The target area is the area occupied by the dynamic target in the image, usually obtained by region annotation of the dynamic target. The weighted coordinate box is an optimized target coordinate box obtained after Kalman filter weighting processing, which combines the confidence information of target detection and classification and can more accurately describe the position of the rockfall. The weighted area is an optimized target area obtained after Kalman filter weighting processing.

[0089] It can be understood that by combining the confidence information of target detection and classification, weighting the target coordinate box and the target area, and further fusing the optimized data, the detection result that fully describes the trajectory of the rockfall is finally obtained, giving full play to the optimization ability of the Kalman filter and the reliability of the confidence information, and significantly improving the accuracy and stability of rockfall detection.

[0090] It can be understood that by combining the first confidence level and the second confidence level, weighting the target coordinate box and the target area, the advantages of target detection and classification can be fully utilized, the error of a single method can be reduced, and thus the detection accuracy can be significantly improved.

[0091] It can be understood that through weighting processing and fusion, the key information of target detection and classification is fully utilized, the interference of redundant data is avoided, and the utilization efficiency of data is improved.

[0092] It can be understood that by fusing the weighted coordinate box and the weighted area, the trajectory and motion state of the rockfall can be fully described, providing high-quality data support for subsequent landing point prediction and safety warning.

[0093] Based on the first embodiment and the second embodiment of the present application, in the third embodiment of the present application, the same or similar content as the above embodiments can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 3 , step S20 further includes steps S01~S02:

[0094] Step S01, cropping the effective frames in the monitoring screen that clearly show the rockfall in a preset number of frames;

[0095] Step S02, performing background reconstruction on the effective frames, removing the static background in the effective frames, and obtaining a background reconstruction map composed of regions based on dynamic targets.

[0096] It should be noted that the preset number of frames is the number of consecutive frames preset in the video stream, which is used to extract effective frames containing rockfall information from the monitoring screen. The effective frames are anti-shake image frames cropped from the monitoring screen that can clearly display rockfalls.

[0097] It can be understood that by cropping the effective frames, the amount of data for subsequent processing can be reduced, the blurred frames caused by jitter can be removed, and thus the processing speed and accuracy can be improved to enhance the processing efficiency.

[0098] It can be understood that by cropping the effective frames that clearly display rockfalls from the monitoring screen and performing background reconstruction processing on these frames to remove the static background, a background reconstruction map based on dynamic targets is finally generated. This process provides high-quality input data for subsequent rockfall detection and analysis, and can significantly improve the detection speed and accuracy, thereby enhancing the detection efficiency.

[0099] Furthermore, step S02 further includes:

[0100] Performing discrete noise removal processing on the effective frames to obtain feature-enhanced frames;

[0101] Identifying static targets from the feature-enhanced frames;

[0102] Removing the static targets to obtain a background reconstruction map composed of regions of dynamic targets.

[0103] It should be noted that discrete noise removal processing is to remove random noise in the image through image processing technology to enhance the image quality; common methods include median filtering, Gaussian filtering, or denoising algorithms based on the frequency domain. The feature-enhanced frames are the result of enhancing the target features (such as the contour and texture of rockfalls) in the image after discrete noise removal processing. Static targets are objects that do not move relative to the background in the image.

[0104] It can be understood that through discrete noise removal processing, enhancing target features, identifying and removing static targets, a background reconstruction map is finally generated, providing high-quality input data for subsequent rockfall detection and analysis.

[0105] Exemplarily, to help understand the implementation process of the rockfall detection method obtained by combining the above embodiments, please refer to Figure 4 , Figure 4 A brief flow schematic diagram of a rockfall detection method is provided, specifically:

[0106] 1. Preprocessing stage:

[0107] A. Implement slight image anti - shake using optical flow method. Since the road side slope detection environment is complex, it is difficult to ensure the absolute stability of the image (for example, a strong cross - wind causes slight camera jitter, resulting in slight image jitter). Therefore, an anti - shake algorithm is needed to maintain the stability of image features as much as possible, which will facilitate the subsequent algorithm progress. The main method is to extract image features, match the image data of the previous n frames, select stable images, and crop the effective area to achieve image anti - shake.

[0108] B. Use the inter - frame difference method to compare consecutive images after anti - shake of the previous n frames, reconstruct the background, remove all static backgrounds, and retain the coordinate information of all dynamic targets within the monitoring field of view.

[0109] 2. Inference stage: This part is divided into two parallel stages without a sequence:

[0110] A. Use a deep - learning algorithm (such as the YOLO series), put the image after background reconstruction into the trained model, and output the coordinate frame, target category, and confidence level of the target.

[0111] B. Use the coordinate frame described in 1.B to intercept the corresponding area in the original image, put it into a classification network (such as the resnet series network), and output the confidence level and target category.

[0112] 3. Post - processing stage:

[0113] A. Combine the two sets of confidence level and category data to eliminate non - warning targets.

[0114] B. Use the two sets of confidence level and coordinate frame data to dynamically allocate the weights of the Kalman filter to achieve smooth prediction of the detection frame and reduce the occurrence of detection frame jumping problems.

[0115] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the rockfall detection method of this application. Based on this technical concept, more forms of simple transformations are within the protection scope of this application.

[0116] This application also provides a rockfall detection device. Please refer to Figure 5 , and the rockfall detection device includes:

[0117] An acquisition module 10 for acquiring monitoring images of rockfalls on the road side slope;

[0118] A processing module 20 for removing the static background from the monitoring image to obtain a background reconstruction map, and performing regional annotation on the dynamic targets in the monitoring image to obtain the dynamic regions of the dynamic targets;

[0119] The detection module 30 is configured to input the background reconstruction map into a preset YOLO algorithm to obtain a first detection result of the dynamic target, and input the dynamic region into a preset classification network to obtain a second detection result of the dynamic target;

[0120] The determination module 40 is configured to determine the landing point of the rockfall based on the first detection result and the second detection result.

[0121] Optionally, the determination module 40 is further configured to fuse the first detection result and the second detection result based on a preset Kalman filter to obtain a fused detection result; predict the landing point path of the rockfall based on the detection frame marking the rockfall in the fused detection result; and determine the landing point of the rockfall based on the landing point path.

[0122] Optionally, the determination module 40 is further configured to screen the first detection data of the rockfall from the first detection result and screen the second detection data of the rockfall from the second detection result; perform weighted fusion processing on the first detection data and the second detection data based on a preset Kalman filter to obtain a fused detection result of the rockfall.

[0123] Optionally, the first detection data includes a first confidence level for detecting the rockfall and the target coordinate frame of the rockfall in the background reconstruction map, and the second detection data includes a second confidence level for detecting the rockfall;

[0124] The determination module 40 is further configured to perform Kalman filter weighted processing on the target coordinate frame and the target region based on the first confidence level and the second confidence level to obtain a weighted coordinate frame and a weighted region, where the target region is the dynamic region corresponding to the rockfall; and fuse the weighted coordinate frame and the weighted region to obtain a fused detection result that completely describes the trajectory of the rockfall.

[0125] Optionally, the processing module 20 is further configured to crop valid frames from the monitoring screen that clearly show the rockfall; perform background reconstruction on the valid frames to remove the static background in the valid frames and obtain a background reconstruction map composed of regions of dynamic targets.

[0126] Optionally, the processing module 20 is further configured to perform discrete noise removal processing on the valid frames to obtain a feature-enhanced frame; identify static targets from the feature-enhanced frame; and remove the static targets to obtain a background reconstruction map composed of regions of dynamic targets.

[0127] The rockfall detection device provided by this application adopts the rockfall detection method in the above embodiment, and can solve the technical problem of low efficiency in rockfall detection. Compared with the prior art, the beneficial effects of the rockfall detection device provided by this application are the same as those of the rockfall detection method provided by the above embodiment, and other technical features in the rockfall detection device are the same as those disclosed in the method of the above embodiment, which will not be elaborated here.

[0128] This application provides a rockfall detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the rockfall detection method in the first embodiment above.

[0129] Refer to the following Figure 6 , which shows a schematic structural diagram of a rockfall detection device suitable for implementing the embodiments of this application. The rockfall detection device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The rockfall detection device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of this application.

[0130] As shown in Figure 6As shown, the rockfall detection device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the rockfall detection device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the rockfall detection device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a rockfall detection device having various systems, it should be understood that it is not required to implement or have all the shown systems. Instead, more or fewer systems may be implemented or had.

[0131] Specifically, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through the communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are executed.

[0132] The rockfall detection device provided by the present application adopts the rockfall detection method in the above-mentioned embodiment, and can solve the technical problem of low efficiency in rockfall detection. Compared with the prior art, the beneficial effects of the rockfall detection device provided by the present application are the same as those of the rockfall detection method provided by the above-mentioned embodiment, and other technical features in the rockfall detection device are the same as those disclosed in the method of the previous embodiment, and will not be elaborated here.

[0133] It should be understood that each part disclosed in this application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0134] As described above, the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any person skilled in the art within the technical scope disclosed in this application can easily think of changes or substitutions, which should all be covered by the protection scope of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.

[0135] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the rockfall detection method in the above embodiments.

[0136] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or combined with an instruction execution system, device, or device. The program code contained on the computer-readable storage medium can be transmitted by any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0137] The above computer-readable storage medium can be included in the rockfall detection device; it can also exist separately and not be assembled into the rockfall detection device.

[0138] The above computer-readable storage medium carries one or more programs, which, when executed by the rockfall detection device, cause the rockfall detection device to: obtain a monitoring image of rockfalls on the roadside slope; perform static background removal processing on the monitoring image to obtain a background reconstruction image, and perform regional annotation on dynamic targets in the monitoring image to obtain the dynamic regions of the dynamic targets; input the background reconstruction image into a preset YOLO algorithm to obtain a first detection result of the dynamic targets, and input the dynamic regions into a preset classification network to obtain a second detection result of the dynamic targets; and determine the landing points of the rockfalls based on the first detection result and the second detection result.

[0139] Computer program code for performing the operations of the present application may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that, in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system for performing the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0141] The modules involved in the embodiments of the present application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation to the unit itself in some cases.

[0142] The readable storage medium provided by the present application is a computer-readable storage medium, and the computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned rockfall detection method, which can solve the technical problem of low efficiency in rockfall detection. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as those of the rockfall detection method provided by the above embodiments, and will not be elaborated here.

[0143] The present application also provides a computer program product, including a computer program, and the steps of the rockfall detection method as described above are implemented when the computer program is executed by a processor.

[0144] The computer program product provided by the present application can solve the technical problem of low efficiency in rockfall detection. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the rockfall detection method provided by the above embodiments, and will not be elaborated here.

[0145] The above are only some embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural transformation made by using the content of the specification and drawings of the present application under the technical concept of the present application, or direct / indirect application in other related technical fields, is included in the patent protection scope of the present application.

Claims

1. A rockfall detection method, characterized in that, The method described above includes: Obtaining a monitoring image of the rockfall on the roadside slope; Performing static background removal processing on the monitoring image to obtain a background reconstruction image, and performing regional annotation on the dynamic objects in the monitoring image to obtain the dynamic regions of the dynamic objects; Inputting the background reconstruction image into a preset YOLO algorithm to obtain a first detection result of the dynamic objects, and inputting the dynamic regions into a preset classification network to obtain a second detection result of the dynamic objects; Screening first detection data of the rockfall from the first detection result, and screening second detection data of the rockfall from the second detection result. The first detection data includes the first confidence level of the rockfall detection and the target coordinate frame of the rockfall in the background reconstruction image, and the second detection data includes the second confidence level of the rockfall detection; Based on the first confidence level and the second confidence level, performing Kalman filter weighting processing on the target coordinate frame and the target region to obtain a weighted coordinate frame and a weighted region, where the target region is the dynamic region corresponding to the rockfall; Fusing the weighted coordinate frame and the weighted region to obtain a fusion detection result that completely describes the trajectory of the rockfall; Predicting the landing path of the rockfall based on the detection frame marking the rockfall in the fusion detection result; Determining the landing point of the rockfall based on the landing path; 2. The method according to claim 1, wherein The step of performing static background removal processing on the monitoring image to obtain a background reconstruction image includes: Cropping an effective image that clearly shows the rockfall in a preset number of frames from the monitoring image; Performing background reconstruction on the effective image to remove the static background in the effective image and obtain a background reconstruction image composed of regions of dynamic objects; 3. The method according to claim 2, wherein The step of performing background reconstruction on the effective image to remove the static background in the effective image and obtain a background reconstruction image composed of regions of dynamic objects includes: Performing discrete noise removal processing on the effective image to obtain a feature-enhanced image; Identifying static objects from the feature-enhanced image; Removing the static objects to obtain a background reconstruction image composed of regions of dynamic objects; 4. A rockfall detection device, characterized in that, The device includes: An acquisition module for acquiring a monitoring image of the rockfall on the roadside slope, and a processing module for performing static background removal processing on the monitoring image to obtain a background reconstruction image, and performing regional annotation on the dynamic objects in the monitoring image to obtain the dynamic regions of the dynamic objects; A detection module for inputting the background reconstruction image into a preset YOLO algorithm to obtain a first detection result of the dynamic objects, and inputting the dynamic regions into a preset classification network to obtain a second detection result of the dynamic objects; A determination module is configured to screen first detection data of the rockfall from the first detection result and screen second detection data of the rockfall from the second detection result. The first detection data includes a first confidence level of detecting the rockfall and a target coordinate frame of the rockfall in the background reconstruction map. The second detection data includes a second confidence level of detecting the rockfall. Based on the first confidence level and the second confidence level, perform Kalman filter weighting processing on the target coordinate frame and the target area to obtain a weighted coordinate frame and a weighted area. The target area is a dynamic area corresponding to the rockfall. Fuse the weighted coordinate frame and the weighted area to obtain a fusion detection result that completely describes the trajectory of the rockfall. Based on the detection frame marking the rockfall in the fusion detection result, predict the landing path of the rockfall. Based on the landing path, determine the landing point of the rockfall.

5. A rockfall detection device, characterized in that, The device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor. The computer program is configured to implement the steps of the rockfall detection method according to any one of claims 1 to 3.

6. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the rockfall detection method according to any one of claims 1 to 3 are implemented.

7. A computer program product, characterized in that, The computer program product includes a computer program. When the computer program is executed by a processor, the steps of the rockfall detection method according to any one of claims 1 to 3 are implemented.

Images