Rockfall detection method, device and equipment, storage medium and product

Abstract

The invention discloses a rockfall detection method, device and equipment, a storage medium and a product, and relates to the technical field of machine vision, and the rockfall detection method comprises the steps that a monitoring picture of rockfall of a road side slope is acquired; performing static background removal processing on the monitoring picture to obtain a background reconstruction image, and performing region labeling on a dynamic target in the monitoring picture to obtain a dynamic region of the dynamic target; inputting the background reconstruction image into a preset YOLO algorithm to obtain a first detection result of the dynamic target, and inputting the dynamic region into a preset classification network to obtain a second detection result of the dynamic target; and the falling point of the rockfall is determined based on the first detection result and the second detection result, that is, the rockfall detection efficiency is improved by improving the rockfall detection rate and the rockfall point detection accuracy.

Description

Technical Field

[0001] This application relates to the field of machine vision technology, and particularly to a rockfall detection method, device, equipment, storage medium and product. Background Art

[0002] The detection of rockfalls on the roadside slopes has the characteristics of complex and variable backgrounds, high similarity between the target objects and the backgrounds, difficult data collection, and a large amount of dynamic interference data.

[0003] Currently, the rockfall detection method can segment the monitored images through an image segmentation algorithm, then eliminate the dynamic interference in the images through a deep learning algorithm, and finally achieve the detection of rockfalls by superimposing an object detection algorithm. However, the implementation of the image segmentation algorithm and the superimposed object detection algorithm consumes a large amount of hardware resources and the processing process is complex, resulting in low efficiency in rockfall detection.

[0004] The above content is only used to assist in understanding the technical solution of this application, and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a rockfall detection method, aiming to solve the technical problem of low efficiency in rockfall detection.

[0006] To achieve the above purpose, this application proposes a rockfall detection method, which includes: Obtain a monitoring image of the rockfall on the roadside slope; Perform static background removal processing on the monitoring image to obtain a background reconstruction map, and perform region annotation on the dynamic objects in the monitoring image to obtain the dynamic regions of the dynamic objects; Input the background reconstruction map into a preset YOLO algorithm to obtain a first detection result of the dynamic object, and input the dynamic region into a preset classification network to obtain a second detection result of the dynamic object; Based on the first detection result and the second detection result, determine the landing point of the rockfall.

[0007] In one embodiment, the step of determining the landing point of the rockfall based on the first detection result and the second detection result includes: Based on a preset Kalman filter, fuse the first detection result and the second detection result to obtain a fused detection result; Based on the detection frame marking the rockfall in the fused detection result, predict the landing point path of the rockfall; Based on the landing point path, determine the landing point of the rockfall.

[0008] In one embodiment, the step of fusing the first detection result and the second detection result based on a preset Kalman filter to obtain a fused detection result includes: Screen the first detection data of the rockfall from the first detection result, and screen the second detection data of the rockfall from the second detection result; Based on a preset Kalman filter, perform weighted fusion processing on the first detection data and the second detection data to obtain a fused detection result of the rockfall.

[0009] In one embodiment, the first detection data includes the first confidence level of the rockfall detection and the target coordinate frame of the rockfall in the background reconstruction map, and the second detection data includes the second confidence level of the rockfall detection. The step of performing weighted fusion processing on the first detection data and the second detection data based on a preset Kalman filter to obtain a fused detection result of the rockfall includes: Based on the first confidence level and the second confidence level, perform Kalman filter weighting processing on the target coordinate frame and the target area to obtain a weighted coordinate frame and a weighted area, where the target area is the dynamic area corresponding to the rockfall; Fuse the weighted coordinate frame and the weighted area to obtain a fused detection result that completely describes the trajectory of the rockfall.

[0010] In one embodiment, the step of performing static background removal processing on the monitoring screen to obtain a background reconstruction map includes: Crop the effective screen that clearly shows the rockfall in a preset number of frames from the monitoring screen; Perform background reconstruction on the effective screen, remove the static background in the effective screen, and obtain a background reconstruction map composed of regions based on dynamic targets.

[0011] In one embodiment, the step of performing background reconstruction on the effective screen, removing the static background in the effective screen, and obtaining a background reconstruction map composed of regions based on dynamic targets includes: Perform discrete noise removal processing on the effective screen to obtain a feature-enhanced screen; Identify static targets from the feature-enhanced screen; Remove the static targets to obtain a background reconstruction map composed of regions based on dynamic targets.

[0012] In addition, to achieve the above object, the present application also proposes a rockfall detection device, where the rockfall detection device includes: An acquisition module, configured to acquire a monitoring screen of a rockfall on a roadside slope; A processing module, configured to perform static background removal processing on the monitoring screen to obtain a background reconstruction map, and perform area annotation on dynamic targets in the monitoring screen to obtain the dynamic areas of the dynamic targets; A detection module, configured to input the background reconstruction map into a preset YOLO algorithm to obtain a first detection result of the dynamic target, and input the dynamic area into a preset classification network to obtain a second detection result of the dynamic target; A determination module, configured to determine the landing point of the rockfall based on the first detection result and the second detection result.

[0013] In addition, to achieve the above object, the present application further provides a rockfall detection device, where the device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the rockfall detection method as described above.

[0014] In addition, to achieve the above object, the present application further provides a storage medium, where the storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the rockfall detection method as described above are implemented.

[0015] In addition, to achieve the above object, the present application further provides a computer program product, where the computer program product includes a computer program, and when the computer program is executed by a processor, the steps of the rockfall detection method as described above are implemented.

[0016] One or more technical solutions proposed by the present application have at least the following technical effects: Since the falling rocks to be detected are dynamic targets, the monitored images of the falling rocks on the roadside slope contain both dynamic targets and static backgrounds. To reduce the computing resources occupied by the static background during detection, after obtaining the monitored images of the falling rocks on the roadside slope, the static background is removed from the monitored images to obtain a background reconstruction map. Since the YOLO algorithm will affect the accuracy of the detection results when dealing with a small amount of data, to compensate for this accuracy, it is necessary to label the dynamic targets in the monitored images to obtain the dynamic regions of the dynamic targets. When using the YOLO algorithm to detect the background reconstruction map, the preset classification network is used to detect the dynamic regions at the same time, and the first detection result obtained by using the YOLO algorithm and the second detection result obtained by using the preset classification network are obtained respectively. Then, according to the first detection result and the second detection result, the landing points of the falling rocks are determined to use the second detection result to compensate for the accuracy of the first detection result, and then the landing points of the falling rocks are accurately detected. Since the YOLO algorithm and the preset classification network process a small amount of data, the detection rate of the falling rocks can be improved. Therefore, by improving the detection rate of the falling rocks and the accuracy of the detection of the landing points of the falling rocks, the detection efficiency of the falling rocks is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into and constitute a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0018] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0019] Figure 1 It is a schematic flowchart provided for Embodiment 1 of the falling rock detection method of the present application; Figure 2 It is a schematic flowchart provided for Embodiment 2 of the falling rock detection method of the present application; Figure 3 It is a schematic flowchart provided for Embodiment 3 of the falling rock detection method of the present application; Figure 4 It is a schematic flowchart of the brief process of the falling rock detection method provided for the embodiments of the present application; Figure 5 It is a schematic diagram of the module structure of the falling rock detection device provided for the embodiments of the present application; Figure 6 It is a schematic diagram of the device structure of the hardware operating environment involved in the falling rock detection method provided for the embodiments of the present application.

[0020] The realization, functional features, and advantages of the present application will be further described in conjunction with embodiments and with reference to the accompanying drawings. Detailed implementation manners

[0021] It should be understood that the specific embodiments described herein are only used to explain the technical solution of the present application and are not used to limit the present application.

[0022] To better understand the technical solution of the present application, the following will be described in detail in conjunction with the drawings of the specification and specific implementation manners.

[0023] The main solution of the embodiment of the present application is as follows: The rockfall monitoring platform obtains the monitoring image of the rockfall on the roadside slope; performs static background removal processing on the monitoring image to obtain a background reconstruction map, and performs regional annotation on the dynamic targets in the monitoring image to obtain the dynamic regions of the dynamic targets; inputs the background reconstruction map into a preset YOLO algorithm to obtain a first detection result of the dynamic targets, and inputs the dynamic regions into a preset classification network to obtain a second detection result of the dynamic targets; based on the first detection result and the second detection result, determine the landing point of the rockfall.

[0024] In this embodiment, for the convenience of description, the rockfall detection platform is used as the execution subject for elaboration below.

[0025] Since the prior art can segment the monitored image through an image segmentation algorithm, then eliminate the dynamic interference in the image through a deep learning algorithm, and finally realize the detection of rockfall by superimposing a target detection algorithm. However, the implementation of the image segmentation algorithm and the superimposed target detection algorithm consumes a large amount of hardware resources and the processing process is complex, resulting in low efficiency of rockfall detection.

[0026] The present application provides a solution. Since the falling rocks to be detected are dynamic targets, the monitoring images of the falling rocks on the roadside slope contain both dynamic targets and static backgrounds. To reduce the computing resources occupied by the static background during detection, after obtaining the monitoring images of the falling rocks on the roadside slope, the static background of the monitoring images is removed to obtain a background reconstruction map. Since the YOLO algorithm will affect the accuracy of the detection results when dealing with a small amount of data, to compensate for this accuracy, it is necessary to label the dynamic targets in the monitoring images to obtain the dynamic regions of the dynamic targets. When using the YOLO algorithm to detect the background reconstruction map, the preset classification network is used to detect the dynamic regions at the same time, and the first detection result obtained by using the YOLO algorithm and the second detection result obtained by using the preset classification network are respectively obtained. Then, according to the first detection result and the second detection result, the landing point of the falling rock is determined to use the second detection result to compensate for the accuracy of the first detection result, and then the landing point of the falling rock is accurately detected. Since the amount of data processed by the YOLO algorithm and the preset classification network is small, the detection rate of the falling rocks can be improved. Therefore, by improving the detection rate of the falling rocks and the accuracy of the detection of the landing point of the falling rocks, the detection efficiency of the falling rocks is improved.

[0027] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a falling rock monitoring platform, etc. that can implement the above functions. Hereinafter, the falling rock monitoring platform will be taken as an example to illustrate this embodiment and the following embodiments.

[0028] Based on this, the embodiment of the present application provides a falling rock detection method, referring to Figure 1 , Figure 1 which is a schematic flowchart of the first embodiment of the falling rock detection method of the present application.

[0029] In this embodiment, the falling rock detection method includes steps S10 to S40: Step S10, obtaining a monitoring image of the falling rocks on the roadside slope; It should be noted that the roadside slope is a sloping terrain structure with an inclination angle on both sides of the road, and it is a slope area where geological disasters such as falling rocks may occur on its surface. A falling rock is a rock or stone that loosens and falls from the roadside slope. The monitoring image is a single frame or multiple frames of images in the video stream continuously collected by the optical imaging device installed at the slope monitoring point, and contains static background elements and dynamic targets.

[0030] In a specific implementation, a high-dynamic range (HDR) camera can be used to collect videos at a frame rate of more than 30 fps and transmit them to the rockfall monitoring platform in real time using the RTSP (Real-Time Streaming Protocol) protocol. Since the RTSP protocol supports multispectral imaging (visible light + infrared), clear monitoring images of rockfalls can be collected even in adverse weather conditions such as rain and fog.

[0031] Step S20: Perform static background removal processing on the monitoring image to obtain a background reconstruction image, and perform region annotation on the dynamic targets in the monitoring image to obtain the dynamic regions of the dynamic targets. It should be noted that the static background refers to the objects whose positions do not change in the monitoring image. For example, mountains, vegetation, or roads, etc. Static background removal processing is an image processing technology used to remove the fixed background part from the monitoring image to highlight the dynamic targets. The background reconstruction image is the image obtained after static background removal processing, in which only the regions of the dynamic targets are retained. Dynamic targets are the objects whose positions change relative to the background in the monitoring image. For example, rockfalls, moving vehicles, or animals, etc. The dynamic region is the image region occupied by the dynamic target in the monitoring image, usually marked with a coordinate frame.

[0032] It can be understood that through static background removal processing, background noise can be effectively removed, dynamic targets can be highlighted, the amount of data that needs to be processed when the subsequent model processes the background reconstruction image can be reduced, the complexity of subsequent rockfall detection can be reduced, the detection rate of rockfalls can be increased, and by increasing the detection rate, the efficiency of rockfall detection can be further improved.

[0033] It can be understood that although the use of a deep learning model can improve the accuracy of rockfall detection, this accuracy is based on a large amount of data. When the data input to the model is less, the detection accuracy will be affected. Therefore, to ensure the accuracy of rockfall detection, the regions of the dynamic targets are marked from the monitoring image. For example, the dynamic targets in the monitoring image are marked with coordinate frames for subsequent use of the marked dynamic targets to supplement the results of rockfall detection by the deep learning model.

[0034] Step S30: Input the background reconstruction image into a preset YOLO algorithm to obtain the first detection result of the dynamic target, and input the dynamic region into a preset classification network to obtain the second detection result of the dynamic target. It should be noted that the YOLO algorithm is a real-time object detection algorithm that can quickly identify objects in an image and output their position and category information. The first detection result is the detection information of dynamic objects obtained after processing the background reconstruction image by the YOLO algorithm, including the coordinate frame, confidence level, and category of the object, etc. The preset classification network can be a series of networks such as resnet, which can accurately identify the dynamic area marked in the background reconstruction image. The second detection result is the classification information of dynamic objects obtained after processing the dynamic area by the classification network, including the category and confidence level of the object, etc.

[0035] It can be understood that the background reconstruction image is input into the preset YOLO algorithm model, and its fast detection ability is used to obtain the position information (such as the object coordinate frame) and confidence level of the dynamic object. However, since the number of dynamic objects in the background reconstruction image is small, when the YOLO algorithm detects falling rocks based on the background reconstruction image, it may cause frame loss, that is, the detection frame (dynamic area) jumps. To make up for the decrease in detection accuracy caused by the jump of the dynamic area when the YOLO algorithm detects falling rocks, based on the dynamic area, the preset classification network is used to detect falling rocks, so that the detection result of the preset classification network for falling rocks is fused with the detection result of the YOLO algorithm for falling rocks, thereby making the prediction of the detection frame smooth, making up for the problem caused by the jump of the detection frame, and then realizing the accurate detection of falling rocks.

[0036] Step S40, based on the first detection result and the second detection result, determine the landing point of the falling rock.

[0037] It should be noted that the landing point is the position where the falling rock may finally reach.

[0038] It can be understood that through Kalman filtering or other data fusion algorithms, the detection results of the YOLO algorithm and the classification network are weighted and fused, comprehensively considering the confidence level and target position information, to obtain a more accurate fused detection result, thereby improving the accuracy and reliability of the landing point prediction.

[0039] This embodiment provides a rockfall detection method. Since the rockfall to be detected is a dynamic target, the monitoring images of the rockfall on the roadside slope contain both dynamic targets and static backgrounds. To reduce the computational resources occupied by the static background during detection, after obtaining the monitoring images of the rockfall on the roadside slope, the static background of the monitoring images is removed to obtain a background reconstruction image. Since the YOLO algorithm will affect the accuracy of the detection results when dealing with a small amount of data, to compensate for this accuracy, it is necessary to label the dynamic targets in the monitoring images to obtain the dynamic regions of the dynamic targets. When using the YOLO algorithm to detect the background reconstruction image, the preset classification network is used to detect the dynamic regions at the same time, and the first detection result obtained by using the YOLO algorithm and the second detection result obtained by using the preset classification network are obtained respectively. Then, according to the first detection result and the second detection result, the landing point of the rockfall is determined to use the second detection result to compensate for the accuracy of the first detection result, and then the landing point of the rockfall is accurately detected. Since the YOLO algorithm and the preset classification network process a small amount of data, the detection rate of the rockfall can be improved. Therefore, by improving the detection rate of the rockfall and the accuracy of the landing point detection of the rockfall, the detection efficiency of the rockfall is improved.

[0040] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar content as that in the above-mentioned first embodiment can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 2 , step S40 further includes steps S41 to S43: Step S41, based on a preset Kalman filter, fuse the first detection result and the second detection result to obtain a fused detection result; Step S42, based on the detection frame marking the rockfall in the fused detection result, predict the landing point path of the rockfall; Step S43, based on the landing point path, determine the landing point of the rockfall.

[0041] It should be noted that the Kalman filter is an efficient self-recursive filter that can estimate the dynamic state of the system from a series of noisy measurements. In this embodiment, the Kalman filter is a pre-configured Kalman filter algorithm used to fuse and optimize the detection results to improve the accuracy and stability of target detection. The fused detection result refers to the comprehensive detection result obtained by weighted fusion of the first detection result and the second detection result through the Kalman filter; this result combines the detection results of multiple different models and can more accurately describe the state of the dynamic target. The detection frame refers to the rectangular frame that marks the target position in the image. The landing point path refers to the movement trajectory of the rockfall from the current position to the final landing point.

[0042] It is understandable that by fusing the detection results of the YOLO algorithm and the classification network, and using Kalman filtering to optimize the data, the accuracy and stability of detection can be improved. Further, by analyzing the detection boxes marking the falling rocks in the fused detection results, predicting their movement trajectories (landing paths), and finally determining the landing points of the falling rocks, that is, by combining the advantages of image processing, deep learning, and kinematic analysis, the landing points of the falling rocks can be predicted quickly and accurately in complex environments.

[0043] It is understandable that since the inference performance of deep learning models is highly dependent on the amount of data, it is very common for detection boxes to jump, fluctuate greatly, and be unstable under the premise of insufficient data. To solve this problem, the method of parallel multi-detection algorithms is adopted, and finally the Kalman filtering method is used to fuse the prediction results, which can improve the accuracy of the overall detection algorithm and thus improve the reliability of the tracking path.

[0044] In the specific implementation, extract the position information (such as coordinate boxes) and confidence levels of the targets from the first detection result; extract the classification confidence levels from the second detection result; based on the dynamic model and observation model of Kalman filtering, perform weighted fusion on the above data to obtain the fused detection result; then extract the center point coordinates of the detection boxes marking the falling rocks in the fused detection result; calculate the movement speed and acceleration of the falling rocks according to the changes in the center point coordinates of the detection boxes in consecutive frames; based on kinematic formulas, predict the movement trajectory of the falling rocks, that is, the landing path; according to the movement trajectory of the landing path, combined with the initial speed, acceleration, and movement direction of the falling rocks, calculate the movement equation of the falling rocks; through the movement equation, calculate the projected position (landing point) of the falling rocks on the ground; the detection error factors (such as air resistance, terrain influence, etc.) can also be further analyzed, and then the landing point can be corrected according to the error factors to accurately obtain the landing point of the falling rock detection.

[0045] Further, step S41 also includes: Screen the first detection data of the falling rocks from the first detection result, and screen the second detection data of the falling rocks from the second detection result; Based on the preset Kalman filtering, perform weighted fusion processing on the first detection data and the second detection data to obtain the fused detection result of the falling rocks.

[0046] It should be noted that the first detection data is the key information related to rockfall selected from the first detection result, mainly including the target coordinate box (i.e., the detection box) of the rockfall in the background reconstruction image and the corresponding first confidence level. The second detection data refers to the key information related to rockfall selected from the second detection result, mainly including the second confidence level for rockfall detection. The weighted fusion process is to comprehensively process the first detection data and the second detection data according to the confidence levels or other weight factors of each detection data to obtain a more accurate fusion detection result.

[0047] It can be understood that by screening out the effective data related to rockfall from the first detection result and the second detection result, and using the preset Kalman filtering algorithm to perform weighted fusion processing on these data, an accurate rockfall detection result is finally obtained. The whole process combines the advantages of object detection and classification. Through the optimization process of Kalman filtering, the reliability of each detection method is fully utilized, and at the same time, the influence of noise and error is suppressed.

[0048] It can be understood that the recursive characteristic of Kalman filtering can effectively process the noise and uncertainty in the detection data and ensure the stability of the detection result. Even in a complex environment, it can maintain a high detection performance.

[0049] It can be understood that by screening out the effective data related to rockfall from the first detection result and the second detection result, and then using the preset Kalman filtering algorithm to process the effective data, the amount of data processing is reduced, and the fusion rate of the first detection data and the second detection data is improved. Therefore, by improving the accuracy and detection rate of the detection result, the efficiency of predicting the landing point of the rockfall is improved.

[0050] Furthermore, the step of performing weighted fusion processing on the first detection data and the second detection data based on the preset Kalman filtering to obtain a fusion detection result of the rockfall includes: Based on the first confidence level and the second confidence level, perform Kalman filtering weighted processing on the target coordinate box and the target area to obtain a weighted coordinate box and a weighted area, where the target area is the dynamic area corresponding to the rockfall; Fuse the weighted coordinate box and the weighted area to obtain a fusion detection result that completely describes the trajectory of the rockfall.

[0051] It should be noted that the first confidence level is a reliability index of the detection result output by the YOLO algorithm when detecting the rockfall target. The second confidence level is a reliability index of whether the target is a rockfall output by the classification network when classifying the dynamic target area. The target coordinate box is a rectangular box output by the target detection algorithm, which is used to identify the position of the rockfall in the image. The target area is the area occupied by the dynamic target in the image, usually obtained by region annotation of the dynamic target. The weighted coordinate box is an optimized target coordinate box obtained after Kalman filter weighting processing. It combines the confidence information of target detection and classification and can more accurately describe the position of the rockfall. The weighted area is an optimized target area obtained after Kalman filter weighting processing.

[0052] It can be understood that by combining the confidence information of target detection and classification, weighting the target coordinate box and the target area, and further fusing the optimized data, the detection result that completely describes the trajectory of the rockfall is finally obtained, giving full play to the optimization ability of the Kalman filter and the reliability of the confidence information, and significantly improving the accuracy and stability of rockfall detection.

[0053] It can be understood that by combining the first confidence level and the second confidence level and weighting the target coordinate box and the target area, the advantages of target detection and classification can be fully utilized, the errors of a single method can be reduced, and thus the detection accuracy can be significantly improved.

[0054] It can be understood that through weighting processing and fusion, the key information of target detection and classification is fully utilized, the interference of redundant data is avoided, and the utilization efficiency of data is improved.

[0055] It can be understood that by fusing the weighted coordinate box and the weighted area, the trajectory and motion state of the rockfall can be completely described, providing high-quality data support for subsequent landing point prediction and safety warning.

[0056] Based on the first embodiment and the second embodiment of the present application, in the third embodiment of the present application, the same or similar content as the above embodiments can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 3 , step S20 further includes steps S01 to S02: Step S01, cropping the effective frames in the monitoring screen that clearly show the rockfall in a preset number of frames; Step S02, performing background reconstruction on the effective frames, removing the static background in the effective frames, and obtaining a background reconstruction map composed of regions based on dynamic targets.

[0057] It should be noted that the preset number of frames is the number of consecutive frames preset in the video stream, which is used to extract valid frames containing rockfall information from the monitoring screen. The valid frames are anti-shake image frames cropped from the monitoring screen that can clearly display the rockfall.

[0058] It can be understood that by cropping the valid frames, the amount of data for subsequent processing can be reduced, the blurred frames caused by jitter can be removed, and thus the processing speed and accuracy can be improved to enhance the processing efficiency.

[0059] It can be understood that by cropping the valid frames that clearly display the rockfall from the monitoring screen and performing background reconstruction processing on these frames to remove the static background, a background reconstruction map based on dynamic targets is finally generated. This process provides high-quality input data for subsequent rockfall detection and analysis, and can significantly improve the detection speed and accuracy, thereby enhancing the detection efficiency.

[0060] Furthermore, step S02 further includes: Performing discrete noise removal processing on the valid frames to obtain feature-enhanced frames; Identifying static targets from the feature-enhanced frames; Removing the static targets to obtain a background reconstruction map composed of regions based on dynamic targets.

[0061] It should be noted that discrete noise removal processing is to remove random noise in the image through image processing technology to enhance the image quality; common methods include median filtering, Gaussian filtering, or denoising algorithms based on the frequency domain. The feature-enhanced frames are the results after discrete noise removal processing, in which the target features in the image (such as the contour and texture of the rockfall) are enhanced. Static targets are objects that do not move relative to the background in the image.

[0062] It can be understood that through discrete noise removal processing, enhancing target features, identifying and removing static targets, a background reconstruction map is finally generated, providing high-quality input data for subsequent rockfall detection and analysis.

[0063] Exemplarily, to help understand the implementation process of the rockfall detection method obtained by combining the above embodiments, please refer to Figure 4 , Figure 4 A brief flow schematic diagram of a rockfall detection method is provided, specifically: 1. Preprocessing stage: A. Implementing slight image anti-shake using the optical flow method. Since the road side slope detection environment is complex, it is difficult to ensure the absolute stability of the image (such as a strong crosswind causing slight camera shake and resulting in slight image shake). Therefore, an anti-shake algorithm is required to maintain the image features as stable as possible, which will facilitate the subsequent algorithm. The main method is to extract image features, match the image data of the previous n frames, select a stable image, and crop the effective area to achieve image anti-shake.

[0064] B. Using the inter-frame difference method, compare the consecutive anti-shake images of the previous n frames, reconstruct the background, remove all static backgrounds, and retain the coordinate information of all dynamic targets within the monitoring field of view.

[0065] 2. Inference stage: This part is divided into two parallel stages, without a specific order: A. Use a deep learning algorithm (such as the YOLO series), put the image after background reconstruction into the trained model, and output the coordinate box, target category, and confidence of the target.

[0066] B. Use the coordinate box described in 1.B to intercept the corresponding area in the original image, put it into a classification network (such as the resnet series network), and output the confidence and target category.

[0067] 3. Post-processing stage: A. Combine the two sets of confidence and category data to eliminate non-warning targets.

[0068] B. Use the two sets of confidence and coordinate box data to dynamically allocate the weights of the Kalman filter to achieve smooth prediction of the detection box and reduce the occurrence of detection box jumping problems.

[0069] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the rockfall detection method of this application. Based on this technical concept, more forms of simple transformations are within the protection scope of this application.

[0070] This application also provides a rockfall detection device. Please refer to Figure 5 The rockfall detection device includes: An acquisition module 10 for acquiring a monitoring image of the rockfall on the road side slope; A processing module 20 for performing static background removal processing on the monitoring image to obtain a background reconstruction map and performing regional annotation on the dynamic targets in the monitoring image to obtain the dynamic regions of the dynamic targets; A detection module 30 for inputting the background reconstruction map into a preset YOLO algorithm to obtain a first detection result of the dynamic target, and inputting the dynamic region into a preset classification network to obtain a second detection result of the dynamic target; A determination module 40, configured to determine the landing point of the rockfall based on the first detection result and the second detection result.

[0071] Optionally, the determination module 40 is further configured to fuse the first detection result and the second detection result based on a preset Kalman filter to obtain a fused detection result; predict the landing point path of the rockfall based on the detection frame marking the rockfall in the fused detection result; and determine the landing point of the rockfall based on the landing point path.

[0072] Optionally, the determination module 40 is further configured to screen first detection data of the rockfall from the first detection result and screen second detection data of the rockfall from the second detection result; and perform weighted fusion processing on the first detection data and the second detection data based on a preset Kalman filter to obtain a fused detection result of the rockfall.

[0073] Optionally, the first detection data includes a first confidence level for detecting the rockfall and a target coordinate frame of the rockfall in the background reconstruction map, and the second detection data includes a second confidence level for detecting the rockfall. The determination module 40 is further configured to perform Kalman filter weighted processing on the target coordinate frame and the target area based on the first confidence level and the second confidence level to obtain a weighted coordinate frame and a weighted area, where the target area is a dynamic area corresponding to the rockfall; and fuse the weighted coordinate frame and the weighted area to obtain a fused detection result that completely describes the trajectory of the rockfall.

[0074] Optionally, the processing module 20 is further configured to crop an effective image in which the rockfall is clearly shown in a preset number of frames from the monitoring image; and perform background reconstruction on the effective image to remove the static background in the effective image to obtain a background reconstruction map composed of areas of dynamic targets.

[0075] Optionally, the processing module 20 is further configured to perform discrete noise removal processing on the effective image to obtain a feature-enhanced image; identify static targets from the feature-enhanced image; and remove the static targets to obtain a background reconstruction map composed of areas of dynamic targets.

[0076] The rockfall detection device provided in this application adopts the rockfall detection method in the above embodiment, and can solve the technical problem of low efficiency in rockfall detection. Compared with the prior art, the beneficial effects of the rockfall detection device provided in this application are the same as those of the rockfall detection method provided in the above embodiment, and other technical features in the rockfall detection device are the same as those disclosed in the method of the above embodiment, and will not be elaborated here.

[0077] The present application provides a rockfall detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the rockfall detection method in the first embodiment above.

[0078] Reference is made below Figure 6 , which shows a schematic structural diagram of a rockfall detection device suitable for implementing the embodiments of the present application. The rockfall detection device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description: tablet computers), PMPs (Portable Media Player: portable multimedia players), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The rockfall detection device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0079] As Figure 6 shown, the rockfall detection device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can execute various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the rockfall detection device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the rockfall detection device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a rockfall detection device with various systems, it should be understood that it is not required to implement or have all the shown systems. More or fewer systems may be alternatively implemented or had.

[0080] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product that includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by a processing device 1001, the above functions defined in the methods of the embodiments disclosed in the present application are executed.

[0081] The rockfall detection device provided by the present application adopts the rockfall detection method in the above embodiments, and can solve the technical problem of low efficiency in rockfall detection. Compared with the prior art, the beneficial effects of the rockfall detection device provided by the present application are the same as those of the rockfall detection method provided by the above embodiments, and other technical features in the rockfall detection device are the same as those disclosed in the method of the previous embodiment, which will not be elaborated here.

[0082] It should be understood that each part disclosed in the present application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0083] As mentioned above, the above are only the specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all of them should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0084] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the rockfall detection method in the above embodiments.

[0085] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0086] The above computer-readable storage medium can be included in the rockfall detection device; it can also exist separately without being assembled into the rockfall detection device.

[0087] The above computer-readable storage medium carries one or more programs. When the one or more programs are executed by the rockfall detection device, the rockfall detection device is caused to: obtain a monitoring image of the rockfall on the roadside slope; perform static background removal processing on the monitoring image to obtain a background reconstruction image, and perform region annotation on the dynamic targets in the monitoring image to obtain the dynamic regions of the dynamic targets; input the background reconstruction image into a preset YOLO algorithm to obtain a first detection result for the dynamic targets, and input the dynamic regions into a preset classification network to obtain a second detection result for the dynamic targets; based on the first detection result and the second detection result, determine the landing point of the rockfall.

[0088] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN: Local Area Network) or a wide area network (WAN: Wide Area Network), or it can be connected to an external computer (for example, by connecting through an Internet service provider via the Internet).

[0089] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of the code, and this module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0090] The modules described in the embodiments of this application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation on the unit itself in some cases.

[0091] The readable storage medium provided by this application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for performing the above-mentioned rockfall detection method, and can solve the technical problem of low efficiency in rockfall detection. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this application are the same as those of the rockfall detection method provided in the above embodiments, and will not be elaborated here.

[0092] The present application also provides a computer program product, including a computer program which, when executed by a processor, implements the steps of the rockfall detection method as described above.

[0093] The computer program product provided by the present application can solve the technical problem of low efficiency in rockfall detection. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the rockfall detection method provided by the above embodiments, and will not be elaborated here.

[0094] The above are only partial embodiments of the present application, and thus do not limit the patent scope of the present application. Any equivalent structural transformation made under the technical concept of the present application by using the content of the specification and drawings of the present application, or direct / indirect application in other related technical fields, is included in the patent protection scope of the present application.

Claims

1. A rockfall detection method, characterized in that: The method includes: Obtain monitoring images of rockfall on the road side slope; Performing static background removal processing on the monitoring picture to obtain a background reconstruction image, and marking the region of the dynamic target in the monitoring picture to obtain the dynamic region of the dynamic target; Inputting the background reconstruction image into a preset YOLO algorithm to obtain a first detection result of the dynamic target, and inputting the dynamic area into a preset classification network to obtain a second detection result of the dynamic target; Based on the first detection result and the second detection result, the landing point of the falling rock is determined.

2. The method according to claim 1, characterized in that The step of determining the falling point of the falling rock based on the first detection result and the second detection result includes: Based on a preset Kalman filter, the first detection result and the second detection result are fused to obtain a fused detection result; Predicting a path where the rockfall will fall based on a detection frame marking the rockfall in the fusion detection result; Based on the landing path, the landing point of the falling rock is determined.

3. The method according to claim 2, characterized in that The step of fusing the first detection result and the second detection result based on the preset Kalman filter to obtain a fused detection result includes: Filtering first detection data of the rockfall from the first detection result, and filtering second detection data of the rockfall from the second detection result; Based on a preset Kalman filter, weighted fusion processing is performed on the first detection data and the second detection data to obtain a fusion detection result of the falling rock.

4. The method according to claim 3, characterized in that The first detection data includes a first confidence level for the rockfall detection and a target coordinate frame of the rockfall in the background reconstruction image, the second detection data includes a second confidence level for the rockfall detection, and the step of performing weighted fusion processing on the first detection data and the second detection data based on a preset Kalman filter to obtain a fusion detection result of the rockfall includes: Based on the first confidence level and the second confidence level, performing Kalman filter weighted processing on the target coordinate frame and the target area to obtain a weighted coordinate frame and a weighted area, wherein the target area is a dynamic area corresponding to the rockfall; The weighted coordinate frame and the weighted region are fused to obtain a fusion detection result that fully describes the trajectory of the falling rock.

5. The method according to claim 1, characterized in that The step of performing static background removal processing on the monitoring image to obtain a background reconstruction image comprises: Cutting out effective images clearly showing the rockfall from a preset number of frames from the monitoring images; The background of the effective picture is reconstructed, and the static background in the effective picture is removed to obtain a background reconstruction image composed of regions based on dynamic targets.

6. The method according to claim 5, characterized in that The step of reconstructing the background of the effective picture, removing the static background in the effective picture, and obtaining a background reconstruction image composed of regions based on dynamic targets comprises: Performing discrete noise removal processing on the effective picture to obtain a feature-enhanced picture; identifying a static target from the feature-enhanced picture; The static target is removed to obtain a background reconstruction image composed of regions based on the dynamic target.

7. A rockfall detection device, characterized in that: The device comprises: An acquisition module is used to acquire monitoring images of rockfall on the road side slope; A processing module, used for performing static background removal processing on the monitoring picture to obtain a background reconstruction image, and for marking the area of ​​the dynamic target in the monitoring picture to obtain the dynamic area of ​​the dynamic target; A detection module, used for inputting the background reconstruction image into a preset YOLO algorithm to obtain a first detection result of the dynamic target, and inputting the dynamic area into a preset classification network to obtain a second detection result of the dynamic target; A determination module is used to determine the landing point of the falling rock based on the first detection result and the second detection result.

8. A rockfall detection device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the rockfall detection method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the rockfall detection method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the rockfall detection method according to any one of claims 1 to 6 are implemented.

Images