Collapse rockfall intelligent monitoring method and system based on video image

Through intelligent video image monitoring technology, combined with seed filling algorithm, convolutional neural network and Kalman filtering algorithm, the existing rockfall monitoring hardware requirements and high misjudgment rate are solved, and low-cost and high-precision rockfall monitoring and Quick warning.

CN119091345APending Publication Date: 2024-12-06CHENGAN ZHILIAN (CHONGQING) INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 5 Cited by

Patent Information

Application Number
CN202411075664.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing rockfall monitoring technology has problems such as high hardware requirements, high cost, low real-time performance, high misjudgment rate and difficulty in large-scale promotion, especially in monitoring of dangerous rock mass on high steep slopes.

Method used

Using an intelligent monitoring method based on video images, the background and motion targets are segmented by obtaining image sequences of consecutive multiple frames in the video image, and the target classification is performed using seed filling algorithm and convolutional neural network recognition model, combining Kalman filtering and Hungary The algorithm performs target trajectory matching, reduces equipment requirements and improves monitoring accuracy and real-time performance.

Benefits of technology

Large-scale promotion and low-cost rockfall monitoring at the project site have been achieved, monitoring accuracy and real-time performance have been improved, and it has helped to quickly fall warning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091345A_ABST
    Figure CN119091345A_ABST
Patent Text Reader

Abstract

The invention provides a collapsed rockfall intelligent monitoring method and system based on a video image, and the method comprises the steps: obtaining a continuous multi-frame image sequence in the video image, and segmenting a video image background and a moving target in the image sequence; searching a connected region of moving target pixels by adopting a seed filling algorithm, and obtaining a detection frame according to the connected region; image data of a monitoring site is collected and subjected to data labeling for convolutional neural network training, a target rockfall recognition model is obtained, an image corresponding to a detection frame is input, and a moving target classification result is output; a target motion state is obtained according to the detection frame, the motion state of the target is predicted based on Kalman filtering, and a target rockfall recognition result is obtained; performing multi-target matching of continuous frames by adopting a Hungary algorithm to obtain a target motion track; and according to the target motion track, determining a motion target conforming to rockfall motion to obtain a rockfall target. According to the invention, equipment requirements and cost can be reduced, and rockfall monitoring precision and real-time performance can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of rockfall monitoring, and in particular to a method and system for intelligently monitoring collapse and rockfall based on video images. Background Art

[0002] Rockfall caused by the collapse of dangerous rock masses on steep slopes can pose a serious threat to life safety. Therefore, it is necessary to set up a rockfall monitoring system in areas where rockfall often occurs. Common rockfall monitoring methods in the existing technology include manual video monitoring, fiber grating sensor detection, 3D laser scanner monitoring, and target detection monitoring.

[0003] Among them, manual video monitoring is to install cameras in areas where rockfalls often occur. Monitoring personnel use the cameras to identify rockfall images. This method is technically simple to implement, but its reliability is affected by human factors, and the long-term labor cost is also higher than the cost of installing and operating automated monitoring equipment.

[0004] Fiber Bragg grating sensor detection is to bury the sensing fiber in the location to be monitored, monitor the environment for impact and vibration, and analyze the rockfall situation. Fiber optic sensors have outstanding advantages such as small size, high sensitivity, anti-electromagnetic interference, and distributed measurement. However, this method cannot distinguish whether the vibration is caused by objects other than the target through sensor information, and the misjudgment rate is high. In addition, as a contact monitoring device, the deployment of fiber grating sensors is restricted by the environment.

[0005] The 3D laser scanner monitoring method is to use the ground laser scanner to obtain the slope point cloud data, and to register the point cloud data collected at two different times to achieve rockfall monitoring. This method has high requirements for monitoring equipment and is difficult to promote on a large scale at the construction site.

[0006] The target detection method is to continuously collect images of the monitoring location through the camera, and use the target tracking algorithm to mark the rockfall trajectory. However, this method has the defect that the video data transmission and processing puts a lot of pressure on the cloud service.

[0007] Therefore, there is an urgent need for an intelligent monitoring method for collapse and rockfall that can reduce the hardware requirements for data acquisition, improve the real-time performance of rockfall monitoring, and reduce communication costs. Summary of the invention

[0008] Based on this, it is necessary to provide an intelligent monitoring method and system for rockfall and collapse based on video images to address the above technical issues.

[0009] A method for intelligent monitoring of collapse and rockfall based on video images comprises the following steps: obtaining a sequence of multiple consecutive frames in a video image, and segmenting the video screen background and a moving target in the image sequence; searching for the connected area of ​​the moving target pixels by using a seed filling algorithm, and obtaining a detection frame according to the connected area; collecting image data of the monitoring site and performing data annotation for training a convolutional neural network to obtain a target rockfall recognition model, inputting an image corresponding to the detection frame, and outputting a moving target classification result; obtaining a target motion state according to the detection frame, predicting the target motion state based on Kalman filtering, and performing multi-target matching of consecutive frames by using the Hungarian algorithm to obtain a target motion trajectory; determining a moving target that conforms to the rockfall motion according to the target motion trajectory, and obtaining a rockfall target.

[0010] In one of the embodiments, the method of acquiring an image sequence of multiple consecutive frames in a video image and segmenting the video screen background and moving targets in the image sequence includes: acquiring a video image and intercepting an image sequence of multiple consecutive frames in the video image; constructing a corresponding background reference model based on the image sequence of multiple consecutive frames using a background difference algorithm, arranging pixels at the same position in the image sequence of multiple consecutive frames from small to large, and taking the middle value as the grayscale value of the corresponding pixel in the background reference model; subtracting the current frame in the image sequence from the background reference model, and calculating an area whose pixel difference with the background reference model exceeds a set threshold as an assumed moving target mask; and comprehensively calculating the background reference model of the current frame based on the background reference model of the previous frame, the difference between the background reference model of the current frame and the previous frame, and the assumed moving target mask, and segmenting the video screen background and moving targets.

[0011] In one of the embodiments, the connected area of ​​the moving target pixel is searched using a seed filling algorithm, and a detection frame is obtained based on the connected area, including: performing image binarization processing on the current frame, wherein the background image is set to 0 and the foreground image is set to 1; using a seed filling algorithm, searching the neighborhood with one or more pixels of interest as seeds, marking the pixels with the same label when equal pixel values ​​are found, and continuing to search in the neighborhood until no equal pixel values ​​exist, thereby obtaining a connected area; reselecting a seed, searching the next connected area, and framing all the connected areas obtained by the search with a minimum rectangular frame to obtain a detection frame.

[0012] In one embodiment, the image data of the monitoring site is collected and the data is annotated for training a convolutional neural network to obtain a target rockfall recognition model, an image corresponding to a detection frame is input, and a moving target classification result is obtained as an output, including: collecting image data of various objects at the monitoring site and annotating the data, wherein the data annotation includes annotation information of the picture name, picture path and target category; preprocessing the collected image data, wherein the preprocessing includes image denoising, image enhancement and data cleaning, and constructing a data set based on the preprocessed image samples; training the convolutional neural network based on the data set and the corresponding annotation information, initializing the network weights, inputting the annotated image samples in the data set into the convolutional neural network, and outputting the classification results; calculating the cross entropy loss function based on the output and input of the convolutional neural network, updating the network weights through back propagation, and iterating multiple times until the loss function converges to obtain a target rockfall recognition model; inputting the image in the detection frame into the target rockfall recognition model, and outputting the moving target classification result.

[0013] In one of the embodiments, the target motion state is obtained according to the detection frame, the motion state of the target is predicted based on Kalman filtering, and the Hungarian algorithm is used to perform multi-target matching of consecutive frames to obtain the target motion trajectory, including: obtaining the center point coordinates, aspect ratio, height and first-order change speed of the detection frame as the motion state; predicting the motion state of the target in the current frame based on the motion state of the previous frame based on Kalman filtering to obtain a prediction result; calculating the IOU distance between the motion state of the prediction result and the motion state of the connected area detection result as a cost matrix, and using the Hungarian algorithm to perform multi-target matching of consecutive frames; combining the connected area detection result and the prediction result, using the Kalman filter algorithm for updating, and obtaining the optimal estimate of the motion state of the current frame as the output matching result, that is, the target motion trajectory.

[0014] A video image-based intelligent monitoring system for collapse and rockfall, used to implement the video image-based intelligent monitoring method for collapse and rockfall as described above, comprising: power supply facilities, cameras, edge computing boxes, alarms, routers and cloud servers; the power supply facilities are used to power the system; the camera is used to collect on-site video image data to obtain a continuous multi-frame image sequence in the video image; the edge computing box is used to perform intelligent monitoring of collapse and rockfall according to the continuous multi-frame image sequence to obtain a rockfall target; the alarm is used to issue an alarm when a rockfall target is detected; the router is used to connect the network between the camera, the edge computing box, the alarm and the cloud server.

[0015] In one embodiment, the power supply facility includes solar panels, solar controllers and batteries, wherein the solar panels are used to convert solar energy into electrical energy; the batteries are used to store electrical energy; the solar controller is used to control the charging and discharging of the batteries, and control the electrical energy output of the solar cell modules and the batteries to the load according to the power demand of the load.

[0016] In one of the embodiments, the edge computing box is developed based on the Rockchip RK3588 chip and has a built-in image processor.

[0017] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: by acquiring an image sequence of multiple consecutive frames in a video image, the video screen background and the moving target in the image sequence are segmented, a seed filling algorithm is used to search for the connected area of ​​the moving target pixels, and a detection frame is obtained according to the connected area; image data of the monitoring site is collected and data is annotated for training a convolutional neural network to obtain a target rockfall recognition model, an image corresponding to the detection frame is input, and a moving target classification result is obtained as output; the target motion state is obtained according to the detection frame, the motion state of the target is predicted based on the Kalman filter, and the Hungarian algorithm is used to perform multi-target matching of continuous frames to obtain the target motion trajectory; the moving target that meets the rockfall motion is determined according to the target motion trajectory to obtain the rockfall target, which reduces the requirements for monitoring equipment, can be widely promoted on engineering sites, and has low cost, can also improve the rockfall monitoring accuracy and real-time monitoring, and helps to achieve faster rockfall warning. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A schematic diagram of a flow chart of a method for intelligently monitoring rockfall and collapse based on video images in one embodiment;

[0019] Figure 2 The figure is a schematic diagram of the structure of a video image-based intelligent monitoring system for collapse and rockfall in one embodiment. DETAILED DESCRIPTION

[0020] Before describing the specific embodiments of the present invention, the overall concept of the present invention is described as follows:

[0021] The present invention is mainly developed based on the rockfall monitoring process. The current rockfall monitoring method has high hardware requirements and costs, and the real-time performance of rockfall monitoring is low.

[0022] Therefore, the present invention proposes an intelligent monitoring method for collapse and rockfall based on video images, which comprises the following steps: obtaining an image sequence of multiple consecutive frames in a video image, segmenting the video screen background and a moving target in the image sequence, searching the connected area of ​​the moving target pixels by using a seed filling algorithm, and obtaining a detection frame according to the connected area; collecting image data of the monitoring site and performing data annotation for training a convolutional neural network to obtain a target rockfall recognition model, inputting an image corresponding to the detection frame, and outputting a moving target classification result; obtaining the target motion state according to the detection frame, predicting the target motion state based on Kalman filtering, and performing multi-target matching of continuous frames by using the Hungarian algorithm to obtain the target motion trajectory; determining the moving target that meets the rockfall motion according to the target motion trajectory, and obtaining the rockfall target, which reduces the requirements for monitoring equipment, can be widely promoted on engineering sites, and has low cost, can also improve the rockfall monitoring accuracy and real-time monitoring, and helps to achieve faster rockfall warning.

[0023] After introducing the overall concept of the present invention, in order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail by specific implementation methods in combination with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0024] In one embodiment, Figure 1 As shown, a method for intelligent monitoring of rockfall and collapse based on video images is provided, comprising the following steps:

[0025] Step S101, obtaining a sequence of multiple consecutive frames of video images, and segmenting the video screen background and moving objects in the image sequence.

[0026] Specifically, in the intelligent monitoring system for landslides and falling rocks, the camera collects the video images of the scene in real time, captures a sequence of multiple consecutive frames in the video images, segments the video screen background and moving targets in the image sequence, and extracts all moving targets in the video screen.

[0027] Among them, step S101 includes: acquiring a video image, intercepting an image sequence of multiple consecutive frames in the video image; constructing a corresponding background reference model based on the image sequence of multiple consecutive frames using a background difference algorithm, arranging the pixels at the same position in the image sequence of multiple consecutive frames from small to large, and taking the middle value as the grayscale value of the corresponding pixel in the background reference model; subtracting the current frame in the image sequence from its background reference model, and calculating the area where the pixel difference with the background reference model exceeds a set threshold as a presumed moving target mask; comprehensively calculating the background reference model of the current frame based on the background reference model of the previous frame, the difference between the background reference model of the current frame and the previous frame, and the presumed moving target mask, and segmenting the video screen background and the moving target.

[0028] Specifically, after acquiring the video image, a sequence of multiple consecutive frames in the video image is intercepted, and a background difference algorithm is used to construct a corresponding background reference model, and the model is updated in real time to obtain background reference models corresponding to the multiple frames, that is, background images; pixels at the same position in the image sequence are arranged from small to large according to the grayscale value, and the middle value is taken as the grayscale value of the pixel corresponding to the background reference model according to the arrangement result; the background image is obtained according to the background reference model, and in order to detect moving targets, a background update algorithm is used to subtract the current frame image from the background image of the corresponding frame, and the result is subjected to threshold binarization processing, and the area whose pixel difference with the background reference model exceeds the set threshold is calculated as the assumed moving target mask, and the complete foreground moving target is extracted, and the background update algorithm is used to continuously update the background to avoid disturbed targets caused by light and shadow, thereby improving the accuracy of target detection results.

[0029] When segmenting the background and moving targets of the current frame, the background image of the current frame is obtained by comprehensive calculation based on the background reference model of the previous frame, the difference between the background reference model of the current frame and the previous frame, and the assumed target mask, so as to achieve the segmentation between the background and moving targets of the image sequence and obtain the video screen background and moving targets in the current frame, so as to facilitate the subsequent recognition of fallen rocks based on the moving targets.

[0030] Step S102: Use a seed filling algorithm to search for connected areas of moving target pixels, and obtain a detection frame based on the connected areas.

[0031] Specifically, after the segmentation of the video screen background and the moving target is completed, the seed filling algorithm is used to search for the connected area of ​​the moving target pixels. The morphological features of the pixels in the image, such as edges, corners, textures, etc., are calculated through the morphological algorithm to realize applications such as image segmentation, recognition and feature extraction. The scattered binary pixels that may belong to the same object are connected together, and the detection frame is obtained according to the connected area to obtain a clean, complete and distinguishable moving target.

[0032] Among them, step S102 includes: performing image binarization processing on the current frame, wherein the background image is set to 0 and the foreground image is set to 1; using a seed filling algorithm, searching the neighborhood with one or more pixel seeds of interest, marking the pixels with the same label if equal pixel values ​​are found, and continuing to search in the neighborhood until no equal pixel values ​​exist, thereby obtaining a connected area; reselecting a seed, searching for the next connected area, and using a minimum rectangular frame to frame all the connected areas obtained by the search to obtain a detection frame.

[0033] Specifically, after separating the moving target and the background, the current frame is binarized, wherein the background image, i.e., the background of the video screen, is set to 0, and the foreground image, i.e., the moving target, is set to 1; a seed filling algorithm is used to select one or more pixels of interest as seeds, and a search is performed in the neighborhood. When the same pixel value is found, the pixel value is marked with the same label, and the visited mark is performed, and the search for pixels not marked as visited is continued in the neighborhood until the same pixel value does not exist in the neighborhood, and a connected area is obtained; then one or more pixels of interest are selected as seeds, and the search for the next connected area is continued, and all connected areas are framed with a minimum rectangular frame to obtain a detection frame, thereby realizing the analysis of the connected area of ​​the moving target pixels, and realizing the detection and analysis of the moving target.

[0034] Step S103, collect image data of the monitoring site and perform data annotation for training a convolutional neural network, obtain a target rockfall recognition model, input the image corresponding to the detection frame, and output the moving target classification result.

[0035] Specifically, image data from the monitoring site is collected and annotated for convolutional neural network training. Since the features of different fallen rocks, such as shape and color, have a large span and are not uniform, a residual convolutional neural network, such as the residual convolutional neural network ResNet50, can be used to build a model to obtain a target rockfall recognition model. The image corresponding to the detection frame is input to the model, and the moving target classification result is output to achieve the classification of moving targets, such as plants, animals, people, vehicles, etc., and the above categories are marked as noise targets.

[0036] Among them, step S103 includes: collecting image data of various objects at the monitoring site and performing data annotation, the data annotation includes annotation information of picture name, picture path and target category; preprocessing the collected image data, the preprocessing includes image denoising, image enhancement and data cleaning, and constructing a data set according to the preprocessed image samples; training the convolutional neural network according to the data set and the corresponding annotation information, initializing the network weights, inputting the labeled image samples in the data set into the convolutional neural network, and outputting the classification results; calculating the cross entropy loss function according to the output and input of the convolutional neural network, updating the network weights through back propagation, iterating multiple times until the loss function converges, and obtaining a target rockfall recognition model; inputting the image in the detection box into the target rockfall recognition model, and outputting the moving target classification result.

[0037] Specifically, image data of various objects at the monitoring site are collected, and the image data are annotated according to the object category. The data annotation may be annotation information such as the image name, image path, and target category. The collected image data are preprocessed. Image preprocessing may be operations such as image denoising, image enhancement, and data cleaning. A data set is constructed based on the preprocessed image samples. Image enhancement includes operations such as randomly changing the brightness, cropping, scaling, and flipping of the image data. The data set is expanded by processing the image data to suppress overfitting.

[0038] Since the image samples included in the data set include image data and corresponding annotation information, the constructed data set can be used to train the convolutional neural network. After initializing the network weights, the labeled image samples in the data set are input into the convolutional neural network, and the classification results are output. The cross entropy loss function is calculated based on the output and input of the convolutional neural network, and the network weights are updated through back propagation. Multiple iterations are performed until the loss function converges, and finally a trained target rockfall recognition model is obtained; the image in the detection frame is input into the target rockfall model, and the moving target classification result is output to obtain a clean, complete, and distinguishable moving target, and the interference targets marked as plants, animals, personnel, vehicles, etc. in the moving target classification results are eliminated, thereby achieving accurate monitoring of rockfalls.

[0039] Step S104, obtaining the target motion state according to the detection frame, predicting the target motion state based on Kalman filtering, and performing multi-target matching of continuous frames using the Hungarian algorithm to obtain the target motion trajectory.

[0040] Specifically, according to the target framed by the detection frame, the corresponding target motion state is obtained. Kalman filtering can realize the tracking of multiple targets. Therefore, the motion state of the target is predicted based on Kalman filtering to realize the recognition of the target. The Hungarian algorithm is used to perform multi-target matching of continuous frames. By matching the same target in continuous frames, the target motion trajectory is obtained, so that the fallen rocks in the target can be identified according to the target motion trajectory.

[0041] Among them, step S104 includes: obtaining the center point coordinates, aspect ratio, height and first-order change speed of the detection frame as the motion state; predicting the motion state of the target in the current frame based on the motion state of the previous frame based on the Kalman filter to obtain the prediction result; calculating the IOU distance between the motion state of the prediction result and the motion state of the connected area analysis detection result as the cost matrix, and using the Hungarian algorithm to perform multi-target matching of consecutive frames; combining the connected area detection result and the prediction result, using the Kalman filter algorithm for update, and obtaining the optimal estimate of the motion state of the current frame as the output matching result, that is, the target motion trajectory.

[0042] Specifically, in order to obtain the motion trajectory of the target in a continuous video sequence, the center point coordinates, aspect ratio, height and first-order change speed of the detection frame are obtained according to the detection frame as the motion state; based on the motion state of the previous frame, the state is updated through Kalman filtering, and each tracker must predict the motion state of the target in the current frame based on the previous motion state as the prediction result; the IOU (Intersection over Union) distance between the motion state of the predicted result and the motion state of the connected area analysis detection result is calculated as the cost matrix, and the Hungarian algorithm is used to perform multi-target matching of continuous frames to obtain the target matching result, and separate the same target in multiple continuous frames; combined with the connected area detection result and the prediction result, the Kalman filter algorithm is used to update the state to obtain the optimal estimate of the motion state of the current frame as the output matching result, that is, the target motion trajectory, and the positions of the same target in multiple continuous frames are connected according to the time sequence to realize the recognition of the target motion trajectory, so as to facilitate the recognition of the falling rock target according to the target motion trajectory.

[0043] Step S105, determining a moving target that conforms to the rockfall motion according to the target motion trajectory, and obtaining the rockfall target.

[0044] Specifically, since there may be multiple interference targets in the obtained target, in order to further identify the falling rocks in the target, the acceleration of the target can be calculated according to the obtained moving target trajectory and analyzed whether it conforms to the falling rock motion, so as to lock the falling rock target and achieve accurate identification of the falling rocks.

[0045] In this embodiment, by acquiring an image sequence of multiple consecutive frames in a video image, the video screen background and the moving target in the image sequence are segmented, and a seed filling algorithm is used to search for the connected area of ​​the moving target pixels, and a detection frame is obtained according to the connected area; image data of the monitoring site is collected and data annotation is performed for convolutional neural network training to obtain a target rockfall recognition model, an image corresponding to the detection frame is input, and a moving target classification result is obtained as output; the target motion state is obtained according to the detection frame, the motion state of the target is predicted based on the Kalman filter, and the Hungarian algorithm is used to perform multi-target matching of continuous frames to obtain the target motion trajectory; the moving target that meets the rockfall motion is determined according to the target motion trajectory to obtain the rockfall target, which reduces the requirements for monitoring equipment, can be promoted on a large scale at the engineering site, and is low in cost, can also improve the rockfall monitoring accuracy and real-time monitoring, and help to achieve faster rockfall warning.

[0046] like Figure 2As shown, a collapse and rockfall intelligent monitoring system 20 based on video images is provided, which is used to implement a collapse and rockfall intelligent monitoring method based on video images as described above, including: power supply facilities 1, camera 2, edge computing box 3, alarm 4, router 5 and cloud server 6; the power supply facility 1 is used for system power supply; the camera 2 is used for collecting on-site video image data to obtain a continuous multi-frame image sequence in the video image; the edge computing box 3 is used for performing intelligent monitoring of collapse and rockfall according to the continuous multi-frame image sequence to obtain the rockfall target; the alarm 4 is used to issue an alarm when a rockfall target is detected; the router 5 is used to connect the network between the camera 2, the edge computing box 3, the alarm 4 and the cloud server 6.

[0047] In this embodiment, the network of the camera 2, the edge computing box 3, the alarm 4 and the cloud server 6 is connected through the router 5, and the power supply facility 1 is used to realize the power supply of the whole system; the camera 2 is used to collect on-site video image data to obtain a continuous frame image sequence for rockfall monitoring; the edge computing box 3 is used to use the above-mentioned video image-based intelligent monitoring method for collapse and rockfall to perform intelligent monitoring of collapse and rockfall according to the obtained image sequence, and obtain the rockfall target. When the rockfall target is monitored, an alarm is issued through the alarm 4, thereby realizing accurate monitoring of rockfall, reducing communication delay and bandwidth requirements, and being able to improve the real-time performance of rockfall monitoring, which helps to achieve faster rockfall warning.

[0048] Specifically, camera 2 is used as a data acquisition device. Compared with monitoring equipment such as fiber grating sensors and three-dimensional laser scanners, camera 2 has lower cost and is easy to deploy, and the construction is less affected by the on-site environment. Therefore, camera 2 can be deployed in large quantities and can be applied to large-scale collapse and rockfall monitoring scenarios. However, the spectral signals and radar signal lights in traditional monitoring solutions are difficult to distinguish between rockfall and other moving objects in the environment, which can easily lead to misjudgment.

[0049] Based on video images, intelligent algorithms are used to avoid the impact of non-rockfall targets from a semantic perspective. Considering the large volume of video image data, direct backhaul will take up a lot of communication resources, and compressed backhaul will cause information distortion. Data processing is performed through edge computing, which can effectively reduce communication delays and loan requirements without losing information.

[0050] In one embodiment, the power supply facility 1 includes a solar panel 11, a solar controller 12 and a battery 13, wherein the solar panel 11 is used to convert solar energy into electrical energy; the battery 13 is used to store electrical energy; the solar controller 12 is used to control the charging and discharging of the battery 13, and control the electrical energy output of the solar cell module and the battery 13 to the load according to the power demand of the load.

[0051] In one embodiment, the edge computing box is developed based on the Rockchip RK3588 chip, uses virtual networking technology, and has a built-in image processor.

[0052] Specifically, the above-mentioned intelligent monitoring methods for collapse and rockfall based on video images are all carried out in the edge computing box 3, which adopts virtual networking computing, has a built-in image signal processor, contains rich display, peripheral and expansion interfaces, and has powerful video encoding and decoding capabilities. The edge computing results can be directly linked with the on-site alarm 4, and can also be transmitted back to the cloud server 6 through the network.

[0053] In one embodiment, the installation of camera 2 should ensure the following conditions: there is no line of sight obstruction in the installation area, and the 0 magnification viewing angle of camera 2 can cover the rockfall monitoring area; when the camera 2 is at maximum magnification, the outermost area of ​​the rockfall monitoring area can be seen; if camera 2 is installed by means of a pole, a suitable pole and foundation must be used to ensure that the equipment does not shake in extreme weather; camera 2 should try to avoid direct sunlight on the lens; camera 2 should be installed to ensure that the lens is vertical and the image presented by the camera is a vertical image, otherwise it will cause missed reports of rockfall monitoring.

[0054] The video image data collected by camera 2 is transmitted to the edge computing box for processing. First, all moving targets in the video screen are extracted, and other interference such as plants, animals, people, and vehicles are identified and eliminated through the convolutional neural network; then the noise targets are eliminated, and the movement trajectory of the effective targets is tracked and recorded; finally, according to the movement characteristics of the collapsed rocks, the collapsed rocks are locked from the moving targets. The data processing results are returned to the cloud server. If dangerous rocks are identified in the results, an alarm command will be issued to the alarm to form an on-site rockfall alarm.

[0055] Compared with the method of transmitting data back to the cloud server for processing and then returning the results, edge computing is implemented through the edge computing box 3, which improves real-time performance and helps to achieve faster rockfall warnings. In addition, by transmitting the results of edge computing instead of huge amounts of original collected data, communication costs can be further reduced.

[0056] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0057] Obviously, those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a computer storage medium (ROM / RAM, magnetic disk, optical disk) and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Therefore, the present invention is not limited to any specific combination of hardware and software.

[0058] The above contents are further detailed descriptions of the present invention in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A method for intelligent monitoring of rockfall and collapse based on video images, characterized in that: The following steps are involved: Acquire a continuous multi-frame image sequence in the video image, and segment the video screen background and the moving target in the image sequence; A seed filling algorithm is used to search for a connected area of ​​the moving target pixel, and a detection frame is obtained according to the connected area; Collect image data from the monitoring site and annotate the data for training the convolutional neural network to obtain a target rockfall recognition model. Input the image corresponding to the detection frame and output the moving target classification result: The target motion state is obtained according to the detection frame, the target motion state is predicted based on Kalman filtering, and the Hungarian algorithm is used to perform multi-target matching of continuous frames to obtain the target motion trajectory; A motion target that conforms to the rockfall motion is determined according to the target motion trajectory to obtain a rockfall target.

2. The method for intelligent monitoring of rockfall and collapse based on video images according to claim 1, characterized in that: The step of acquiring a sequence of multiple consecutive frames of video images and segmenting the video screen background and moving targets in the sequence of images includes: Acquire a video image, and capture a sequence of multiple consecutive frames of the video image; According to the continuous multi-frame image sequence, a corresponding background reference model is constructed using a background difference algorithm, the pixels at the same position in the continuous multi-frame image sequence are arranged from small to large, and the middle value is taken as the grayscale value of the corresponding pixel in the background reference model; Subtracting the current frame in the image sequence from the background reference model, and calculating an area where the pixel difference with the background reference model exceeds a set threshold as a mask of an assumed moving target; According to the background reference model of the previous frame, the difference between the background reference model of the current frame and the previous frame, and the assumed moving target mask, the background reference model of the current frame is calculated comprehensively, and the video screen background and moving target are segmented.

3. The method for intelligent monitoring of rockfall and collapse based on video images according to claim 2 is characterized in that: The step of searching the connected area of ​​the moving target pixel by using the seed filling algorithm and obtaining the detection frame according to the connected area includes: Performing image binarization processing on the current frame, wherein the background image is set to 0 and the foreground image is set to 1; A seed filling algorithm is used to search the neighborhood with one or more pixels of interest as seeds. If equal pixel values ​​are found, they are marked with the same label and the search is continued in the neighborhood until there are no equal pixel values, thus obtaining a connected area. The seed is reselected to search for the next connected area, and all the connected areas obtained by the search are framed with the minimum rectangular frame to obtain a detection frame.

4. The method for intelligent monitoring of rockfall and collapse based on video images according to claim 1, characterized in that: The image data of the monitoring site is collected and annotated for data training of a convolutional neural network to obtain a target rockfall recognition model, an image corresponding to a detection frame is input, and a moving target classification result is output, including: Collect image data of various objects at the monitoring site and perform data annotation, wherein the data annotation includes annotation information of the image name, image path and target category; Preprocess the collected image data, including image denoising, image enhancement and data cleaning, and construct a data set based on the preprocessed image samples; Training the convolutional neural network according to the data set and the corresponding annotation information, initializing the network weights, inputting the annotated image samples in the data set into the convolutional neural network, and outputting the classification results; The cross entropy loss function is calculated based on the output and input of the convolutional neural network, and the network weights are updated through back propagation. Multiple iterations are performed until the loss function converges to obtain the target rockfall recognition model. The image in the detection frame is input into the target rockfall recognition model, and the moving target classification result is output.

5. The method for intelligent monitoring of rockfall and collapse based on video images according to claim 3 is characterized in that: The target motion state is obtained according to the detection frame, the target motion state is predicted based on Kalman filtering, and the Hungarian algorithm is used to perform multi-target matching of continuous frames to obtain the target motion trajectory, including: According to the detection frame, the coordinates of the center point, aspect ratio, height and first-order change speed are obtained as the motion state; The motion state of the target in the current frame is predicted based on the motion state of the previous frame and the Kalman filter to obtain the prediction result; Calculate the IOU distance between the motion state of the prediction result and the motion state of the connected region detection result as a cost matrix, and use the Hungarian algorithm to perform multi-target matching of consecutive frames; The connected region detection result and the prediction result are combined and updated using the Kalman filter algorithm to obtain the optimal estimate of the motion state of the current frame as the output matching result, that is, the target motion trajectory.

6. An intelligent monitoring system for rockfall and collapse based on video images, characterized in that: A method for intelligently monitoring rockfall and collapse based on video images according to any one of claims 1 to 5, comprising: Power supply facilities, cameras, edge computing boxes, alarms, routers and cloud servers; The power supply facilities are used for powering the system; The camera is used to collect on-site video image data to obtain a continuous multi-frame image sequence in the video image; The edge computing box is used to perform intelligent monitoring of rockfall and collapse according to the continuous multi-frame image sequence to obtain a rockfall target; The alarm is used to issue an alarm when a rockfall target is detected; The router is used to connect the network between the camera, the edge computing box, the alarm and the cloud server.

7. The intelligent monitoring system for rockfall and collapse based on video images according to claim 6 is characterized in that: The power supply facility includes solar panels, solar controllers and batteries, wherein the solar panels are used to convert solar energy into electrical energy; the batteries are used to store electrical energy; the solar controllers are used to control the charging and discharging of the batteries, and control the electrical energy output of the solar cell modules and batteries to the loads according to the power requirements of the loads.

8. The intelligent monitoring system for rockfall and collapse based on video images according to claim 6 is characterized in that: The edge computing box is developed based on the Rockchip RK3588 chip and has a built-in image processor.

Citation Information

Cited By

  • Road slope rockfall detection method based on improved FCOS

    CN119399558A

  • Mineral identification method and device and storage medium

    CN119580016A

  • Rockfall detection method, device and equipment, storage medium and product

    CN120147763A

  • Steel structure bridge framework area rockfall monitoring system

    CN120525512A

  • Intelligent AI target detection method and system for 3D trend of elevator door

    CN120544187A