Rockfall motion tracking method and system based on computer vision

Abstract

The invention discloses a rockfall motion tracking method and system based on computer vision. The method comprises the steps that an improved Gaussian mixture model (PGMM) is adopted to monitor and detect a real-time video stream, and a moving target is extracted; classifying the detected target by using a YOLOv5 deep learning model, and identifying whether the target is rockfall; in combination with the output of the PGMM and the YOLOv5 model, screening and accurately identifying a rockfall target; calculating and predicting a motion track of the rockfall based on the rockfall motion model and a free falling body formula; and outputting the motion trail and the predicted position of the rockfall, generating early warning information and displaying the early warning information in real time. The method can effectively improve the precision and real-time performance of rockfall motion monitoring, is suitable for complex environments such as high and steep mountainous areas and construction sites, provides important rockfall disaster early warning and prevention and control support, and has high adaptability and wide application prospects.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a rockfall motion tracking method and system based on computer vision. Background Art

[0002] In many engineering construction projects, especially those located in high and steep mountainous areas, rockfall, as a common geological disaster, seriously threatens the safety of projects and personnel. Rockfall may not only cause direct property losses and casualties, but also trigger secondary disasters such as traffic interruptions and landslides, causing serious damage to the surrounding environment and infrastructure. Therefore, rockfall monitoring and prevention has become one of the key issues in mountainous engineering construction.

[0003] Although existing rockfall monitoring technologies, such as sensors, radars, and laser scanning, can provide rockfall information to a certain extent, they still have obvious limitations. These traditional technologies usually rely on the deployment of equipment in a specific area, and are mostly suitable for monitoring in fixed places, making it difficult to achieve large-scale, real-time dynamic monitoring. In practical applications, the detection of moving targets in rockfall events faces many challenges from complex environmental factors, including background similarity interference, changes in lighting conditions, occlusion, and other issues. These factors make existing monitoring methods ineffective in complex environments, especially in low-cost and large-scale real-time monitoring, making it difficult to achieve ideal results.

[0004] In addition, the high cost of traditional technologies and their poor adaptability to terrain, environment and construction sites have limited their widespread application in most engineering projects. Although rockfall disasters occur frequently on site, research and application on rockfall tracking, disaster prevention and emergency response are still insufficient. Traditional monitoring methods are difficult to provide sufficient information in an efficient and low-cost manner, and cannot achieve timely warning and effective response to sudden rockfall disasters.

[0005] Given the limitations of existing technologies, a new method is urgently needed to monitor rockfall in real time, provide effective warnings, and provide key information for the prevention and control of rockfall disasters. Computer vision-based technology, combined with modern target detection, tracking and prediction methods, has the advantages of real-time and efficient detection and tracking of moving targets in complex environments, and can provide new ideas and solutions for solving this problem. Summary of the invention

[0006] In view of the defects of the prior art, the present invention provides a rockfall motion tracking method and system based on computer vision.

[0007] In order to achieve the above invention object, the technical solution adopted by the present invention is as follows:

[0008] A rockfall motion tracking method based on computer vision comprises the following steps:

[0009] S1: Use improved Gaussian mixture model (PGMM) for real-time video stream monitoring and moving target detection.

[0010] S2: Classify the detected moving targets based on the YOLOv5 model and identify whether they are falling rocks.

[0011] S3: Combine the outputs of the PGMM and YOLOv5 models to filter and accurately identify rockfall targets.

[0012] S4: Calculate and predict the trajectory of falling rocks based on the rockfall motion model and free fall formula.

[0013] S5: Based on the calculation result of step S4, the movement trajectory and predicted position of the falling rock are output, and the alarm information is generated and displayed.

[0014] Furthermore, the PGMM model includes:

[0015] For each pixel, if multiple Gaussian distributions are matched successfully, all matching Gaussian distributions are updated instead of just the first matching Gaussian distribution.

[0016] In the background update process, a dynamic learning rate is introduced to adapt to background changes at different stages by adjusting the learning rate, and the background update rate is dynamically controlled to improve the detection response speed of rockfall events.

[0017] When multiple Gaussian distributions fail to match successfully, a new Gaussian distribution is created, and the weight of each Gaussian distribution is adjusted by normalization to ensure the stability and accuracy of background modeling.

[0018] Furthermore, the specific steps of S1 are as follows:

[0019] A Gaussian mixture model (GMM) is constructed to describe the image background. The intensity distribution of each pixel in the image is modeled through multiple Gaussian distributions, and statistical information is used for modeling in order to accurately depict the background and extract moving targets.

[0020] For the background of each pixel, Gaussian distribution is used for background modeling, and the pixel intensity is separated according to the matching degree with Gaussian distribution, and the background model is updated.

[0021] Dynamically adjust the learning rate and dynamically adjust the update rate of the model by introducing the standard deviation parameter S: improve the response capability to sudden rockfall events.

[0022] Through the matching process, the Gaussian distribution weights are updated to further improve the detection accuracy of moving targets.

[0023] Furthermore, the specific steps of S2 are as follows:

[0024] The YOLOv5 network structure is used for target detection, including adaptive anchor box calculation, image preprocessing and Mosaic data enhancement.

[0025] For the detected targets, the FPN+PAN structure in the YOLOv5 network and the generalized intersection-over-union loss (GIoULoss) are used to perform target bounding box regression for precise positioning and classification.

[0026] In the optimization process of YOLOv5, a small target detection layer is added and the depth and width of the network are adjusted to adapt to the detection of small and irregularly shaped falling rock targets in complex backgrounds.

[0027] Furthermore, the specific steps of S3 are as follows:

[0028] The location and category of the falling rock target are determined by combining the outputs of the PGMM and YOLOv5 models through the intersection over union (IoU) and non-maximum suppression (NMS) algorithms.

[0029] According to the confidence threshold of the model, the final rockfall targets are screened out and further classified.

[0030] Furthermore, the specific steps of S4 are as follows:

[0031] The particle concentration method is used to divide the rockfall motion into several stages: free fall, rolling and sliding, and collision.

[0032] During the free fall stage, the classic parabolic motion formula is used to calculate the motion trajectory of the falling rock.

[0033] During the sliding stage, the Coulomb dynamic friction model is used to describe the sliding friction and calculate the sliding trajectory of the falling rock.

[0034] During the collision phase, the velocity change of the rockfall colliding with the ground is taken into account, and the collision restitution coefficient is used to calculate the velocity and position changes before and after the collision.

[0035] The present invention also discloses a rockfall motion tracking system based on computer vision, which can be used to implement the above-mentioned rockfall motion tracking method, specifically comprising:

[0036] Video monitoring module: The video stream of the monitored area is obtained in real time through the camera as the input data source.

[0037] Moving target detection module: Based on the improved Gaussian mixture model PGMM, the background of the video stream is modeled and moving targets are detected.

[0038] Target classification and recognition module: Use the YOLOv5 model to classify the detected moving targets and determine whether they are falling rocks.

[0039] Rockfall precision identification module: Combines the outputs of the PGMM and YOLOv5 models to screen and accurately identify targets to confirm whether they are rockfall targets.

[0040] Rockfall trajectory prediction module: Calculate and predict the trajectory of rockfall based on the rockfall motion model and free fall formula.

[0041] Result output and alarm module: outputs the motion trajectory and predicted position of falling rocks according to the prediction results, and generates alarm information.

[0042] The present invention also discloses a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned rockfall motion tracking method when executing the program.

[0043] The present invention also discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned rockfall motion tracking method is implemented.

[0044] Compared with the prior art, the advantages of the present invention are:

[0045] 1. By combining the improved Gaussian mixture model (PGMM) and the YOLOv5 model, the present invention can detect and identify rockfall targets in real time and accurately in complex environments, significantly improving the accuracy and reliability of rockfall monitoring.

[0046] 2. Compared with traditional monitoring technologies such as sensors and radars, the present invention adopts computer vision and deep learning technology, can work effectively under various lighting conditions, complex backgrounds and environmental interference, has strong adaptability, can cover a larger range, has high real-time performance, and is suitable for rockfall monitoring in large areas and high and steep mountainous areas.

[0047] 3. Compared with the high-cost traditional monitoring technology, the present invention realizes rockfall monitoring through image acquisition equipment and vision-based algorithms, which reduces the cost of the system while maintaining high accuracy and efficiency and has better popularization.

[0048] 4. The present invention can accurately predict the motion trajectory of falling rocks based on the kinematic model and the free-fall formula, calculate the position changes of falling rocks in real time in combination with image data, generate early warning information in a timely manner, and effectively prevent and reduce the safety risks brought about by falling rocks.

[0049] 5. By introducing intelligent algorithms such as dynamic learning rate, motion target classification, and trajectory prediction, the present invention realizes automatic detection and tracking of falling rocks, reduces manual intervention, improves the automation level of the system, and further improves monitoring efficiency and early warning response speed.

[0050] 6. The present invention can provide continuous and real-time rockfall monitoring in key areas such as construction sites or mountain roads, provide timely early warning information to relevant departments and personnel, enhance the ability to respond to sudden geological disasters, and ensure the safety of personnel and projects. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a block diagram of the module components of an embodiment of the present invention;

[0052] Figure 2 is a flow chart of a rockfall motion tracking method according to an embodiment of the present invention;

[0053] Figure 3 It is a functional diagram of the modules of the embodiment of the present invention;

[0054] Figure 4 Schematic diagram of a data set of an embodiment of the present invention; (a) is rockfall beside a road, (b) is rockfall on a mountainside, (c) is rockfall in a ditch, and (d) is rockfall at a rockfall accumulation site;

[0055] Figure 5 is a diagram of data analysis results of an embodiment of the present invention;

[0056] Figure 6 This is a graph showing the results of a data depth analysis of an embodiment of the present invention;

[0057] Figure 7 is a flow chart of obtaining the spatial coordinates of a rockfall according to an embodiment of the present invention;

[0058] Figure 8 It is a characteristic diagram of rockfall motion predicted by a rockfall trajectory model according to an embodiment of the present invention; (a) is a visualization result of the acquired rockfall motion speed characteristic, (b) is a visualization result of the acquired rockfall motion time characteristic, (c) is the acquired rockfall motion speed characteristic-motion distance, and (d) is the acquired rockfall motion time-motion distance. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples.

[0060] The present invention mainly includes: motion detection model, rockfall target detection model, rockfall motion tracking model, and rockfall motion trajectory calculation method. Figure 1 The monitoring process is as shown in Figure 2 As shown, the function is Figure 3 shown.

[0061] The present invention proposes a method for tracking rockfall motion based on computer vision, comprising the following steps:

[0062] (1) Motion detection model: The motion detection model of the present invention is based on an improved Progressed Gaussian Mixture Model (PGMM) and is used to monitor the rockfall motion in a video stream in real time. The model mainly includes the following steps:

[0063] a. Establishment of Gaussian mixture model. In the field of image processing, Gaussian mixture model (GMM) is a powerful background modeling tool. The model is based on the assumption that the intensity distribution of each pixel in the image can be described by the center position of multiple Gaussian distributions. By analyzing the statistical information of pixel samples, multiple Gaussian models can be assigned to each pixel, thereby enhancing the algorithm's ability to depict the background and accurately depict the image background, thereby effectively extracting moving targets.

[0064]

[0065] Where P is the mixed Gaussian model at point (x, y) at time t, and f (x,y),t represents the pixel intensity at point (x, y) at time t; K is the number of Gaussian distributions in MOG, usually 3-5; ω (x,y),t,n , μ (x,y),t,n ,∑ (x,y),t,n are the weight, mean and covariance matrix of the nth Gaussian distribution at point (x, y) corresponding to time t; η is the Gaussian probability density function; E is the identity matrix; Represents the variance of the nth Gaussian distribution at point (x, y) at time t; ∑ (x,y),t,n Represents the covariance matrix of the nth Gaussian distribution at point (x,y) at time t.

[0066] b. Separation of background and foreground.

[0067] As the image sequence is input, the Gaussian mixture model is adjusted to adapt to the new background changes. First, the current frame pixel intensity is matched with the set Gaussian mixture model. If the confidence parameter D is satisfied, it means the match is successful.

[0068] |f (x,y),t -μ (x,y),t,n |≤Dσ (x,y),t,n

[0069] Where: D is the confidence parameter, usually 2.5; σ (x,y),t,nshows the standard deviation of the nth Gaussian distribution at point (x,y) at time t.

[0070] c. Update the Gaussian distribution. Use the IIR filter to update the first successfully matched Gaussian distribution of a single pixel:

[0071] ω (x,y),t,n =(1-α)ω (x,y),t-1,n +α

[0072] μ (x,y),t,n =(1-β)μ (x,y),t-1,n +βf (x,y),t

[0073]

[0074] Where α and β are learning rates.

[0075] The rest of the Gaussian distribution of this pixel only has weight ω (x,y),t,n To modify:

[0076] ω (x,y),t,n =(1-α)ω (x,y),t-1,n

[0077] For a single pixel, if none of the K Gaussian distributions are matched successfully, a new Gaussian distribution is created (with a smaller initial weight value ω0 and a larger initial variance value σ0 to replace the Gaussian distribution with the smallest weight in the original distribution. Then all weights of the current pixel are normalized.

[0078] Then follow ω (x,y),t,n / σ (x,y),t,n The size of ω (x,y),t,n / σ (x,y),t,n The larger the value, the greater the weight of this Gaussian distribution, the closer it is to the center point, the more matches there are, and the more stable the model is. Then, the first B Gaussian distributions are selected from large to small as the description of the scene.

[0079]

[0080] Where T is the hyperparameter threshold.

[0081] However, for rockfall events, we found that the original model only updated the first matching Gaussian model after matching, which resulted in a slow convergence of the model. This led to a slow response to sudden events such as rockfall disasters. Therefore, in order to improve the detection capability of sudden rockfall events, we updated all matching Gaussian items.

[0082]

[0083] Where S is the standard deviation

[0084] d. Dynamic adjustment of learning rate. Since the model’s hyperparameter learning rate and fixed value are selected, the occurrence of rockfall events is not synchronized with the background update rate. Therefore, the design of parameter S is introduced to allow the learning rate to be dynamically adjusted within a wide range to match the convergence requirements of the model at different stages.

[0085]

[0086] Where δ is the initial hyperparameter, f is the current frame number, n is the set threshold frame number, and ΔN8 is the sum of the corresponding grayscale differences in the eight neighborhoods of the point (x, y) at time t and time t-1. new is the updated learning rate.

[0087] (2) Rockfall target detection model. The rockfall target detection model of the present invention is based on the YOLO (You Only LookOnce) algorithm. According to the characteristics of the rockfall disaster scene, the convergence speed and information correlation of the model are adaptively optimized. The processing methods used in the input part of the YOLOv5 network structure include adaptive anchor frame calculation, image preprocessing and Mosaic data enhancement. The backbone network of YOLOv5 includes modules such as slice structure and spatial pyramid pooling structure. The FPN+PAN structure is adopted in the neck network, and YOLOv5 adopts a cross-stage local structure. In the output stage, the generalized intersection-over-union loss (GIoULoss) is adopted as the loss function of the target bounding box regression, which shows good performance in the target detection task.

[0088] Some basic formulas of YOLOv5:

[0089] a. Loss function related formula:

[0090] Loss = λ coord L coord +λ conf L conf +λ class L class

[0091] Where λ coord , conf , class They are the weight coefficients corresponding to position loss, confidence loss, and classification loss, which are used to adjust the importance of each part of the loss in the total loss.

[0092]

[0093] Where S represents the number of grids into which the input image is divided, B represents the number of bounding boxes predicted for each grid, is an indicator function, whose value is 1 when the j-th prediction box of the i-th grid is responsible for the corresponding target (i.e., the prediction box has the highest match with the true target, etc.), otherwise it is 0.

[0094]

[0095] Where C i is the confidence of the prediction, is the true confidence (1 when it contains the target, 0 when it does not contain the target), λ noobj is the weight coefficient used to balance the confidence loss in the target and non-target cases, is the indicator function for the no-target situation (similar to opposite logic).

[0096]

[0097] In the formula is the probability of the true category c (usually in one-hot encoding form, the corresponding category is 1, and the rest are 0), is the predicted probability of class c.

[0098] b. Prediction box calculation related formula:

[0099]

[0100] Where t x ,t y is the eigenvalue of the corresponding coordinate output by the network, c x 、c y is the coordinate of the upper left corner of the grid cell in the feature map (c x 、c y The value range is 0 to S-1 integer), w img 、h img are the width and height of the input image respectively, and σ is the activation function used to normalize the output value to the range of 0 to 1.

[0101]

[0102] Where p w 、p h is the width and height of the prior box (pre-set reference boxes of different sizes), t w ,t h It is the characteristic value output by the network used to adjust the width and height of the prior box.

[0103] The technical solution of this model mainly includes the following steps:

[0104] a. Design and optimization of YOLO model.

[0105] The Yolo-Rock model is a mature application in the field of target detection, and it has the characteristics of fast and efficient detection. Compared with Mask R-CNN, Fast-R-CNN, SSD, etc., it not only has a significant advantage in detection speed, but also takes into account detection accuracy. This embodiment selects Yolo to detect rockfall targets, and adaptively optimizes the convergence speed and information correlation of the model based on the characteristics of rockfall disaster scenarios.

[0106] b. Addition of small target detection layer and network optimization.

[0107] Due to the small size of the rockfall targets in the image, the changeable shapes, and the complex and easily confused background, the accuracy is not high during the recognition process. And because the downsampling multiple of Yolo is relatively large, it is difficult for the deeper feature map to learn the feature information of small targets. Therefore, in order to better adapt to the rockfall target detection task, we add a small target detection layer to detect after splicing the shallower feature map with the deep feature map.

[0108] c. Adjustment of network structure

[0109] In order to achieve better detection results, we adjusted the depth and width of the network. Specifically, we increased the number of convolutional and residual components in the CSP structure.

[0110] (3) Rockfall motion tracking model - combined model

[0111] a. In the workflow of the joint model, PGMM is first used to detect moving objects in the image sequence (to detect whether any objects are moving).

[0112] b. Use the Yolo-Rock model to conduct a detailed analysis of these moving objects to accurately determine whether they are falling rocks and effectively distinguish them from other non-related objects such as pedestrians and flying birds.

[0113] c. Set the confidence thresholds for Intersection over Union (IoU) and Non-Maximum Suppression (NMS), and select confidence based on IoU and NMS to integrate the two model outputs.

[0114] (4) Calculation method of rockfall trajectory:

[0115] a. The mass concentration method is selected as the main model for solving the trajectory of falling rocks. The movement process of falling rocks is divided into several stages: first, free fall, followed by rolling and sliding, and finally, collision may be involved.

[0116] b. The free fall phase is accurately calculated using the classical parabolic motion formula.

[0117]

[0118] Where: is the initial velocity of free fall motion, which is usually 0 except for earthquake effects; x0, y0, z0 are the three-dimensional spatial coordinates of the starting position of the rolling stone; t is the motion time; g is the gravitational acceleration.

[0119] c. The sliding friction in the sliding stage is represented by the Coulomb dynamic friction model. Since its motion trajectory must satisfy Newton's second law, after substituting it into the equation and solving it, we get the equation representing the sliding of falling rocks.

[0120]

[0121] Where: k s is the sliding friction coefficient; T is the friction force (vector); N is the normal stress along the slope; V is the velocity of the block in the motion plane.

[0122]

[0123] Where: F is the total external load vector; m is the mass of the block; x is the spatial coordinate vector; t is the time.

[0124]

[0125] d. In the collision phase, before and after the rockfall collides with the ground, its incident velocity and exit velocity should satisfy the following two equations:

[0126]

[0127] Where: are the normal velocity and tangential velocity of the block before it collides on the slope; are the normal velocity, tangential velocity and actual velocity of the rolling stone after it collides with the slope; R n , R t are the normal restitution coefficient and the tangential restitution coefficient, respectively.

[0128] In this embodiment, a high-resolution camera was used to capture images with a spatial resolution of 1920×1080 pixels per frame, and the operating speed was set to 25 frames per second (fps). This configuration matches the specifications of cameras used for on-site video detection in actual projects. In order to simulate a real rockfall event, ten stones of different sizes were thrown from the top of the slope, with diameters ranging from 0.1 m to 1 m. In the field test, the camera was carefully set to record the entire process of each rockfall in detail. The duration of each rockfall event was approximately between 1 second and 15 seconds. In the test, the ratio of the plane projection area of ​​the stone to the size of the study area was changed from 1m 2 :416m2 Change to 1m 2 : 550m 2 That is, within a range of about 400 to 500 square meters, this method can effectively detect the plane projection area of ​​about 1m 2 It is expected that by testing rocks of different sizes, it can be verified that this method can meet the high-resolution requirements and is also applicable to small-target rockfall detection in larger areas. This verification process is crucial to improving the applicability and effectiveness of monitoring technology.

[0129] In addition, in order to meet the needs of deep learning model training, a comprehensive rockfall dataset was constructed by carefully annotating rockfall images in different environments. The dataset contains 543 images, covering 9775 accurately annotated targets. These data were collected under a variety of light source environments, different shooting equipment and lens conditions, covering the distribution of rockfall on natural slopes and material yard slopes, ensuring the breadth and representativeness of the dataset. In this way, the dataset not only provides rich training materials for deep learning models, but also enhances the generalization and adaptability of the model by simulating the changeable conditions in the real world. This combination of diversity and quality provides a solid foundation for research to develop advanced algorithms that can accurately identify and predict rockfall motion. Figure 4 shown.

[0130] In addition, this embodiment also uses drone technology to establish a high-precision terrain model of the test site, providing accurate terrain data for later data supplementation. In the engineering practice of high and steep slopes, the establishment of terrain models is a standard operation, which provides key spatial information for engineering planning and implementation. Therefore, the application of drone technology to the collection of terrain data is not only feasible in actual engineering, but also these data are usually acquired and analyzed at the beginning of the project. In this way, it can be ensured that the most accurate and up-to-date terrain information is available when designing the rockfall monitoring and early warning system, which is crucial to improving the accuracy and efficiency of the monitoring system. The application of drone technology not only improves the efficiency of data collection, but also provides strong support for engineering decision-making.

[0131] Based on the detailed analysis of the field test video using the Rockfall Movement Tracking Model (RMTM), a most representative video was selected for demonstration. The falling rocks in the video became the focus of analysis due to their small size and fast movement speed. The analysis effects of different models on this video are demonstrated. The video records a tiny falling rock target, which moves at an extremely high speed and causes disasters. The entire movement process lasted about 4 seconds. The falling rock began to fall from the 65th frame and ended at the 160th frame. The analysis results show that the improved PGMM model exhibits excellent motion tracking capabilities, and the model has good convergence performance and can update the background model in time to adapt to fast-moving objects. However, it should be noted that factors such as the movement of personnel (frame 65), shadow changes caused by the movement of falling rocks (frame 125), and smoke generated by the collision (frame 137) have caused certain interference to the accuracy of motion recognition. Figure 5 As shown. For the Yolo-Rock model, although it can successfully identify all potential rockfall targets, it cannot distinguish the motion state of the rockfall and the number of identified targets is too large, which limits its application in the rockfall motion tracking task. Despite this, the Yolo-Rock model still successfully captured the movement of small rockfalls, which proves that the improvement of the Yolo model by the present invention is effective. Finally, by using the RMTM model, accurate tracking of falling rocks in motion is achieved. From the 65th frame to the 160th frame of the video, the RMTM model always uniquely and accurately identifies the rockfall targets, which further verifies the effectiveness and reliability of the model of the present invention in practical applications, as shown Figure 6 shown.

[0132] In addition, the present invention also expands the scope of analysis and conducts in-depth analysis of other data sets. The RMTM model can uniquely and accurately identify the moving rocks in these videos. These rockfall samples cover different sizes and shapes, further demonstrating the wide applicability and high accuracy of the RMTM model in identifying moving rocks. Through these analyses, the robustness and reliability of the RMTM model in handling diverse rockfall motion detection tasks are verified.

[0133] After identifying the trajectory of the rockfall, the spatial coordinates of the rockfall identified by RMTM are obtained through simple feature point matching through the model front view of the rockfall location. After RMTM identifies the rockfall, the center point of the candidate frame for identifying the rockfall is used as the reference for obtaining the spatial coordinates of the rockfall. Specifically, the center point of the candidate frame of the rockfall identified by the RMTM model is used as the reference for obtaining its spatial coordinates. Through image feature matching technology, matching points at corresponding positions in the model can be found under a unified perspective. Since the monitoring equipment is fixedly installed, this perspective is therefore constant. In the model, the coordinate points corresponding to these perspectives are pre-defined. In this way, the spatial coordinates of the monitored rockfall can be continuously obtained through a simple image matching process. Such as Figure 7 Through this method, the present invention can not only track the movement trajectory of the falling rocks, but also accurately locate the spatial position of the falling rocks, providing more accurate data support for the assessment and prevention of falling rock disasters.

[0134] Once the spatial coordinates of the rockfall are obtained, its possible trajectory can be calculated using the rockfall trajectory model. Figure 8 As shown. Analysis shows that the data obtained from the rockfall trajectory simulation model shows that the maximum movement speed of the rockfall is about 7 meters per second, and the time from the start of movement to the final departure from the monitoring field of view is 4 seconds. At the same time, the data obtained by manually analyzing the video frame by frame shows that the maximum speed of the rockfall is 6.7 meters per second, and the number of frames from movement to leaving the monitoring field of view is 95 frames, which is calculated at a rate of 25 frames per second, and the time is 3.8 seconds. The results of the manual analysis are highly consistent with the results of trajectory prediction using the RMTM model, which verifies the accuracy and feasibility of prediction using the trajectory model, and the error range is acceptable. This comparison further confirms the reliability of the trajectory prediction model of the present invention in practical applications, and provides an effective technical means for the prediction and analysis of rockfall motion.

[0135] In another embodiment of the present invention, a rockfall motion tracking system based on computer vision is provided. The system can be used to implement the above-mentioned rockfall motion tracking method, and specifically comprises: a module

[0136] Video monitoring module: The video stream of the monitored area is obtained in real time through the camera as the input data source.

[0137] Moving target detection module: Based on the improved Gaussian mixture model PGMM, the background of the video stream is modeled and moving targets are detected.

[0138] Target classification and recognition module: Use the YOLOv5 model to classify the detected moving targets and determine whether they are falling rocks.

[0139] Rockfall precision identification module: Combines the outputs of the PGMM and YOLOv5 models to screen and accurately identify targets to confirm whether they are rockfall targets.

[0140] Rockfall trajectory prediction module: Calculate and predict the trajectory of rockfall based on the rockfall motion model and free fall formula.

[0141] Result output and alarm module: outputs the motion trajectory and predicted position of falling rocks according to the prediction results, and generates alarm information.

[0142] In another embodiment of the present invention, a terminal device is provided, the terminal device includes a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions to implement the corresponding method flow or corresponding functions; the processor described in the embodiment of the present invention can be used for the operation of the rockfall motion tracking method.

[0143] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a terminal device for storing programs and data. It can be understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and the extended storage medium supported by the terminal device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage.

[0144] One or more instructions stored in a computer-readable storage medium can be loaded and executed by a processor to implement the corresponding steps of the rockfall motion tracking method in the above-mentioned embodiment; one or more instructions in the computer-readable storage medium are loaded and executed by the processor.

[0145] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0149] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the implementation methods of the present invention, and should be understood that the protection scope of the present invention is not limited to such special statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A rockfall motion tracking method based on computer vision, characterized in that: The following steps are involved: S1: Use improved Gaussian mixture model PGMM for real-time video stream monitoring and moving target detection; S2: Classify the detected moving targets based on the YOLOv5 model and identify whether they are falling rocks; S3: Combine the outputs of PGMM and YOLOv5 models to screen and accurately identify rockfall targets; S4: Calculate and predict the trajectory of falling rocks based on the rockfall motion model and free fall formula; S5: Based on the calculation result of step S4, the movement trajectory and predicted position of the falling rock are output, and the alarm information is generated and displayed.

2. The rockfall motion tracking method according to claim 1, characterized in that: The PGMM model includes: For each pixel, if multiple Gaussian distributions are matched successfully, all matching Gaussian distributions are updated instead of just the first matching Gaussian distribution. In the background update process, a dynamic learning rate is introduced to adapt to background changes at different stages by adjusting the learning rate, and the background update rate is dynamically controlled to improve the detection response speed of rockfall events. When multiple Gaussian distributions fail to match successfully, a new Gaussian distribution is created, and the weight of each Gaussian distribution is adjusted by normalization to ensure the stability and accuracy of background modeling.

3. The rockfall motion tracking method according to claim 2, characterized in that: The specific steps of S1 are as follows: Construct a Gaussian mixture model to describe the image background. The intensity distribution of each pixel in the image is modeled through multiple Gaussian distributions, and statistical information is used for modeling to accurately depict the background and extract moving targets. For the background of each pixel, use Gaussian distribution to model the background, separate it according to the matching degree between the pixel intensity and Gaussian distribution, and update the background model; Dynamically adjust the learning rate and dynamically adjust the update rate of the model by introducing the standard deviation parameter S: improve the response capability to sudden rockfall events; Through the matching process, the Gaussian distribution weights are updated.

4. The rockfall motion tracking method according to claim 1, characterized in that: The specific steps of S2 are as follows: Use the YOLOv5 network structure for target detection, including adaptive anchor box calculation, image preprocessing, and Mosaic data enhancement; For detected targets, the FPN+PAN structure in the YOLOv5 network and the generalized intersection-over-union loss are used to perform target bounding box regression for precise positioning and classification. In the optimization process of YOLOv5, a small target detection layer is added and the depth and width of the network are adjusted to adapt to the detection of small and irregularly shaped falling rock targets in complex backgrounds.

5. The rockfall motion tracking method according to claim 1, characterized in that: The specific steps of S3 are as follows: The location and category of the rockfall target are determined by combining the output of the PGMM and YOLOv5 models through the intersection-over-union and non-maximum suppression algorithms. According to the confidence threshold of the model, the final rockfall targets are screened out and further classified.

6. The rockfall motion tracking method according to claim 1, characterized in that: The specific steps of S4 are as follows: The mass point concentration method is used to divide the rockfall motion into several stages: free fall, rolling and sliding, and collision; In the free fall stage, the classic parabolic motion formula is used to calculate the trajectory of the falling rock; In the sliding stage, the Coulomb dynamic friction model is used to describe the sliding friction and calculate the sliding trajectory of the falling rock; During the collision phase, the velocity change of the rockfall colliding with the ground is taken into account, and the collision restitution coefficient is used to calculate the velocity and position changes before and after the collision.

7. A rockfall motion tracking system based on computer vision, characterized in that: The system can be used to implement the rockfall motion tracking method described in any one of claims 1 to 6, specifically comprising: Video monitoring module: obtains the video stream of the monitoring area in real time through the camera as the input data source; Moving target detection module: Based on the improved Gaussian mixture model PGMM, the background of the video stream is modeled and moving targets are detected; Target classification and recognition module: Use the YOLOv5 model to classify the detected moving targets and determine whether they are falling rocks; Rockfall precision identification module: Combines the output of PGMM and YOLOv5 models to screen and accurately identify targets to confirm whether they are rockfall targets; Rockfall trajectory prediction module: Calculate and predict the rockfall trajectory based on the rockfall motion model and free fall formula; Result output and alarm module: outputs the motion trajectory and predicted position of falling rocks according to the prediction results, and generates alarm information.

8. A computer device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the rockfall motion tracking method as claimed in one of claims 1 to 6 is implemented.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the rockfall motion tracking method described in one of claims 1 to 6 is implemented.

Images