Landslide detection method, device and equipment and storage medium

Abstract

The invention discloses a landslide detection method and device, equipment and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: carrying out the real-time photographing and obtaining a current frame image of a side slope region of a road side; based on the current frame image, detecting through a preset target classification model to obtain a potential landslide target, realizing preliminary landslide target classification, and tracking the trajectory of the potential landslide target through a preset target tracking model to obtain the motion trajectory of the potential landslide target; and on the basis of the motion track, whether a landslide sign exists in the side slope area of the road side is judged, so that the landslide sign is found in time, and the problem of low reliability of landslide detection is solved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and particularly to a landslide detection method, device, equipment, and storage medium. Background Art

[0002] Traditional landslide detection for slopes on the side of a road is usually carried out by remote sensing detection (drone / satellite) for periodic shooting. However, the detection frequency of this method is relatively low. Since the occurrence of slope landslides is usually sudden and uncertain, the delayed information may lead to missing the best emergency response time; or it is detected by sensors (such as inclinometers, strain gauges, etc.). This method can only provide data of local points and it is difficult to comprehensively cover the entire slope area, affecting the accuracy of landslide detection.

[0003] Therefore, at present, it is impossible to achieve real-time and accurate landslide detection for slopes on the side of a road, that is, the reliability of landslide detection is relatively low. Summary of the Invention

[0004] The main purpose of this application is to provide a landslide detection method, device, equipment, and storage medium, aiming to solve the technical problem of relatively low reliability of landslide detection.

[0005] To achieve the above purpose, this application proposes a landslide detection method, and the method includes: Obtain the current frame image of the slope area on the side of the road captured; Based on the current frame image, perform detection through a preset target classification model to obtain potential landslide targets; Through a preset target tracking model, track the trajectory of the potential landslide target to obtain the movement trajectory of the potential landslide target; Based on the movement trajectory, determine whether there are landslide signs in the slope area on the side of the road.

[0006] In an embodiment, the step of performing detection through a preset target classification model based on the current frame image to obtain potential landslide targets includes: Separate the dynamic targets and static targets in the current frame image through a preset background separation model to obtain a foreground image containing dynamic targets; Calculate the candidate moving targets based on a preset target recognition algorithm for the foreground image; Classify the candidate moving targets through a preset target classification model to obtain potential landslide targets.

[0007] In one embodiment, the preset target classification model includes a first target classification sub-model and a second target classification sub-model. The step of classifying the candidate moving target through the preset target classification model to obtain potential landslide targets includes: Classify the candidate moving target through the first target classification sub-model, screen out the interference targets among the candidate moving targets, and obtain candidate landslide targets; Based on the area of the candidate landslide target and a preset area threshold, determine whether the candidate landslide target has the possibility of landslide; If it has, identify the motion characteristics of the candidate landslide target through the second target classification sub-model; If the motion characteristics are landslide characteristics, determine that the candidate landslide target is a potential landslide target.

[0008] In one embodiment, the step of based on the area of the candidate landslide target and a preset area threshold to determine whether the candidate landslide target has the possibility of landslide includes: Judge whether the area of the candidate landslide target is greater than or equal to the preset area threshold; If the area of the candidate landslide target is less than the preset area threshold, determine that the candidate landslide target does not have the possibility of landslide; After the step of based on the area of the candidate landslide target and a preset area threshold to determine whether the candidate landslide target has the possibility of landslide, it further includes: If it does not have, judge whether the number of candidate landslide targets with an area less than the preset area threshold is greater than the preset number threshold; If it is greater, determine whether the area with the most candidate landslide targets with an area less than the preset area threshold has landslide characteristics through the second target classification sub-model.

[0009] In one embodiment, the preset target recognition algorithm includes multi-frame difference method and dense optical flow algorithm. The step of calculating the candidate moving target based on the preset target recognition algorithm for the foreground image includes: Based on the multi-frame difference method, compare the differences between the foreground images including the current frame and its first preset consecutive previous frames, and determine the initial motion area based on the differences; Calculate the best motion vector of each pixel in the initial motion area based on the dense optical flow algorithm; Based on the best motion vector, preset motion speed threshold and preset motion direction, determine the candidate moving target.

[0010] In one embodiment, the step of tracking the trajectory of the potential landslide target through the preset target tracking model to obtain the motion trajectory of the potential landslide target includes: By means of a preset target tracking model, track the potential landslide targets in the foreground image including the current frame and its second consecutive previous preset frame to obtain the movement trajectories of the potential landslide targets; The step of judging whether there are landslide signs in the slope area on the roadside based on the movement trajectory includes: If the movement trajectory is downward and the area change of the potential landslide target increases with time, it is determined that there are landslide signs in the slope area on the roadside; After the step of judging whether there are landslide signs in the slope area on the roadside based on the movement trajectory, it further includes: If there are landslide signs, issue a warning based on the movement trajectory.

[0011] In an embodiment, the step of separating the dynamic targets and static targets in the current frame image by means of a preset background separation model to obtain a foreground image containing dynamic targets includes: Separate the dynamic targets and static targets in the current frame and its third consecutive previous preset frame images by means of a preset background separation model to obtain a foreground image containing dynamic targets; After the step of separating the dynamic targets and static targets in the current frame image by means of a preset background separation model to obtain a foreground image containing dynamic targets, it further includes: Update the model parameters of the preset background separation model to separate the dynamic targets and static targets in the next frame image and its previous preset frame images based on the updated background separation model.

[0012] In addition, to achieve the above object, the present application also proposes a landslide detection device, which includes: An acquisition module, configured to acquire the current frame image of the slope area on the roadside captured; A detection module, configured to perform detection based on the current frame image through a preset target classification model to obtain potential landslide targets; A tracking module, configured to track the trajectories of the potential landslide targets by means of a preset target tracking model to obtain the movement trajectories of the potential landslide targets; A judgment module, configured to judge whether there are landslide signs in the slope area on the roadside based on the movement trajectory.

[0013] In addition, to achieve the above object, the present application also proposes a landslide detection device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the landslide detection method as described above.

[0014] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the landslide detection method described above are implemented.

[0015] In addition, to achieve the above object, the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the landslide detection method described above are implemented.

[0016] One or more technical solutions proposed by the present application have at least the following technical effects: The present application obtains the current frame image of the slope area on the roadside by real-time shooting; based on the current frame image, it is detected by a preset target classification model to obtain potential landslide targets, realizing preliminary landslide target classification, so as to be able to more specifically detect whether the potential landslide targets have changed. Specifically, a preset target tracking model can be used to track the trajectories of the potential landslide targets to obtain the movement trajectories of the potential landslide targets, realizing dynamic tracking of the movement trajectories of the potential landslide targets; based on the movement trajectories, it is judged whether there are landslide signs in the slope area on the roadside. By dynamically tracking the potential landslide targets in real time, the obtained movement trajectories can reflect whether there are overall landslide signs in the slope area on the roadside.

[0017] It can be understood that the present application uses target classification technology to detect potential landslide targets in real time and uses target tracking technology to analyze the dynamic changes in the slope area, so as to timely discover landslide signs and improve the reliability of landslide detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings here are incorporated into the specification and constitute a part of this specification, showing the embodiments of the present application and used together with the specification to explain the principles of the present application.

[0019] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

[0020] Figure 1 It is a schematic flow chart provided for the first embodiment of the landslide detection method of the present application; Figure 2 It is a schematic scenario diagram provided for the first embodiment of the landslide detection method of the present application; Figure 3It is a schematic flowchart of the landslide detection method provided in the second embodiment of this application; Figure 4 It is a flowchart of the landslide detection method provided in the second embodiment of this application; Figure 5 It is a schematic diagram of the module structure of the landslide detection device according to the embodiment of this application; Figure 6 It is a schematic diagram of the device structure of the hardware operating environment involved in the landslide detection method according to the embodiment of this application.

[0021] The realization of the purpose, functional characteristics and advantages of this application will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0022] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not used to limit this application.

[0023] In order to better understand the technical solutions of this application, the following will be described in detail in combination with the drawings of the specification and specific implementation manners.

[0024] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, a landslide detection device, etc. that can implement the above functions. The following takes the landslide detection device as an example to illustrate this embodiment and the following embodiments.

[0025] Based on this, the embodiment of this application provides a landslide detection method, referring to Figure 1 , Figure 1 It is a flowchart of the first embodiment of the landslide detection method of this application.

[0026] In this embodiment, the landslide detection method includes steps S10 to S40: Step S10, obtaining the current frame image of the slope area on the side of the road captured; It should be noted that traditional landslide detection for slopes on the roadside is usually carried out by remote sensing detection (drone / satellite) with periodic shooting (for example, once a day or once a week). However, this method has a low detection frequency. At the same time, this method is greatly affected by environmental factors such as weather and light, and cannot work effectively at night or in bad weather. Moreover, due to the suddenness and uncertainty of slope landslides, the delayed information may lead to missing the best emergency response time (unable to capture sudden landslides in real time); traditional landslide detection for slopes on the roadside also includes detection by physical sensors (such as inclinometers, strain gauges, etc.). This method can only provide data of local points, with high deployment and maintenance costs, and it is difficult to comprehensively cover the entire slope area. Moreover, this method can only monitor specific points and cannot sense the landslide state of the entire slope area, affecting the accuracy of landslide detection.

[0027] Therefore, at present, it is impossible to achieve real-time and accurate landslide detection for slopes on the roadside, that is, the reliability of landslide detection is relatively low.

[0028] To solve the above problems, in this embodiment, monitoring cameras are deployed in the slope area on the roadside according to a preset density, without the need to deploy complex sensors. Refer to Figure 2 Specifically, the deployment density of cameras can be increased in areas with complex terrain (such as areas with steep slopes, fragile rock formations, etc.) (for example, one camera is deployed every 50 meters); in flat terrain areas, the deployment density can be reduced, for example, one camera is deployed every 100 meters, balancing costs and monitoring requirements.

[0029] The minimum configuration of the camera can be 1080P resolution, 30fps frame rate, and support for wide dynamic range (WDR) to adapt to backlight scenarios; in addition, an infrared function can be equipped to achieve night monitoring.

[0030] Based on the above method, images or videos of the slope area on the roadside can be obtained. Refer to Figure 3 Specifically, in order to detect in real time whether there is a landslide in the slope area, this embodiment can obtain the current frame image of the slope area on the roadside in real time, so as to facilitate subsequent processing of the current frame image and improve the real-time performance of detection.

[0031] Step S20, based on the current frame image, perform detection through a preset target classification model to obtain potential landslide targets; It should be noted that before processing the current frame image, image stabilization processing can be performed first; refer to Figure 3, specifically, the real-time anti-shake technology based on feature point matching can be used to extract the SIFT feature points (Scale-invariant feature transform, local areas in the current frame image) of the current frame image and the corresponding previous frame image of the current frame respectively. Based on multiple pairs of the best matching points, an affine transformation matrix (used to describe the relative motion between two frames, including translation, rotation, and scaling, etc.) is calculated.

[0032] Then, based on the affine transformation matrix, the jitter situation of the image corresponding to the next frame of the current frame is predicted through a Kalman filter; according to the jitter situation of the next frame image, reverse displacement adjustment is performed on the current frame image to offset the expected jitter; thereby reducing the false detection caused by the mechanical vibration of the camera, and at the same time, the processing delay is less than 10 ms, that is, the calculation and compensation process can be completed in an extremely short time, ensuring that the output video stream is as smooth as possible and without obvious delay.

[0033] Furthermore, based on the current frame image, detection is performed through a preset target classification model, that is, each target in the image is classified and recognized, so as to identify potential landslide targets.

[0034] Specifically, the preset target classification model can be trained based on a convolutional neural network model, or a Support Vector Machine (SVM), etc.; potential landslide targets can be newly emerged surface cracks, displaced soil bodies, stones, or accumulated sediments, etc.

[0035] Step S30, through a preset target tracking model, track the trajectory of the potential landslide target to obtain the movement trajectory of the potential landslide target; Since detection through the preset target classification model can only achieve preliminary landslide target classification, but cannot indicate that a landslide is currently occurring, in this embodiment, through the preset target tracking model, the trajectory of the potential landslide target is tracked, that is, it is specifically detected whether the potential landslide target has changed, so as to realize the dynamic analysis of the movement state of the landslide target.

[0036] Among them, the preset target tracking model can be trained based on a convolutional neural network, or can be a model based on the optical flow method; the movement trajectory of the potential landslide target can be an irregular downward movement trajectory, a straight downward movement trajectory, stationary, or a displacement trajectory to the left and right sides, etc.

[0037] Step S40, based on the movement trajectory, judge whether there are landslide signs in the slope area on the side of the road.

[0038] It can be understood that based on the movement trajectory of the potential landslide target analyzed in real time, it can be determined whether a landslide is about to occur, that is, it can be determined whether there are landslide signs in the slope area on the side of the road.

[0039] Further, after the step of determining whether there are landslide signs in the slope area on the side of the road based on the movement trajectory, the following steps are further included: If there are landslide signs, a warning is issued based on the movement trajectory.

[0040] Specifically, once it is confirmed that there are landslide signs, a warning can be issued based on the movement trajectory so as to take timely measures to reduce possible risks. Issuing a warning based on the movement trajectory can be based on the location of the landslide or closing the road according to the movement trajectory, etc.

[0041] Among them, the warning can be divided into multiple levels of warnings. For example, potential risks (primary warning), confirmed alarms (secondary warning), etc.

[0042] In this embodiment, by deploying cameras to obtain real-time images of the slope area and using the cooperation of multi-level image processing and deep learning models, a full-automatic closed-loop from data collection to warning is realized. It solves the problems of low monitoring frequency and low detection real-time performance existing at present, provides a real-time landslide detection method based on pure algorithms, reduces the hardware cost and improves the warning timeliness.

[0043] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar content as that in the above-mentioned first embodiment can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 4 and step S20 includes steps S01 to S03: Step S01, separating the dynamic target and the static target in the current frame image through a preset background separation model to obtain a foreground image containing the dynamic target; In order to reduce the calculation amount and improve the landslide detection speed: separating the dynamic target and the static target in the current frame image through a preset background separation model to obtain a foreground image containing the dynamic target; in the subsequent processing process, only the part containing dynamic information (that is, the foreground image) needs to be processed; at the same time, removing the part of static information helps to remove unnecessary background interference, making the landslide target recognition for the foreground image more accurate.

[0044] For example, in the monitoring of the slope area on the side of the road, the background may include static elements such as vegetation and buildings, while the soil movement caused by the landslide is the dynamic target that needs more attention; effective background separation can ensure that the dynamic target will not be ignored or misjudged as part of the background in the subsequent landslide target recognition process; it can also ensure that the static target will not be misjudged as a landslide target, such as the floating of vegetation driven by the wind.

[0045] Among them, the dynamic targets include soil, rocks, people, vehicles, animals, etc., and the static targets include trees, bushes, etc.

[0046] Specifically, the implementation manner of separating the dynamic targets and static targets in the current frame image through the preset background separation model to obtain a foreground image containing the dynamic targets may be: Separate the dynamic targets and static targets in the current frame and its third consecutive previous preset frame images through the preset background separation model to obtain a foreground image containing the dynamic targets.

[0047] Among them, the preset background separation model can be a Gaussian mixture model (GMM) or a frame difference method. By analyzing consecutive multiple frame images through the Gaussian mixture model, it is possible to more accurately distinguish and separate static targets and dynamic targets.

[0048] Further, after the step of separating the dynamic targets and static targets in the current frame image through the preset background separation model to obtain a foreground image containing the dynamic targets, the model parameters of the preset background separation model can also be updated to separate the dynamic targets and static targets in the next frame image and its previous preset frame images based on the updated background separation model.

[0049] Specifically, since models such as the Gaussian mixture model analyze consecutive multiple frame images based on the color distribution of pixel points, gradual illumination changes (such as sunrise and sunset) will affect the analysis accuracy of the pixel points of each frame image by the Gaussian mixture model during background separation. Therefore, in this embodiment, after the step of separating the dynamic targets and static targets in the current frame image through the preset background separation model to obtain a foreground image containing the dynamic targets, the model parameters of the preset background separation model are updated (for example, adjusting the Gaussian distribution of the Gaussian mixture model or adjusting the difference threshold in the frame difference method), so as to separate the dynamic targets and static targets in the next frame image and its previous preset frame images based on the updated background separation model, and the deviation caused by gradual illumination changes can be adjusted.

[0050] Due to gradual illumination changes, the influence on pixel points between two consecutive frames or five consecutive frames is relatively small. In this embodiment, after the step of separating the dynamic targets and static targets in the current frame image through the preset background separation model to obtain a foreground image containing the dynamic targets, the model parameters of the preset background separation model can be updated every preset frame.

[0051] Among them, considering that the weather is different and the speed of light change is different, a preset number of frames required for each update of the model parameters of the preset background separation model can be determined based on the current weather.

[0052] S02. Calculate the foreground image based on a preset target recognition algorithm to obtain candidate moving targets. Furthermore, the foreground image only includes targets that may move (i.e., targets with a moving attribute). In order to be able to detect landslides in real time, it is possible to further detect whether there are targets that are currently moving in the current frame image.

[0053] Specifically, the preset target recognition algorithm includes a multi-frame difference method and a dense optical flow algorithm. The specific implementation of calculating the candidate moving targets by calculating the foreground image based on the preset target recognition algorithm can be: Compare the differences between the foreground images including the current frame and its first preset consecutive previous frames based on the multi-frame difference method, and determine the initial motion area based on the differences; calculate the best motion vector of each pixel in the initial motion area based on the dense optical flow algorithm; determine the candidate moving targets based on the best motion vector, a preset motion speed threshold, and a preset motion direction.

[0054] Specifically, the multi-frame difference method is used to detect moving objects by comparing the differences between the current frame and its first preset consecutive previous frames (such as 3 consecutive frames or 5 consecutive frames) of images; and calculate the differences in pixel values between each pair of adjacent frames; if the brightness of a certain pixel has changed significantly in consecutive frames, then this pixel may belong to a moving object, mark the pixel points where the change occurs, and form a preliminary motion area (i.e., determine the initial motion area based on the differences).

[0055] Based on the dense optical flow algorithm, calculate the best motion vector of the pixel points in the initial motion area, which contains information about the speed and direction of how the pixels move from one frame to the next; set a reasonable speed range and direction as thresholds to filter out those moving targets with too slow speeds (which may be noise or minor changes that are not of concern) and deviated motion directions; obtain the coordinates and motion parameters (speed, direction, area, etc.) of the candidate moving targets.

[0056] For example, screen moving targets with a speed greater than 0.15 m / s and a motion direction conforming to the gravity trend (deflected downward by more than 30°).

[0057] It should be noted that before calculating the foreground image based on the preset target recognition algorithm to obtain candidate moving targets, noise suppression and enhanced target feature processing can also be performed. Specifically, it can work by replacing each pixel value in the foreground image with the median value of its neighboring pixel values. This can effectively eliminate small-sized discrete noise points such as rain, snow, and flying insects without affecting important information such as image edges. Further, mark those short-term moving areas with a duration of less than 0.5 seconds and regard them as noise to be eliminated. For example, the changes caused by the slight shaking of plants due to a gentle breeze usually do not last long, so these interference factors can be identified and removed in this way.

[0058] Further, enhance the local contrast of the image through the Contrast Limited Adaptive Histogram Equalization (CLAHE) algorithm to highlight the texture features of targets such as soil and stones, referring to Figure 3 .

[0059] Step S03, classify the candidate moving targets through a preset target classification model to obtain potential landslide targets.

[0060] Further, since the candidate moving targets may still contain some non-landslide-related dynamic targets (such as pedestrians, vehicles, leaves blown by the wind, etc.); the foreground image can be calculated based on the preset target recognition algorithm to obtain candidate moving targets, thereby removing noise and irrelevant areas, screening out candidate moving targets that may be landslide targets, and reducing the complexity of subsequent processing.

[0061] Among them, the preset target classification algorithm can be a lightweight MobileNetV3 network, which is used to quickly identify and distinguish "natural interference objects (interference objects such as birds, branches, pedestrians, and vehicles)" from "potential landslide targets", referring to Figure 3 .

[0062] In this embodiment, through the above method, interference objects can be accurately distinguished, and potential landslide targets can be identified, contributing to the accuracy of subsequent landslide detection.

[0063] Based on the first and second embodiments of the present application, in the third embodiment of the present application, the same or similar content as in the above-mentioned first embodiment can be referred to the above introduction and will not be repeated hereinafter. On this basis, in a feasible implementation manner, the preset target classification model includes a first target classification sub-model and a second target classification sub-model; the specific implementation manner of classifying the candidate moving targets through the preset target classification model to obtain potential landslide targets can be: Classify the candidate moving target through the first target classification sub-model, screen out the interfering targets among the candidate moving targets, and obtain candidate landslide targets; based on the area of the candidate landslide targets and a preset area threshold, determine whether the candidate landslide targets have the possibility of a landslide; if so, identify the motion characteristics of the candidate landslide targets through the second target classification sub-model; if the motion characteristics are landslide characteristics, determine that the candidate landslide targets are potential landslide targets.

[0064] Specifically, the candidate moving target can be classified through the first target classification sub-model to distinguish the interfering targets (such as pedestrians, vehicles, animals, etc.) among the candidate moving targets and the candidate landslide targets that may be related to the landslide.

[0065] Furthermore, landslides usually involve large-scale soil movement, so the area is an important screening condition; if the area of a certain candidate landslide target is too small (such as a single pedestrian or a small object), the possibility of it being a landslide can be excluded. Therefore, based on the area of the candidate landslide targets and a preset area threshold, it can be determined whether the candidate landslide targets have the possibility of a landslide, referring to Figure 3 .

[0066] Among them, the area of the candidate landslide target can be calculated based on the dense optical flow algorithm, or can be calculated through the target detection box identified by the first target classification sub-model; the preset area threshold can be greater than 500 or 1000 pixels, etc.

[0067] In a feasible implementation manner, the determination of whether the candidate landslide target has the possibility of a landslide based on the area of the candidate landslide target and a preset area threshold can also be: Determine whether the area of the candidate landslide target is greater than or equal to the preset area threshold; if the area of the candidate landslide target is less than the preset area threshold, determine that the candidate landslide target does not have the possibility of a landslide.

[0068] Specifically, if the area of the candidate landslide target is less than the preset area threshold, it is considered that it cannot be a landslide target and is excluded; otherwise, the motion characteristics of the candidate landslide target are further identified through the second target classification sub-model.

[0069] Among them, the second target classification sub-model can be a ResNet101 V2 network, which is used to identify the motion features of candidate landslide targets. Since motion features usually contain multi-level information, such as texture changes, shape changes, displacement patterns, etc.; these features may be distributed at different abstraction levels: low-level features: such as edges, corners, color changes, etc.; middle-level features: such as local shape and texture changes; and high-level features: such as overall motion trajectory consistency, direction regularity, etc. The deep structure of ResNet101 V2 can gradually extract features from low-level to high-level, capturing richer motion information, thereby providing comprehensive input for subsequent classification.

[0070] Specifically, based on superimposing virtual interference objects (such as 3D bird models) on real slope videos, combined with a negative sample library (containing 20 common interference scenarios (animal activities, vehicles, etc.)), etc., the target classification model to be trained can be trained to obtain the above-mentioned preset target classification model, improving the generalization ability of the model.

[0071] In a feasible implementation manner, after the step of determining whether the candidate landslide target has the possibility of landslide based on the area of the candidate landslide target and the preset area threshold, the following steps are further included: If not, it is determined whether the number of candidate landslide targets with an area smaller than the preset area threshold is greater than the preset number threshold; if it is greater, it is determined whether the area with the largest number of candidate landslide targets with an area smaller than the preset area threshold has landslide characteristics through the second target classification sub-model.

[0072] It can be understood that since in the initial stage of a landslide, large-scale soil movement may not occur, but rather a series of small-scale phenomena such as cracks and soil loosening occur, which are manifested as multiple small-area targets in the image. Even if a single small-area change is not sufficient to be recognized as a landslide, but if they are concentrated in a specific area, this may be a precursor to a larger-scale landslide about to occur, referring to Figure 3 . Therefore, special attention needs to be paid to these small-area but densely distributed changes. Therefore, when the candidate landslide target does not have the possibility of landslide, in order to avoid missed detection, it can be further determined whether the number of candidate landslide targets with an area smaller than the preset area threshold is greater than the preset number threshold; Further, if the number of candidate landslide targets with an area smaller than the preset area threshold is greater than the preset number threshold, the second target classification sub-model is used to determine whether the area with the largest number of candidate landslide targets with an area smaller than the preset area threshold has landslide characteristics; it can be understood that once the area with a large number of small-area candidate landslide targets is determined, using the second target classification sub-model to conduct a detailed analysis of this area can help identify more subtle landslide characteristics (such as tiny displacements, progressive deformations, etc.). Such refined analysis helps to capture early signs of landslides and improve the timeliness and efficiency of detection. Refer to Figure 3 。

[0073] Meanwhile, in order to improve the accuracy of early warning, the conditions for the above first-level early warning can be that landslide characteristics are detected in a single frame and the area of the landslide target is greater than 1 square meter; the action for the first-level early warning can be to mark the area and start high-frequency sampling (increased to 60 times per second); the conditions for the second-level early warning can be that landslide characteristics are detected in 60 consecutive frames (with an interval of 1 second); the action for the second-level early warning can be to trigger an audible and visual alarm and push the coordinate information to the maintenance platform. Refer to Figure 3 。

[0074] The specific implementation manner of tracking the trajectory of the potential landslide target through the preset target tracking model to obtain the motion trajectory of the potential landslide target can be as follows: Through the preset target tracking model, the potential landslide target in the foreground image including the current frame and its second consecutive previous preset frames is tracked to obtain the motion trajectory of the potential landslide target.

[0075] In order to accurately analyze the motion trajectory of the potential landslide target, the potential landslide target in the foreground image including the current frame and its second consecutive previous preset frames (for example, the foreground images of the current frame and its 5 consecutive previous frames) can be tracked through the preset target tracking model, and based on the position changes of the tracked potential landslide target, the motion trajectory of the potential landslide target is calculated.

[0076] Among them, the preset target tracking model can be the SORT (Simple Online and Realtime Tracking) algorithm or the Deep SORT algorithm.

[0077] Specifically, according to the characteristics of the landslide phenomenon, the implementation manner of judging whether there are landslide signs in the slope area on the side of the road based on the motion trajectory can be: if the motion trajectory is downward and the area change of the potential landslide target increases with time, it is determined that there are landslide signs in the slope area on the side of the road.

[0078] If the movement trajectory is downward, it matches the slope gradient of the slope. If the area change of the potential landslide target increases with time, it reflects the process of soil and rock accumulation. When the above conditions are met simultaneously, it can be determined that there are landslide signs in the slope area on the side of the road.

[0079] Specifically, in the process of detecting landslide targets based on area or speed above, the preset area threshold and preset speed threshold can be dynamically adjusted based on time or weather, so as to avoid misjudgment caused by reduced visibility at night or increased object movement speed in rainy or snowy weather. Specifically, the preset area threshold can be automatically relaxed at night, or the preset speed threshold can be increased in rainy or snowy weather (to avoid misjudging raindrops).

[0080] In this embodiment, by combining multiple noise suppression and deep learning algorithms, the problem of false alarms caused by complex environmental interference is solved; a multi-level early warning mechanism is used to achieve risk grading response and avoid over-alarming.

[0081] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the landslide detection method of this application. Based on this technical concept, more simple transformations in various forms are within the protection scope of this application.

[0082] This application also provides a landslide detection device. Please refer to Figure 5 , the landslide detection device includes: An acquisition module 10, configured to acquire the current frame image of the slope area on the side of the road captured; A detection module 20, configured to detect based on the current frame image through a preset target classification model to obtain potential landslide targets; A tracking module 30, configured to track the trajectory of the potential landslide target through a preset target tracking model to obtain the movement trajectory of the potential landslide target; A judgment module 40, configured to judge whether there are landslide signs in the slope area on the side of the road based on the movement trajectory.

[0083] In one embodiment, the detection module 20 includes: A separation processing sub-module, configured to separate the dynamic target and the static target in the current frame image through a preset background separation model to obtain a foreground image containing the dynamic target; A calculation sub-module, configured to calculate the foreground image based on a preset target recognition algorithm to obtain candidate moving targets; A classification sub-module, configured to classify the candidate moving targets through a preset target classification model to obtain potential landslide targets.

[0084] In one embodiment, the preset target classification model includes a first target classification sub-model and a second target classification sub-model, and the classification sub-module includes: A classification unit, configured to classify the candidate moving target through the first target classification sub-model, screen out the interfering targets among the candidate moving targets, and obtain candidate landslide targets; A first judgment unit, configured to judge whether the candidate landslide target has the possibility of landslide based on the area of the candidate landslide target and a preset area threshold; A second judgment unit, configured to, if so, identify the motion characteristics of the candidate landslide target through the second target classification sub-model; A first determination unit, configured to, if the motion characteristics are landslide characteristics, determine the candidate landslide target as a potential landslide target.

[0085] In one embodiment, the first judgment unit includes: A judgment sub-unit, configured to judge whether the area of the candidate landslide target is greater than or equal to the preset area threshold; A determination sub-unit, configured to, if the area of the candidate landslide target is less than the preset area threshold, determine that the candidate landslide target does not have the possibility of landslide; Wherein, after the step of judging whether the candidate landslide target has the possibility of landslide based on the area of the candidate landslide target and the preset area threshold, the classification sub-module further includes: A third judgment unit, configured to, if not, judge whether the number of candidate landslide targets with an area less than the preset area threshold is greater than a preset number threshold; A second determination unit, configured to, if it is greater, determine whether the area with the largest number of candidate landslide targets with an area less than the preset area threshold has landslide characteristics through the second target classification sub-model.

[0086] In one embodiment, the preset target recognition algorithm includes a multi-frame difference method and a dense optical flow algorithm, and the calculation sub-module includes: A comparison unit, configured to compare the differences between the foreground images including the current frame and its first preset consecutive previous frames based on the multi-frame difference method, and determine an initial motion area based on the differences; A calculation unit, configured to calculate the optimal motion vector of each pixel in the initial motion area based on the dense optical flow algorithm; A third determination unit, configured to determine candidate moving targets based on the optimal motion vector, a preset motion speed threshold, and a preset motion direction.

[0087] In one embodiment, the tracking module 30 includes: A trajectory tracking sub-module, configured to perform trajectory tracking on potential landslide targets in a foreground image including the current frame and its second consecutive previous preset frames through a preset target tracking model, so as to obtain the movement trajectory of the potential landslide targets; Wherein, the determination module 40 includes: A determination sub-module, configured to determine that there are landslide signs in the slope area on the roadside if the movement trajectory is downward movement and the area change of the potential landslide targets increases with time; Wherein, after the step of determining whether there are landslide signs in the slope area on the roadside based on the movement trajectory, the landslide detection device further includes: An early warning module, configured to issue an early warning based on the movement trajectory if there are landslide signs.

[0088] In an embodiment, the separation processing sub-module includes: A separation processing unit, configured to perform separation processing on dynamic targets and static targets in the current frame and its third consecutive previous preset frame images through a preset background separation model, so as to obtain a foreground image including dynamic targets; Wherein, after the step of performing separation processing on dynamic targets and static targets in the current frame image through a preset background separation model to obtain a foreground image including dynamic targets, the separation processing sub-module further includes: An update unit, configured to update the model parameters of the preset background separation model, so as to perform separation processing on dynamic targets and static targets in the next frame image and its previous preset frame images based on the updated background separation model.

[0089] The landslide detection device provided by the present application adopts the landslide detection method in the above embodiment, and can solve the technical problem of low reliability of landslide detection. Compared with the prior art, the beneficial effects of the landslide detection device provided by the present application are the same as those of the landslide detection method provided by the above embodiment, and other technical features in the landslide detection device are the same as the features disclosed in the above embodiment method, which will not be elaborated herein.

[0090] The present application provides a landslide detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the landslide detection method in the first embodiment above.

[0091] Next, refer to Figure 6, which shows a schematic structural diagram of a landslide detection device suitable for implementing the embodiments of the present application. The landslide detection device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, tablet computers, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The shown landslide detection device is merely an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0092] As Figure 6 shown, the landslide detection device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which may perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 into the random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the landslide detection device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 may allow the landslide detection device to communicate with other devices wirelessly or wiredly to exchange data. Although the figure shows a landslide detection device with various systems, it should be understood that it is not required to implement or have all the shown systems. More or fewer systems may be alternatively implemented or had.

[0093] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product that includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by a processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are executed.

[0094] The landslide detection device provided by the present application adopts the landslide detection method in the above-mentioned embodiment, and can solve the technical problem of relatively low reliability of landslide detection. Compared with the prior art, the beneficial effects of the landslide detection device provided by the present application are the same as those of the landslide detection method provided by the above-mentioned embodiment, and other technical features in the landslide detection device are the same as the features disclosed in the method of the previous embodiment, and will not be elaborated here.

[0095] It should be understood that each part disclosed in the present application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0096] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0097] The present application provides a computer-readable storage medium with computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the landslide detection method in the above-mentioned embodiment.

[0098] The computer-readable storage medium provided by the present application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, device, or component. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0099] The above computer-readable storage medium can be included in the landslide detection device; or it can exist independently without being assembled into the landslide detection device.

[0100] The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by the landslide detection device, the landslide detection device is caused to: execute the above landslide detection method.

[0101] Computer program code for performing the operations of the present application can be written in one or more programming languages or combinations thereof. The above programming languages include object-oriented programming languages - such as Java, Smalltalk, C++; and also include conventional procedural programming languages - such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or can be connected to an external computer (e.g., by using an Internet service provider to connect through the Internet).

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0103] The modules described in the embodiments of the present application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation on the unit itself in some cases.

[0104] The readable storage medium provided by the present application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned landslide detection method, and can solve the technical problem of low reliability of landslide detection. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as those of the landslide detection method provided by the above embodiments, and will not be elaborated here.

[0105] The present application also provides a computer program product, including a computer program, and the steps of the above-mentioned landslide detection method are implemented when the computer program is executed by a processor.

[0106] The computer program product provided by the present application can solve the technical problem of low reliability of landslide detection. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the landslide detection method provided by the above embodiments, and will not be elaborated here.

[0107] The above are only some embodiments of the present application, and do not limit the patent scope of the present application. Any equivalent structural transformation made under the technical concept of the present application by using the content of the specification and drawings of the present application, or any direct / indirect application in other related technical fields, is included in the patent protection scope of the present application.

Claims

1. A landslide detection method, characterized in that, The method described includes: Obtain the current frame image of the slope area on the roadside captured; Based on the current frame image, perform detection through a preset target classification model to obtain potential landslide targets; Through a preset target tracking model, track the trajectory of the potential landslide target to obtain the movement trajectory of the potential landslide target; Based on the movement trajectory, determine whether there are landslide signs in the slope area on the roadside.

2. The method according to claim 1, characterized in that, The step of performing detection through a preset target classification model based on the current frame image to obtain potential landslide targets includes: Separate the dynamic targets and static targets in the current frame image through a preset background separation model to obtain a foreground image containing dynamic targets; Calculate the foreground image based on a preset target recognition algorithm to obtain candidate moving targets; Classify the candidate moving targets through a preset target classification model to obtain potential landslide targets.

3. The method according to claim 2, wherein The preset target classification model includes a first target classification sub-model and a second target classification sub-model. The step of classifying the candidate moving targets through the preset target classification model to obtain potential landslide targets includes: Classify the candidate moving targets through the first target classification sub-model, screen out the interfering targets among the candidate moving targets, and obtain candidate landslide targets; Based on the area of the candidate landslide target and a preset area threshold, determine whether the candidate landslide target has the possibility of landslide; If so, identify the movement characteristics of the candidate landslide target through the second target classification sub-model; If the movement characteristics are landslide characteristics, determine that the candidate landslide target is a potential landslide target.

4. The method according to claim 3, characterized in that, The step of determining whether the candidate landslide target has the possibility of landslide based on the area of the candidate landslide target and a preset area threshold includes: Determine whether the area of the candidate landslide target is greater than or equal to the preset area threshold; If the area of the candidate landslide target is less than the preset area threshold, determine that the candidate landslide target does not have the possibility of landslide; After the step of determining whether the candidate landslide target has the possibility of landslide based on the area of the candidate landslide target and a preset area threshold, it further includes: If not, determine whether the number of candidate landslide targets with an area less than the preset area threshold is greater than the preset number threshold; If it is greater, determine whether the area with the most candidate landslide targets with an area less than the preset area threshold has landslide characteristics through the second target classification sub-model.

5. The method according to claim 2, wherein The preset target recognition algorithm includes a multi-frame difference method and a dense optical flow algorithm. The step of calculating the foreground image based on the preset target recognition algorithm to obtain candidate moving targets includes: Based on the multi-frame difference method, compare the differences between the foreground images including the current frame and its first preset consecutive previous frames, and determine the initial movement area based on the differences; Calculate the optimal motion vector of each pixel in the initial movement area based on the dense optical flow algorithm; Based on the optimal motion vector, a preset motion speed threshold, and a preset motion direction, determine candidate moving targets.

6. The method according to claim 2, wherein The step of tracking the trajectory of the potential landslide target through a preset target tracking model to obtain the motion trajectory of the potential landslide target includes: Tracking the trajectory of the potential landslide target in the foreground image including the current frame and its second consecutive previous preset frames through a preset target tracking model to obtain the motion trajectory of the potential landslide target; The step of judging whether there is a landslide sign in the slope area on the side of the road based on the motion trajectory includes: If the motion trajectory is downward and the area change of the potential landslide target increases with time, it is determined that there is a landslide sign in the slope area on the side of the road; After the step of judging whether there is a landslide sign in the slope area on the side of the road based on the motion trajectory, it further includes: If there is a landslide sign, an early warning is issued based on the motion trajectory.

7. The method according to claim 2, characterized in that, The step of separating the dynamic target and the static target in the current frame image through a preset background separation model to obtain a foreground image containing the dynamic target includes: Separating the dynamic target and the static target in the current frame and its third consecutive previous preset frame images through a preset background separation model to obtain a foreground image containing the dynamic target; After the step of separating the dynamic target and the static target in the current frame image through a preset background separation model to obtain a foreground image containing the dynamic target, it further includes: Updating the model parameters of the preset background separation model to separate the dynamic target and the static target in the next frame image and its previous preset frame images based on the updated background separation model.

8. A landslide detection device, characterized in that The device includes: An acquisition module for acquiring the current frame image of the slope area on the side of the road captured; A detection module for detecting potential landslide targets based on the current frame image through a preset target classification model; A tracking module for tracking the trajectory of the potential landslide target through a preset target tracking model to obtain the motion trajectory of the potential landslide target; A judgment module for judging whether there is a landslide sign in the slope area on the side of the road based on the motion trajectory.

9. A landslide detection device, characterized in that, The device includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the landslide detection method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by the processor, it implements the steps of the landslide detection method according to any one of claims 1 to 7.

Images