Road geological disaster monitoring and early warning method integrating road segmentation and visual inspection
By integrating road segmentation and visual detection methods, and utilizing edge computing and cloud analytics, the SAM2 and YOLO algorithms for large visual models were optimized, solving the real-time and accuracy problems in highway geological disaster monitoring and achieving efficient monitoring and early warning.
Patent Information
- Application Number
- CN202510834483.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-20
- Publication Date
- 2025-11-11
AI Technical Summary
Existing technologies for monitoring geological disasters on highways suffer from poor real-time performance, low accuracy, and high false alarm rate. They are particularly difficult to identify small foreign objects and cannot meet the needs for intelligent monitoring that requires all-weather, multi-dimensional, and high precision.
A method integrating road segmentation and visual detection is adopted. An improved visual large model SAM2 is run on edge computing devices for real-time data processing. Combined with image processing algorithms and cloud server analysis, it can achieve road surface segmentation and foreign object detection. The recognition capability is improved by optimizing the SAM2 and YOLO models.
It enables real-time, high-precision, and integrated monitoring and early warning of highway geological disasters, improving emergency response capabilities and reducing traffic safety risks.
Smart Images
Figure CN120932418A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and intelligent transportation technology, and in particular to a method for monitoring and early warning of highway geological disasters that integrates road segmentation and visual detection. Background Technology
[0002] In recent years, geological disasters such as rockfalls, landslides, and collapses have frequently occurred in areas such as high slopes and tunnel entrances along highways, seriously threatening traffic safety. Because highway geological disasters endanger the lives and property of the people and maintain social stability, their monitoring requires extremely high standards.
[0003] Currently, mainstream detection methods mainly rely on manual inspections or traditional video detection algorithms, which suffer from poor real-time performance, low accuracy, and high false alarm rates. In particular, they lack the ability to identify small objects whose color is similar to the road surface, making it difficult to meet the needs of all-weather, multi-dimensional, high-precision, and intelligent disaster prevention and control. Therefore, there is an urgent need for a highway geological disaster monitoring and early warning method that integrates road segmentation and visual detection. This method would provide a real-time, high-precision, intelligent monitoring and early warning system for highway geological disasters such as landslides, rockfalls, mudslides, and road subsidence and collapse. It would achieve closed-loop management of the entire process from multi-dimensional data perception and AI intelligent analysis to early warning push, thereby improving the dynamic monitoring and early warning capabilities and emergency response capabilities for highway geological disasters. Summary of the Invention
[0004] The invention proposes a highway geological disaster monitoring and early warning analysis method that integrates road segmentation and visual detection. Combining edge computing, machine vision technology and video streaming technology, and through AI recognition and analysis technology, it can achieve integrated, all-weather, high-precision, and comprehensive identification and tracking monitoring of highway geological disasters and abnormal accidents such as debris spills, and promptly push out early warnings and block traffic to reduce road traffic safety risks.
[0005] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:
[0006] A method for monitoring and early warning of highway geological hazards that integrates road segmentation and visual detection includes the following steps:
[0007] Step S1: Data is acquired through real-time monitoring video of highway or tunnel entrances. The corresponding edge computing device reads the real-time data stream and configures the visual large model algorithm.
[0008] Step S2: The edge computing device runs a large visual model algorithm to segment the road surface and obtain the road surface segmentation results.
[0009] Step S3: The edge computing device runs an image processing algorithm to detect foreign objects in the road surface segmentation results and sends the detection results to the cloud server.
[0010] In step S4, the cloud server identifies the road disaster results based on the detection results and issues a warning with danger information.
[0011] Further technology of the present invention:
[0012] Preferably, step S1 specifically includes:
[0013] Step S11: Data is acquired through real-time monitoring video of highway or tunnel entrance, and the corresponding edge computing device reads the real-time data stream;
[0014] Step S12: Configure the visual large model algorithm on the edge computing device.
[0015] Preferably, step S12, configuring the large visual model algorithm on the edge computing device, includes:
[0016] Step S121: Load the improved visual large model algorithm: Segment Anything Model 2, abbreviated as SAM2;
[0017] Step S122: Initialize the SAM2 inference state and define the prompt messages.
[0018] Preferably, step S121 specifically includes:
[0019] To adapt to the real-time processing requirements of consecutive frames in video streams, this invention structurally optimizes the original SAM2 model. First, an inter-frame memory mechanism is constructed to store image features and masking results from historical frames in an updatable memory for reference in subsequent frames. Second, a Transformer-based spatiotemporal feature propagation structure is adopted, introducing a cross-frame attention mechanism between the current frame and historical frames to achieve spatiotemporal consistency propagation of masking prompts across multiple frames.
[0020] Preferably, step S122 specifically includes:
[0021] The first frame of the real-time data stream is loaded. The user provides prompts to the SAM2 model by clicking on road regions. Clicking on positive points indicates adding the target road region, and clicking on negative points indicates deleting non-target road regions. This is used to define and refine the segmentation of the target road.
[0022] Preferably, step S2 specifically includes:
[0023] In step S21, after receiving the initial prompt information Point Prompts in the first frame, the SAM2 model continuously propagates the prompts to the real-time loaded video stream, generating the spatiotemporal mask of the target road in all frames.
[0024] Preferably, step S21, which involves running the Visual Large Model Algorithm (SAM2) to achieve cue-based frame-level real-time road segmentation, includes:
[0025] Step S211: The SAM2 model extracts image features from the current frame using an image encoder to generate unconditional feature embeddings.
[0026] In step S212, the SAM2 model embeds the current frame image features through the Transformer module and conditions them with the historical frame image feature embeddings and prediction mask results in the memory bank.
[0027] In step S213, the SAM2 model encodes the point prompt information PointsPrompts defined in the first frame into prompt feature embeddings through the prompt encoder;
[0028] In step S214, the SAM2 model decodes the conditional image feature embedding from step S212 and the cue feature embedding from step S213 through the mask decoder, generates the target road segmentation prediction mask for the current frame, and saves the result in the memory bank.
[0029] Preferably, step S211 includes:
[0030] SAM2's image encoder uses the MAE pre-trained Hiera encoder, supports multi-scale feature decoding, and leverages contextual information to extract features from the current frame loaded in real time, generating raw unconditional feature embeddings. These feature embeddings do not depend on any specific task or cue information; they simply represent the image content of the current frame.
[0031] Preferably, step S212 includes:
[0032] SAM2 uses four identical Transformer blocks. It takes the unconditional feature embedding of the current frame output from step S211 as input, and conditions it with the feature embedding of the historical frames in the memory and the prediction mask result to generate the conditional feature embedding of the current frame. This feature embedding contains the spatial information of the image and enhances the spatiotemporal consistency of the target object in the current frame.
[0033] Each Transformer block includes a self-attention layer to capture dependencies within the current frame, i.e., the relationships between different positions in the current frame; a cross-attention layer to capture dependencies between the current frame and historical frame features and prediction mask results in the memory; and a Multi-Layer Perceptron.
[0034] Preferably, step S214 includes:
[0035] First, SAM2 adds learnable output tokens to the prompt embedding output in step S213 to form tokens.
[0036] Secondly, the SAM2 mask decoder uses a Two-way Transformer structure, taking the conditional frame feature embedding and tokens output in step S212 as input, and using self-attention and cross-attention mechanisms to update the frame feature embedding and tokens.
[0037] Finally, the updated frame feature embedding and the learned output tokens are multiplied by a matrix to obtain the road prediction mask for the current frame. This prediction mask and the conditional frame feature embedding output in step S212 are stored in memory to provide information for road target segmentation in the next frame.
[0038] Preferably, in step S3, the edge computing device runs an image processing algorithm to detect foreign objects in the road surface segmentation results and sends the detection results to the cloud server, including:
[0039] Step S31: Perform foreign object detection on the road surface segmentation result of step S2;
[0040] Step S32: Send the detection results to the cloud server.
[0041] Preferably, step S31, which involves foreign object detection of the road surface segmentation result from step S2, includes:
[0042] The road surface segmentation mask of the current frame obtained in step S2 is compared with the "no foreign object reference road surface mask" by differential analysis to extract the cavity area. The cavity part is then extracted and input into the YOLO model for identification and classification. The categories are [pedestrian, motor vehicle, non-motor vehicle, foreign object].
[0043] This invention optimizes the structure and inference strategy of the original YOLO object detection algorithm to efficiently identify small foreign objects within specific areas in highway scenarios. First, while maintaining the lightweight structure of the backbone network CSPDarknet, high-resolution feature channels are added to improve the model's receptive field and spatial detail preservation for small objects. Second, a Region Attention Module is introduced, enabling the model to focus on the hollow regions extracted after SAM2 segmentation, ignoring irrelevant background, thereby improving recognition accuracy and inference speed.
[0044] Preferably, step S4, in which the cloud server identifies the road disaster results based on the detection results and issues a warning of danger, includes:
[0045] The cloud server receives and analyzes the foreign object detection result of the current frame sent in step S3. If the detection result category is [pedestrian, motor vehicle, non-motor vehicle], no alarm information is triggered. If the detection category is [foreign object], the cloud server will store and analyze the detection results of the next ten consecutive frames. When the [foreign object] detection result appears in the same location in ten consecutive frames, the system will trigger an alarm signal and issue an alarm.
[0046] The beneficial effects of this invention are:
[0047] This invention integrates advanced AI algorithms, high-performance hardware, and flexible cloud services to construct an efficient, intelligent, and sustainable highway geological disaster monitoring and early warning system. This system overcomes the limitations of manual inspections and traditional road video detection algorithms, achieving integrated, all-weather, high-precision, and comprehensive identification and tracking monitoring of highway geological disasters and abnormal incidents such as debris spills. It also enables timely early warning pushes and traffic closures, ensuring rapid response and scientific decision-making after a disaster, and saving lives and property. Attached Figure Description
[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0049] Figure 1 Flowchart of a highway geological disaster monitoring and early warning method integrating road segmentation and visual detection according to the present invention;
[0050] Figure 2 : A schematic diagram illustrating the definition prompts of this invention. Detailed Implementation
[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0052] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.
[0053] According to embodiments of the present invention, a method for monitoring and early warning of highway geological disasters that integrates road segmentation and visual detection is proposed, such as... Figure 1 As shown, it includes the following steps:
[0054] Step S1 involves acquiring data through real-time monitoring videos of highways or tunnel entrances. The corresponding edge computing device reads the real-time data stream and configures the large-scale visual model algorithm, specifically including:
[0055] Step S11: Data is acquired through real-time monitoring video of highway or tunnel entrance, and the corresponding edge computing device reads the real-time data stream;
[0056] Step S12, configure the visual large model algorithm model on the edge computing device. The specific process includes:
[0057] Step S121: Load the improved visual large model algorithm: Segment Anything Model 2 (SAM2).
[0058] Specifically, to adapt to the real-time processing requirements of consecutive frames in a video stream, this invention performs structural optimization on the original SAM2 model. First, an inter-frame memory mechanism is constructed to store image features and masking results from historical frames in an updatable memory for reference in subsequent frames. Second, a Transformer-based spatiotemporal feature propagation structure is adopted, introducing a cross-frame attention mechanism between the current frame and historical frames to achieve spatiotemporal consistency propagation of masking information across multiple frames.
[0059] Step S122: Initialize the SAM2 inference state and define the prompts.
[0060] Specifically, the image frames in the real-time data stream are loaded. In the first frame, the user provides prompts to the SAM2 model by clicking on road regions. Clicking on positive points indicates adding the target road region, while clicking on negative points indicates deleting non-target road regions. This is used to define and refine the segmentation of the target road.
[0061] Specifically, such as Figure 2 As shown, blue dots represent positive points, used to point to road targets, while red dots represent negative points, used to exclude non-road areas. By clicking on different positive and negative points in the first frame, the user obtains the pixel coordinates (X,Y) of the points, which indicate the location of roads in the image to the SAM2 model, thus eliminating interference from non-road areas and improving the model's accuracy in recognizing and tracking road areas in real-time video streams.
[0062] Step S2: The edge computing device runs a large visual model algorithm to segment the highway surface and obtain the segmentation results, specifically including:
[0063] Step S21: After receiving the initial cue information of the first frame, the SAM2 model continuously propagates the cue to the real-time loaded video stream, generating the spatiotemporal mask of the target road across all frames, specifically including:
[0064] In step S211, the SAM2 model extracts image features from the current frame through an image encoder, providing unconditional feature embedding.
[0065] Specifically, SAM2's image encoder uses a MAE-pretrained Hiera encoder, supporting multi-scale feature decoding and leveraging contextual information to extract features from the currently loaded frame in real-time, generating raw unconditional feature embeddings. The image encoder employs a pyramid structure, using convolutional layers with stride=16 and stride=32 in the third and fourth layers respectively to extract image features. The features from the two layers are then fused to generate the image feature embeddings for each frame. These feature embeddings do not depend on any specific task or cue information; they purely represent the image content of the current frame.
[0066] In step S212, the SAM2 model embeds the current frame image features through the Transformer module and conditions them with the historical frame feature embeddings and prediction mask results in the memory.
[0067] Specifically, by utilizing the self-attention and cross-attention mechanisms in four identical Transformer blocks, the unconditional feature embedding of the current frame output in step S211 is taken as input and conditionally combined with the feature embeddings of historical frames in the memory and the prediction mask results to generate the conditional feature embedding of the current frame. This feature embedding contains the spatial information of the image and enhances the spatiotemporal consistency of the target object in the current frame.
[0068] Each Transformer block includes a self-attention layer to capture dependencies within the current frame, i.e., the relationships between different locations (pixels or regions) in the current frame; a cross-attention layer to capture dependencies between the current frame and historical frame features and prediction mask results in the memory; and a Multi-Layer Perceptron (MLP).
[0069] Specifically, the self-attention mechanism takes the feature embedding of the current frame as input, then calculates the attention weight between each position and other positions to represent their correlation, and finally performs a weighted sum of the features based on the attention weights to generate a new feature representation of the current frame.
[0070] The cross-attention mechanism embeds the features of the current frame as a query, and the features of historical frames and mask features in the memory as keys and values. Then, it calculates the attention weights between the current frame and the historical frames in the memory to represent their correlation. Finally, it performs a weighted summation of the mask features in the memory based on the attention weights to generate a new feature representation of the current frame.
[0071] The specific attention weights are calculated as follows:
[0072]
[0073] In the self-attention mechanism, Q (Query), K (key), and V (Value) come from the same sequence, while in the cross-attention mechanism, Q, K, and V come from different sequences.
[0074] The updated current frame feature embedding is input into the MLP layer to obtain the conditional current frame image feature embedding.
[0075] In step S213, the SAM2 model encodes the point prompts (PointsPrompts) defined in the first frame into prompt feature embeddings through the prompt encoder;
[0076] In step S214, the SAM2 model decodes the conditional image feature embedding from step S212 and the cue feature embedding from step S213 through the mask decoder, generates the target road segmentation prediction mask for the current frame, and saves the result in the memory bank.
[0077] Specifically, firstly, SAM2 adds learnable output tokens to the prompt embedding output in step S213 to form tokens.
[0078] Secondly, the SAM2 mask decoder uses a Two-way Transformer structure, taking the conditional frame feature embedding and tokens output in step S212 as input, and using self-attention and cross-attention mechanisms to update the frame feature embedding and tokens.
[0079] Specifically, the conditional frame feature embedding is used as a query, and tokens are used as keys and values for cross-attention calculation to update the current frame feature embedding. Simultaneously, tokens are used as queries, and the conditional frame feature embeddings are used as keys and values for cross-attention calculation to update tokens, forming a Two-way Transformer structure.
[0080] Finally, the output tokens extracted from the updated tokens are multiplied by the updated current frame feature embedding to obtain the road prediction mask for the current frame. This prediction mask and the conditional frame feature embedding output in step S212 are stored in the memory to provide information for road target segmentation in the next frame.
[0081] Step S3: The edge computing device runs an image processing algorithm to detect foreign objects in the road surface segmentation results and sends the detection results to the cloud server, specifically including:
[0082] Step S31: Perform foreign object detection on the road surface segmentation results of step S2.
[0083] Specifically, the road surface segmentation result of the current frame obtained in step S2 is compared with the "no foreign object reference road surface mask" by differential analysis to extract the cavity area. The cavity part is then extracted and input into the YOLO model for identification and classification, with the categories being [pedestrian, motor vehicle, non-motor vehicle, foreign object].
[0084] This invention optimizes the structure and inference strategy of the original YOLO object detection algorithm to efficiently identify small foreign objects in specific areas (such as voids in road masks) in highway scenarios. First, while maintaining the lightweight structure of the backbone network CSPDarknet, high-resolution feature channels are added to improve the model's receptive field and spatial detail preservation for small objects. Second, a Region Attention Module is introduced, enabling the model to focus on the void regions extracted after SAM2 segmentation, ignoring irrelevant background, thereby improving recognition accuracy and inference speed.
[0085] Specifically, this invention improves upon the YOLO results as follows: In the Backbone part, the original first-layer convolution is adjusted from Conv(3,32,6,2,2) to Conv(3,64,6,1,1) to enhance initial feature preservation; simultaneously, Conv(128,128,3,1,1) and a Region Attention Module are introduced into the output path of the third layer to improve the response capability to targets with weak textures. In the Neck part, a feature fusion module is introduced to fuse multi-scale shallow features from the second, third, and fourth layers to enhance the contextual understanding of small targets.
[0086] Step S32: Send the detection results to the cloud server.
[0087] In step S4, the cloud server identifies the road disaster results based on the detection results and issues a warning with danger information.
[0088] Specifically, the cloud server receives and analyzes the hole detection result of the current frame sent in step S3. If the detection result category is [pedestrian, motor vehicle, non-motor vehicle], no alarm information is triggered; if the detection category is [foreign object], the cloud server will store and analyze the detection results of the next ten consecutive frames. When the [foreign object] detection result appears in the same position in ten consecutive frames, an alarm information is issued.
[0089] The above description is merely an example and illustration of the structure of the present invention. Those skilled in the art can make various modifications or additions to the specific embodiments described, or use similar methods to replace them, as long as they do not deviate from the structure of the invention or exceed the scope defined in the claims, all of which should fall within the protection scope of the present invention.
Claims
1. A highway geological disaster monitoring and early warning method integrating road segmentation and visual detection, characterized in that, Includes the following steps: Step S1: Data is acquired through real-time monitoring video of highway or tunnel entrances. The corresponding edge computing device reads the real-time data stream and configures the visual large model algorithm. Step S2: The edge computing device runs a large visual model algorithm to segment the road surface and obtain the road surface segmentation results. Step S3: The edge computing device runs an image processing algorithm to detect foreign objects in the road surface segmentation results and sends the detection results to the cloud server. In step S4, the cloud server identifies the road disaster results based on the detection results and issues a warning with danger information.
2. The highway geological disaster monitoring and early warning method integrating road segmentation and visual detection as described in claim 1, characterized in that, Configuring large visual model algorithms on edge computing devices includes: Step S121: Load the improved visual large model algorithm: Segment Anything Model 2, abbreviated as SAM2; Step S122: Initialize the SAM2 inference state and define the prompt messages.
3. The method for monitoring and early warning of highway geological disasters integrating road segmentation and visual detection according to claim 1, characterized in that, Step S2 specifically includes: In step S21, after receiving the initial prompt information Point Prompts in the first frame, the SAM2 model continuously propagates the prompts to the real-time loaded video stream, generating the spatiotemporal mask of the target road in all frames.
4. The method for monitoring and early warning of highway geological disasters integrating road segmentation and visual detection according to claim 3, characterized in that, Step S21, which involves running the Visual Large Model Algorithm (SAM2) to achieve prompt-based frame-level real-time road segmentation, includes: Step S211: The SAM2 model extracts image features from the current frame using an image encoder to generate unconditional feature embeddings. In step S212, the SAM2 model embeds the current frame image features through the Transformer module and conditions them with the historical frame image feature embeddings and prediction mask results in the memory bank. In step S213, the SAM2 model encodes the Point Prompts information defined in the first frame into a cue feature embedding through the cue encoder; In step S214, the SAM2 model decodes the conditional image feature embedding from step S212 and the cue feature embedding from step S213 through the mask decoder, generates the target road segmentation prediction mask for the current frame, and saves the result in the memory bank.
5. The highway geological disaster monitoring and early warning method integrating road segmentation and visual detection according to claim 4, characterized in that, Step S211 includes: SAM2's image encoder uses the MAE pre-trained Hiera encoder, supports multi-scale feature decoding, and utilizes contextual information to extract features from the current frame loaded in real time, generating the original unconditional feature embedding.
6. The highway geological disaster monitoring and early warning method integrating road segmentation and visual detection according to claim 4, characterized in that, Step S212 includes: SAM2 uses four Transformer blocks with the same structure. It takes the unconditional feature embedding of the current frame output from step S211 as input, and conditions it with the feature embedding of the historical frames in the memory and the prediction mask result to generate the conditional feature embedding of the current frame. Each Transformer block includes a self-attention layer to capture dependencies within the current frame, a cross-attention layer to capture dependencies between the current frame and historical frame features and prediction mask results in the memory, and a Multi-Layer Perceptron.
7. The highway geological disaster monitoring and early warning method integrating road segmentation and visual detection according to claim 4, characterized in that, Step S214 includes: First, SAM2 adds learnable output tokens to the prompt embedding output in step S213 to form tokens; Secondly, the SAM2 mask decoder uses a Two-way Transformer structure, taking the conditional frame feature embedding and tokens output in step S212 as input, and using self-attention and cross-attention mechanisms to update the frame feature embedding and tokens. Finally, the updated frame feature embedding and the learned output tokens are multiplied by a matrix to obtain the road prediction mask for the current frame. This prediction mask and the conditional frame feature embedding output in step S212 are stored in the memory bank to provide information for road target segmentation in the next frame.
8. The method for monitoring and early warning of highway geological disasters integrating road segmentation and visual detection according to claim 1, characterized in that, In step S3, the edge computing device runs an image processing algorithm to detect foreign objects in the road segmentation results and sends the detection results to the cloud server, including: Step S31: Perform foreign object detection on the road surface segmentation result of step S2; Step S32: Send the detection results to the cloud server.
9. A method for monitoring and early warning of highway geological disasters integrating road segmentation and visual detection according to claim 8, characterized in that, Step S31, which involves foreign object detection of the road surface segmentation results from step S2, includes: The road surface segmentation mask of the current frame obtained in step S2 is compared with the "no foreign object reference road surface mask" by differential analysis to extract the cavity region. The cavity part is then extracted and input into the YOLO model for identification and classification, and the categories are pedestrians, motor vehicles, non-motor vehicles and foreign objects.
10. A method for monitoring and early warning of highway geological disasters integrating road segmentation and visual detection according to claim 1, characterized in that, Step S4, in which the cloud server identifies the road disaster results based on the detection results and issues a warning of danger, includes: The cloud server receives and analyzes the foreign object detection result of the current frame sent in step S3. If the detection result category is pedestrian, motor vehicle and non-motor vehicle, no alarm information is triggered; if the detection category is foreign object, the cloud server will store and analyze the detection results of the next ten consecutive frames. When the foreign object detection result appears in the same location in ten consecutive frames, the system triggers the alarm signal and issues an alarm.