A flood disaster real-time monitoring method based on unmanned aerial vehicle video panoramic segmentation

Abstract

This invention discloses a real-time monitoring method for flood disasters based on UAV video panoramic segmentation, comprising the following steps: collecting a large amount of UAV flood disaster video data; processing it into multiple non-repeating short videos; manually annotating each short video frame by frame; preprocessing and augmenting the frame images; dividing the data into training, validation, and test sets; constructing an improved lightweight video panoramic segmentation Video-DL network; training and optimizing the model using the annotated training data; outputting the initial segmentation results and corresponding pixel-level confidence maps; fine-tuning the multimodal segmentation model using the annotated training data; constructing a hierarchical inference mechanism for the lightweight visual segmentation Video-DL sub-network and the multimodal segmentation model to work collaboratively; and inputting the test data into the above model to achieve video-based panoramic segmentation. This invention can be widely applied in key areas such as emergency management and disaster relief.

Description

Technical Field

[0001] This invention relates to image processing technology in computer vision, and in particular to a real-time segmentation method for flood disaster scene images based on deep learning. Background Technology

[0002] In recent years, traditional methods for monitoring flood disasters in China, especially urban flooding, have mainly included remote sensing monitoring and ground-based equipment monitoring. Remote sensing monitoring obtains information on urban flooding through satellite data, but its disadvantages include the inability to provide real-time data and its susceptibility to weather and cloud cover. Ground-based equipment monitoring mainly acquires data through sensors and monitoring stations, but its disadvantages include limited coverage, high monitoring costs, and the need for significant manpower and resources for maintenance and management.

[0003] Compared to these methods, the UAV-based video panorama segmentation method has the following advantages:

[0004] 1. Real-time capability: Drones can quickly deploy for monitoring during floods, acquiring real-time panoramic video of the affected areas. Through video panoramic segmentation technology, drones can automatically identify key areas in the video, such as flooded areas, damaged buildings, and trapped people, providing accurate and real-time information for rescue efforts.

[0005] 2. Coverage range: Drones can monitor large areas in real time, without being limited by geographical location, and have a wide coverage range.

[0006] 3. Data accuracy: The UAV-based video panoramic segmentation method can obtain high-precision flood disaster information, which helps to improve the accuracy of disaster assessment and rescue work.

[0007] 4. Flexibility: Drones can adjust their shooting angle and altitude as needed to acquire video data from different perspectives and resolutions, providing more diverse information for flood disaster monitoring.

[0008] However, the flood disaster monitoring method based on drone video panoramic segmentation still has some limitations.

[0009] Video data acquired by drones may be affected by factors such as lighting conditions and weather conditions, leading to a decline in data quality and affecting the accuracy of segmentation results. Due to the altitude limitations of drones, the detection effect of small targets such as people and vehicles is poor. Image segmentation networks such as FCN and SegNet have slow inference speed in video panorama segmentation, which will affect real-time performance. Summary of the Invention

[0010] The purpose of this invention is to address the shortcomings of existing technologies that are limited by weather, lighting, small target detection, and real-time performance, and to provide an improved flood disaster monitoring method based on UAV video panoramic segmentation.

[0011] The technical solution adopted in this invention is: a real-time monitoring method for flood disasters based on UAV video panoramic segmentation, comprising the following steps:

[0012] Step 1: Collect a large amount of video data on flood disasters using drones, including video data under different lighting and weather conditions;

[0013] Step 2: Trim the video data to create multiple non-repeating short videos, each approximately 10 seconds long.

[0014] Step 3: Perform panoramic manual annotation on each frame of each short video, including categories such as vehicles, people, buildings, water surfaces, roads, low vegetation, and others;

[0015] Step 4: Preprocess and augment the frame image;

[0016] Step 5: Divide the dataset into training, validation, and test sets in a 7:2:1 ratio;

[0017] Step 6: Based on the semantic segmentation network with DeepLabv3+ encoding and decoding structure, construct an improved lightweight video panorama segmentation network, Video-DL network;

[0018] Step 7: Based on the constructed Video-DL network for panoramic video segmentation, train the network using labeled training data, evaluate the model performance using the validation set, and optimize the model.

[0019] Step 8: Use the improved lightweight Video-DL network to segment the input video frames in real time, and output the initial segmentation results and the corresponding pixel-level confidence map.

[0020] Step 9: Fine-tune the large multimodal segmentation model using the labeled training set data;

[0021] Step 10: Construct a hierarchical inference mechanism for the collaborative operation of a lightweight visual segmentation Video-DL sub-network and a multimodal segmentation large model. When the indicators meet the preset triggering conditions, the multimodal segmentation large model outputs a high-precision instance segmentation mask and panoramic semantic labels based on the mask results output by the sub-network, thereby optimizing the panoramic segmentation results.

[0022] Step 11: Input the test set data into the above model to achieve video-based panoramic segmentation.

[0023] Furthermore, step 4 specifically includes the following steps:

[0024] 1) Convert the format of the dataset; the labeled videos cannot be used directly for the panoptic segmentation task and need to be converted into the COCO panoptic segmentation dataset format to facilitate subsequent network training and validation.

[0025] 2) The preprocessing process includes cleaning the dataset, analyzing data categories, adjusting the weights of long-tail categories, and uniformly cropping the images to 640×640.

[0026] 3) Image augmentation includes image contrast transformation, image rotation, image flipping, and image cropping.

[0027] Furthermore, in step 6, the constructed video panoramic segmentation Video-DL network is based on the DeepLabv3+ encoding and decoding semantic segmentation network, and mainly includes a backbone network, a decoder, an optical flow network, a DFF (Deep Feature Flow) network, and a post-processing module.

[0028] 1) The Video-DL network for video panorama segmentation uses an encoder-decoder structure; the original encoder part uses ResNet-101 as the backbone network for feature extraction; to reduce network parameters and improve model speed, this invention uses the MHSA (Multi-head Self-attention) structure to replace the 3×3 convolution; the core part of MHSA is the self-attention layer, which consists of three sub-layers: query, key, and value; attention weights are calculated between the query and key, and then these weights are used to perform a weighted summation of the value to obtain the contextual information of each position in the sequence;

[0029] In feature extraction networks, MHSA can weightedly fuse the input information at each position in the sequence with information from other positions, better capturing contextual information, which helps extract features and improves the model's performance in video temporal prediction. Multi-head setup allows the model to focus on different semantic information at different levels, resulting in richer expressive power and improved generalization ability in complex tasks. MHSA reduces the number of model parameters and increases the model's running speed by sharing parameters. The improved backbone network reduces model parameters while maintaining model accuracy, making model training and prediction faster.

[0030] 2) Accelerate video panorama segmentation using the DFF (Deep Feature Flow) network; the main steps are to set a keyframe every K frames, and obtain the feature map of the keyframe by computing the entire network. The feature map of non-keyframes is obtained by calculating the optical flow between the current frame and the keyframe through the flownet (optical flow estimation network), and then passing the optical flow to the current frame. Since the flownet is shallow, the computational cost is much less than that of the feature extraction network. This structure can accelerate the panorama segmentation of images and greatly improve the model speed.

[0031] 3) The decoder module mainly consists of 1×1 convolution, 3×3 convolution and upsampling layers, which fuse low-level features from the encoder and high-level features obtained after upsampling through skip connections;

[0032] 4) The post-processing module includes statistical analysis of the segmentation results to obtain information such as the number of people and vehicles in the image and whether the road is flooded.

[0033] Furthermore, step 7 specifically includes the following steps:

[0034] 1) The difference between the predicted segmentation map and the actual segmentation map is calculated using three loss functions: cross entropy loss, focal loss, and dice soft loss.

[0035] In the cross entropy loss, the loss for each pixel is calculated as follows:

[0036]

[0037] in, This represents the loss per pixel. Indicates the true label, Indicates the predicted label, Indicates category;

[0038] The overall cross-entropy loss is the average of the losses of all pixels;

[0039] The focal loss is calculated as follows:

[0040]

[0041] in, This represents focal loss. Indicates the true label, Indicates the predicted label, Indicates category, Indicates category weight, This indicates adjusting the weights of "difficult samples";

[0042] In dice soft loss, the loss for each category is calculated as follows:

[0043]

[0044] in, This represents the dice soft loss for each category. Indicates the true label, This represents the predicted label; the Dice soft loss is the sum of the Dice losses for each class and the average of the sums.

[0045] 2) Use the AdamW optimization algorithm to optimize the loss function and accelerate model convergence;

[0046] 3) After each training round, the validation set data is input into the constructed network. The optimal validation result is obtained through validation set evaluation. The network model is saved and used as the basis for parameter tuning.

[0047] Furthermore, step 8 specifically includes the following steps:

[0048] 1) Test set prediction is performed using the TTA (Test-Time Augmentation) strategy, and the augmentation methods used include shifting, cropping, and image flipping;

[0049] 2) Use the tensorRT inference engine to accelerate model inference and ensure real-time prediction.

[0050] Furthermore, step 9 specifically includes the following steps:

[0051] 1) Using high-precision panoramic segmentation mask data annotated manually as training data, segmentation prompts (Mask Prompt) and target box prompts (Box Prompt) generated based on real annotations are constructed for each frame of image.

[0052] 2) A weighted combination of Dice loss and Focal loss is used during training.

[0053] Furthermore, step 10 specifically includes the following steps:

[0054] 1) Based on the output of the improved lightweight video panorama segmentation Video-DL network, a confidence evaluation module is constructed to calculate the average category confidence, small target region confidence, and continuous frame result difference for each frame segmentation result;

[0055] The average category confidence score is calculated as follows:

[0056]

[0057] The confidence score for a small target region is calculated as follows:

[0058]

[0059] in, This is a pixel-level category probability map. The values ​​represent pixel coordinates, c represents the category, and H and W represent the image height and width, respectively. Represents pixels in a small target area;

[0060] The difference between consecutive frame results is evaluated using segmentation volatility, calculated as follows:

[0061]

[0062]

[0063] The current frame segmentation result is: The previous frame was , Indicates inter-frame , Indicates the segmented volatility;

[0064] 2) When the calculated average class confidence, small target region confidence, and continuous frame result difference index are lower than the threshold, connected component analysis is performed on the low confidence region to extract the candidate region bounding box. The candidate bounding box is used as the Box Prompt, and the initial mask output by the small model is used as the Mask Prompt and input into the fine-tuned multimodal segmentation large model to generate a high-precision instance mask.

[0065] 3) The fusion model segmentation results are fused using the following method:

[0066]

[0067] in, This indicates the final segmentation result. This represents the segmentation results of the lightweight small model. This represents the segmentation results of a large multimodal model. Represents pixel coordinates, This indicates a region with low confidence.

[0068] The beneficial effects of the present invention: The present invention provides an improved flood disaster monitoring method based on UAV video panoramic segmentation, which can solve the problems of existing technologies being limited by weather, lighting, small target detection, and real-time performance, and can be widely applied in key areas such as emergency management and disaster relief. Attached Figure Description

[0069] Figure 1 This invention provides an improved Video-DL network diagram for panoramic video segmentation.

[0070] Figure 2 This is a diagram of the MHSA network structure of the present invention;

[0071] Figure 3 This is a flowchart of the overall process of drone video panoramic segmentation of the present invention. Detailed Implementation

[0072] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0073] like Figure 3 As shown, a real-time flood disaster monitoring method based on UAV video panoramic segmentation includes the following steps:

[0074] Step 1: Collect a large amount of video data on flood disasters using drones, including video data under different lighting and weather conditions;

[0075] Step 2: Trim the video data to create multiple non-repeating short videos, each approximately 10 seconds long.

[0076] Step 3: Perform panoramic manual annotation on each frame of each short video, including categories such as vehicles, people, buildings, water surfaces, roads, low vegetation, and others;

[0077] Step 4: Preprocess and augment the frame image;

[0078] Specifically, the following steps are included:

[0079] 1) Convert the format of the dataset; the labeled videos cannot be used directly for the panoptic segmentation task and need to be converted into the COCO panoptic segmentation dataset format to facilitate subsequent network training and validation.

[0080] 2) The preprocessing process includes cleaning the dataset, analyzing data categories, adjusting the weights of long-tail categories, and uniformly cropping the images to 640×640.

[0081] 3) Image augmentation includes image contrast transformation, image rotation, image flipping, and image cropping;

[0082] Step 5: Divide the dataset into training, validation, and test sets in a 7:2:1 ratio;

[0083] Step 6: Based on the semantic segmentation network with DeepLabv3+ encoding and decoding structure, construct an improved lightweight video panorama segmentation network, Video-DL network;

[0084] The constructed Video-DL network for panoramic video segmentation is based on the DeepLabv3+ encoding and decoding semantic segmentation network, and mainly includes a backbone network, decoder, optical flow network, DFF (Deep Feature Flow) network, and post-processing module;

[0085] 1) The Video-DL network for video panorama segmentation uses an encoder-decoder structure, and the overall process is shown in the attached figure. Figure 1 The original encoder used ResNet-101 as the backbone network for feature extraction; to reduce network parameters and improve model speed, this invention uses an MHSA (Multi-head Self-attention) structure instead of 3×3 convolutions, as shown in the attached diagram. Figure 2 ;

[0086] The core of MHSA is the Self-Attention Layer, which consists of three sub-layers: query, key, and value. Attention weights are calculated between the query and key, and then these weights are used to sum the values ​​in a weighted manner to obtain contextual information for each position in the sequence.

[0087] In feature extraction networks, MHSA can weightedly fuse the input information at each position in the sequence with information from other positions, better capturing contextual information, which helps extract features and improves the model's performance in video temporal prediction. Multi-head setup allows the model to focus on different semantic information at different levels, resulting in richer expressive power and improved generalization ability in complex tasks. MHSA reduces the number of model parameters and increases the model's running speed by sharing parameters. The improved backbone network reduces model parameters while maintaining model accuracy, making model training and prediction faster.

[0088] 2) Accelerate video panorama segmentation using the DFF (Deep Feature Flow) network; the main steps are to set a keyframe every K frames, and obtain the feature map of the keyframe by computing the entire network. The feature map of non-keyframes is obtained by calculating the optical flow between the current frame and the keyframe through the flownet (optical flow estimation network), and then passing the optical flow to the current frame. Since the flownet is shallow, the computational cost is much less than that of the feature extraction network. This structure can accelerate the panorama segmentation of images and greatly improve the model speed.

[0089] 3) The decoder module mainly consists of 1×1 convolution, 3×3 convolution and upsampling layers, which fuse low-level features from the encoder and high-level features obtained after upsampling through skip connections;

[0090] 4) The post-processing module includes statistical analysis of the segmentation results to obtain information such as the number of people and vehicles in the image and whether the road is flooded;

[0091] Step 7: Based on the constructed Video-DL network for panoramic video segmentation, train the network using labeled training data, evaluate the model performance using the validation set, and optimize the model.

[0092] Specifically, the following steps are included:

[0093] 1) The difference between the predicted segmentation map and the actual segmentation map is calculated using three loss functions: cross entropy loss, focal loss, and dice soft loss.

[0094] In the cross entropy loss, the loss for each pixel is calculated as follows:

[0095]

[0096] in, This represents the loss per pixel. Indicates the true label, Indicates the predicted label, Indicates category;

[0097] The overall cross-entropy loss is the average of the losses of all pixels;

[0098] The focal loss is calculated as follows:

[0099]

[0100] in, This represents focal loss. Indicates the true label, Indicates the predicted label, Indicates category, Indicates category weight, This indicates adjusting the weights of "difficult samples";

[0101] In dice soft loss, the loss for each category is calculated as follows:

[0102]

[0103] in, This represents the dice soft loss for each category. Indicates the true label, This represents the predicted label; the Dice soft loss is the sum of the Dice losses for each class and the average of the sums.

[0104] 2) Use the AdamW optimization algorithm to optimize the loss function and accelerate model convergence;

[0105] 3) After each training round, the validation set data is input into the constructed network. The optimal validation result is obtained through validation set evaluation. The network model is saved and used as the basis for parameter tuning.

[0106] Step 8: Use the improved lightweight Video-DL network to segment the input video frames in real time, and output the initial segmentation results and the corresponding pixel-level confidence map.

[0107] Specifically, the following steps are included:

[0108] 1) Test set prediction is performed using the TTA (Test-Time Augmentation) strategy, and the augmentation methods used include shifting, cropping, and image flipping;

[0109] 2) Use the TensorRT inference engine to accelerate model inference and ensure real-time prediction;

[0110] Step 9: Fine-tune the large multimodal segmentation model using the labeled training set data;

[0111] Specifically, the following steps are included:

[0112] 1) Using high-precision panoramic segmentation mask data annotated manually as training data, segmentation prompts (Mask Prompt) and target box prompts (Box Prompt) generated based on real annotations are constructed for each frame of image.

[0113] 2) A weighted combination of Dice loss and Focal loss is used during training;

[0114] Step 10: Construct a hierarchical inference mechanism for the collaborative operation of a lightweight visual segmentation Video-DL sub-network and a multimodal segmentation large model. When the indicators meet the preset triggering conditions, the multimodal segmentation large model outputs a high-precision instance segmentation mask and panoramic semantic labels based on the mask results output by the sub-network, thereby optimizing the panoramic segmentation results.

[0115] Specifically, the following steps are included:

[0116] 1) Based on the output of the improved lightweight video panorama segmentation Video-DL network, a confidence evaluation module is constructed to calculate the average category confidence, small target region confidence, and continuous frame result difference for each frame segmentation result;

[0117] The average category confidence score is calculated as follows:

[0118]

[0119] The confidence score for a small target region is calculated as follows:

[0120]

[0121] in, This is a pixel-level category probability map. The values ​​represent pixel coordinates, c represents the category, and H and W represent the image height and width, respectively. Represents pixels in a small target area;

[0122] The difference between consecutive frame results is evaluated using segmentation volatility, calculated as follows:

[0123]

[0124]

[0125] The current frame segmentation result is: The previous frame was , Indicates inter-frame , Indicates the segmented volatility;

[0126] 2) When the calculated average class confidence, small target region confidence, and continuous frame result difference index are lower than the threshold, connected component analysis is performed on the low confidence region to extract the candidate region bounding box. The candidate bounding box is used as the Box Prompt, and the initial mask output by the small model is used as the Mask Prompt and input into the fine-tuned multimodal segmentation large model to generate a high-precision instance mask.

[0127] 3) The fusion model segmentation results are fused using the following method:

[0128]

[0129] in, This indicates the final segmentation result. This represents the segmentation results of the lightweight small model. This represents the segmentation results of a large multimodal model. Represents pixel coordinates, This indicates a region with low confidence.

[0130] Step 11: Input the test set data into the above model to achieve video-based panoramic segmentation.

[0131] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for real-time monitoring of flood disasters based on panoramic video segmentation from unmanned aerial vehicles (UAVs), characterized in that: Includes the following steps: Step 1: Collect a large amount of video data on flood disasters using drones, including video data under different lighting and weather conditions; Step 2: Trim the video data to create multiple non-repeating short videos, each approximately 10 seconds long. Step 3: Perform panoramic manual annotation on each frame of each short video, including categories such as vehicles, people, buildings, water surfaces, roads, low vegetation, and others; Step 4: Preprocess and augment the frame image; Step 5: Divide the dataset into training, validation, and test sets in a 7:2:1 ratio; Step 6: Based on the semantic segmentation network with DeepLabv3+ encoding and decoding structure, construct an improved lightweight video panorama segmentation network, Video-DL network; Step 7: Based on the constructed Video-DL network for panoramic video segmentation, train the network using labeled training data, evaluate the model performance using the validation set, and optimize the model. Step 8: Use the improved lightweight Video-DL network to segment the input video frames in real time, and output the initial segmentation results and the corresponding pixel-level confidence map. Step 9: Fine-tune the large multimodal segmentation model using the labeled training set data; Step 10: Construct a hierarchical inference mechanism for the collaborative operation of a lightweight visual segmentation Video-DL sub-network and a multimodal segmentation large model. When the indicators meet the preset triggering conditions, the multimodal segmentation large model outputs a high-precision instance segmentation mask and panoramic semantic labels based on the mask results output by the sub-network, thereby optimizing the panoramic segmentation results. Step 11: Input the test set data into the above model to achieve video-based panoramic segmentation.

2. The method for real-time monitoring of flood disasters based on UAV video panoramic segmentation according to claim 1, characterized in that: Step 4 specifically includes the following steps: 1) Convert the format of the dataset; the labeled videos cannot be used directly for the panoptic segmentation task and need to be converted into the COCO panoptic segmentation dataset format to facilitate subsequent network training and validation. 2) The preprocessing process includes cleaning the dataset, analyzing data categories, adjusting the weights of long-tail categories, and uniformly cropping the images to 640×640. 3) Image augmentation includes image contrast transformation, image rotation, image flipping, and image cropping.

3. The method for real-time monitoring of flood disasters based on UAV video panoramic segmentation according to claim 2, characterized in that: In step 6, the constructed video panoramic segmentation Video-DL network is based on the DeepLabv3+ encoding and decoding semantic segmentation network, and mainly includes a backbone network, decoder, optical flow network, DFF network and post-processing module. 1) The Video-DL network for video panorama segmentation uses an encoder-decoder structure; the original encoder part uses ResNet-101 as the backbone network for feature extraction; to reduce network parameters and improve model speed, the MHSA structure is used instead of 3×3 convolution; the core of MHSA is the self-attention layer, which consists of three sub-layers: query, key, and value; attention weights are calculated between the query and key, and then these weights are used to perform a weighted summation of the value to obtain the contextual information of each position in the sequence; 2) Accelerating video panorama segmentation using DFF network; the main steps are to set a keyframe every K frames. The feature map of the keyframe is obtained by computing the entire network. The feature map of the non-keyframe is obtained by calculating the optical flow between the current frame and the keyframe through FlowNet and then passing the optical flow to the current frame. Since FlowNet is shallow, its computational cost is much less than that of the feature extraction network. This structure accelerates the panorama segmentation of the image and greatly improves the model speed. 3) The decoder module mainly consists of 1×1 convolution, 3×3 convolution and upsampling layers, which fuse low-level features from the encoder and high-level features obtained after upsampling through skip connections; 4) The post-processing module includes statistical analysis of the segmentation results to obtain information such as the number of people and vehicles in the image and whether the road is flooded.

4. The method for real-time monitoring of flood disasters based on UAV video panoramic segmentation according to claim 3, characterized in that: Step 7 specifically includes the following steps: 1) The difference between the predicted segmentation map and the actual segmentation map is calculated using three loss functions: cross entropy loss, focal loss, and dice soft loss. In the cross entropy loss, the loss for each pixel is calculated as follows: ， in, This represents the loss per pixel. Indicates the true label, Indicates the predicted label, Indicates category; The overall cross-entropy loss is the average of the losses of all pixels; The focal loss is calculated as follows: ， in, This represents focal loss. Indicates the true label, Indicates the predicted label, Indicates category, Indicates category weight, This indicates adjusting the weights of "difficult samples"; In dice soft loss, the loss for each category is calculated as follows: ， in, This represents the dice soft loss for each category. Indicates the true label, This represents the predicted label; the Dice soft loss is the sum of the Dice losses for each class and the average of the sums. 2) Use the AdamW optimization algorithm to optimize the loss function and accelerate model convergence; 3) After each training round, the validation set data is input into the constructed network. The optimal validation result is obtained through validation set evaluation. The network model is saved and used as the basis for parameter tuning.

5. A method for real-time monitoring of flood disasters based on UAV video panoramic segmentation according to claim 4, characterized in that: Step 8 specifically includes the following steps: 1) The TTA strategy was used for test set prediction, and the enhancement methods used included shifting, cropping, and image flipping; 2) Use the tensorRT inference engine to accelerate model inference and ensure real-time prediction.

6. A method for real-time monitoring of flood disasters based on UAV video panoramic segmentation according to claim 5, characterized in that: Step 9 specifically includes the following steps: 1) Using manually annotated high-precision panoramic segmentation mask data as training data, segmentation prompts and target box prompts based on real annotations are constructed for each frame of image; 2) A weighted combination of Dice loss and Focal loss is used during training.

7. A method for real-time monitoring of flood disasters based on UAV video panoramic segmentation according to claim 6, characterized in that: Step 10 specifically includes the following steps: 1) Based on the output of the improved lightweight video panorama segmentation Video-DL network, a confidence evaluation module is constructed to calculate the average category confidence, small target region confidence, and continuous frame result difference for each frame segmentation result; The average category confidence score is calculated as follows: ， The confidence score for a small target region is calculated as follows: ， in, This is a pixel-level category probability map. The values ​​represent pixel coordinates, c represents the category, and H and W represent the image height and width, respectively. Represents pixels in a small target area; The difference between consecutive frame results is evaluated using segmentation volatility, calculated as follows: ， ， The current frame segmentation result is: The previous frame was , Indicates inter-frame , Indicates the segmented volatility; 2) When the calculated average class confidence, small target region confidence, and continuous frame result difference index are lower than the threshold, connected component analysis is performed on the low confidence region to extract the candidate region bounding box. The candidate bounding box is used as the Box Prompt, and the initial mask output by the small model is used as the Mask Prompt and input into the fine-tuned multimodal segmentation large model to generate a high-precision instance mask. 3) The fusion model segmentation results are fused using the following method: ， in, This indicates the final segmentation result. This represents the segmentation results of a lightweight small model. This represents the segmentation results of a large multimodal model. Represents pixel coordinates, This indicates a region with low confidence.

Images