Ice condition video image recognition method based on TS-MaskFormer

Abstract

The invention discloses an ice condition video image recognition method based on TS-MaskFormer. A rapid ACE defogging algorithm is adopted to enhance a night image, and an ice condition image detection and identification data set covering different stages of freezing, starting of ice melting, floating of floating ice and the like is constructed based on a semi-supervised labeling method of SAM; a TS-MaskFormer ice condition image recognition model is provided, a MaskFormer model and a fusion module of two different backbone networks are integrated, and accurate ice condition image detection is achieved by training, adjusting and optimizing model parameters. And identifying an ice condition image based on the optimized model, and calculating an ice surface coverage rate through an identification result. According to the method, more detail information can be reserved in the aspect of ice condition image recognition, and higher ice surface recognition precision and higher calculation speed are achieved.

Description

Technical Field

[0001] The invention relates to an ice condition video image recognition method based on TS-MaskFormer, and belongs to the field of computer vision image recognition. Background Art

[0002] In recent years, sea ice and river (lake) ice have been listed as "Essential Climate Variables (ECV)" by the Global Observing System (GCOS). Long-term monitoring of the freezing time, thawing time, duration and pattern of river (lake) ice can reflect the changes in local climate. In the rivers in the cold regions of northern China, drifting ice and freezing will occur in the rivers every year, leading to ice floods, also known as "ice floes", which is a hydrological phenomenon in which ice blocks the river channel and creates resistance to water flow, causing a significant rise in the river water level. In rivers with frozen water surfaces, the upstream river ice melts first, while the downstream river channel has not yet thawed, causing floating ice to block the water level and cause the water level to rise. Large areas of floating ice will hit and cut water buildings such as embankments, bridges and docks, which may lead to embankment breaches and floods. Timely ice-breaking measures must be taken when dangerous conditions occur during the ice flood period. Therefore, timely and accurate monitoring of changes in ice information can effectively guide ice flood prevention work and prevent disasters caused by ice floods. In summary, ice condition monitoring is becoming an increasingly important research direction.

[0003] At present, ice condition monitoring data mainly comes from remote sensing satellite images, lidar or infrared sensors, and on-site manual measurements. However, the acquisition of these data usually requires high hardware costs and cannot guarantee the real-time nature of the data. There are few studies on ice condition monitoring based on ice condition video data. Xiao [Xiao, M., Rothermel, M., Tom, M., Galliani, S., Baltsavias, E., Schindler, K., 2018. Lake ice monitoring with webcams [J]. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, IV-2.] et al. first proposed a method for lake ice detection using webcams in 2018. By setting up webcams around the lake and combining them with a convolutional neural network (CNN), they can monitor the morphology and changes of lake ice in real time.

[0004] There are three main methods for processing and analyzing the acquired ice condition data: spectral feature-based methods, which use the differences in spectral characteristics such as the absorption rate and refractive index of the ice surface and the surrounding environment, but this method usually requires repeated experiments to determine the distinction threshold and has poor generalization performance; methods based on traditional machine learning, such as support vector machine (SVM), random forest (RandomForest), decision tree, etc., but their learning ability is not strong enough and the accuracy of ice surface extraction is low; methods based on deep learning, such as convolutional neural network (CNN), residual network (ResNet), etc., have received more attention and better results, but there are currently few studies on the use of deep learning for ice condition video image recognition. Zhang [Zhang H, Liu Y, Li X. Ice concentration estimation using deep learning from SAR imagery[J].IEEE Geoscience and Remote Sensing Letters.2018;15(2):238-242.] et al. proposed a deep learning-based method in 2019, using synthetic aperture radar (SAR) images to estimate ice concentration, and using convolutional neural networks (CNN) to extract and classify SAR images, achieving accurate ice concentration estimation. However, the direct application of ordinary deep learning models to ice condition recognition scenarios is not ideal, and the model still needs to be further improved and optimized. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and propose a method for ice condition video image recognition based on TS-MaskFormer. The proposed method obtains data through monitoring video, has low hardware cost and high real-time performance, and proposes a TS-MaskFormer model that can recognize ice condition video images with high accuracy and calculate the ice surface coverage based on the recognition results.

[0006] Technical solution: The present invention is a method for ice condition video image recognition based on TS-MaskFormer, comprising the following steps:

[0007] S1, collect the original video data of the surveillance camera, intercept the image frames of the read video and classify them;

[0008] S2, preprocessing of ice condition image frames, using a SAM-based semi-supervised annotation method to annotate the preprocessed images, and constructing an ice condition image dataset, including a training set, a validation set, and a test set;

[0009] S3, constructing a TS-MaskFormer model, using the constructed ice condition image detection and recognition dataset to train the model, and obtaining an ice condition image detection and recognition model based on TS-MaskFormer;

[0010] S4, use the TS-MaskFormer-based ice condition image detection and recognition model trained in S3 to detect ice condition images, and calculate the ice surface coverage (C i ).

[0011] Furthermore, the implementation process of step S1 is as follows: video data from a surveillance camera is collected, image frames are captured from the read video, and the images are divided into five categories: ice cover, beginning of ice melting, drifting of floating ice, end of ice melting, and night.

[0012] Furthermore, the implementation process of step S2 is as follows: pre-processing of ice condition image frames, data cleaning and deduplication of 5 types of images, and the problem of foggy and blurred images at night, a fast ACE algorithm is proposed to achieve defogging and enhancement of ice condition images. The implementation process of the fast ACE algorithm is as follows:

[0013] I. Input a color image I, and the dehazing parameters include the contrast enhancement factor ration and the convolution template radius radius.

[0014] II. Create a zero matrix res of the same size as the input image to store the dehazing result.

[0015] III. Loop through the three RGB channels (red, green, blue).

[0016] IV. For each channel, use the single-channel ACE algorithm to perform fast enhanced dehazing. The specific recursive calling process is:

[0017] g. If the minimum value of the image height and width is less than or equal to 2, return a matrix with the same size as the input image, with all elements set to 0.5, and stop the recursion;

[0018] h. If the image size is greater than 2, the input image is reduced by half, that is, the width and height are divided by 2 respectively;

[0019] i. Recursively call the single-channel ACE algorithm on the reduced image, passing in the reduced image, defogging parameters ratio and radius;

[0020] j. Enlarge the result of the recursive call and restore the width and height to the size of the original image;

[0021] k. Restore the reduced image to the size of the original image.

[0022] l. Calculate the final result by adding the result of the recursive call to the result of the dehazing process of the original image by the conventional ACE algorithm, and then subtracting the result of the dehazing process of the reduced image.

[0023] V. Perform linear stretching on the processed image to map the pixel values ​​of the image to the range of [0, 1] to enhance the contrast of the image.

[0024] VI. Store the processed channel image in the corresponding channel of the result matrix res to obtain the dehazing result.

[0025] According to the problem definition, a semi-supervised annotation method based on SAM is used to annotate the preprocessed data, and the image pixels are divided into three categories: background, water, and ice. An ice image dataset is constructed, including a training set, a validation set, and a test set. The implementation process of the semi-supervised annotation method based on SAM is as follows: Figure 3 As shown, the pre-processed ice image data is input, and the pre-trained SAM (Segment Anything) model is used for inference annotation to determine whether the annotation error is less than the threshold. If the error is greater than the threshold, the prompt is manually input to assist in the loop correction. If the error is less than the threshold, the final annotation result is obtained.

[0026] Furthermore, the implementation process of step S3 is as follows: construct a TS-MaskFormer model, use the constructed ice condition image detection and recognition data set to train the model, and obtain an ice condition image detection and recognition model based on TS-MaskFormer. The TS-MaskFormer model structure is as follows: Figure 5 As shown, it consists of the first and second stage MaskFormer and fusionmodule fusion modules.

[0027] The MaskFormer model of each stage consists of a pixel-level module, a transformer module, and a segmentation module. The pixel-level module contains a backbone network and a pixel decoder. The backbone of the first stage model is the Swin-Transformer backbone, and the backbone of the second stage model is the ResNet backbone. The image input passes through the backbone network to generate a low-resolution image feature F, and then passes through the pixel decoder pixel-level decoder to gradually upsample the feature to generate a pixel-by-pixel embedding of binary mask prediction. The Transformer module is a standard Transformer decoder that calculates the output from the image feature F and N learnable position queries. The segmentation module first passes the result of the Transformer module through an MLP multi-layer perceptron, and then uses a linear classifier to obtain the probability prediction for each class; the MLP result is converted into a mask embedding of dimension C and then dot-producted with the result of the previous pixel-by-pixel module to obtain the binary mask prediction probability. The results of the category prediction and mask prediction are matrix multiplied to obtain the final segmentation result.

[0028] The two-stage model recognition results are fused through the fusion module to obtain the final semantic segmentation result. The specific process of the fusion module to achieve the weighted fusion of the confidence of the two-stage model segmentation results is as follows:

[0029] I. Set the weights of Model 1 and Model 2 to W1 and W2 respectively.

[0030] II. Set the prediction accuracy of the model for the three categories on the validation set to the confidence [α1, α2, α3] and [β1, β2, β3] of Model 1 and Model 2 for the three categories.

[0031] III. Calculate the weighted average [P1, P2, P3] based on the confidence and weight. For each category, multiply the confidence of model 1 and model 2 by the corresponding weight, then add the two results and divide by the sum of the weights.

[0032]

[0033] IV. Based on the weighted average, find the index of the maximum value in the weighted average and determine the final pixel prediction category. The final prediction category is the category with the highest confidence.

[0034] Furthermore, the implementation process of step S4 is as follows: the ice condition image detection and recognition model based on TS-MaskFormer trained in S3 is used to detect the ice condition image, and the ice surface pixel points P are counted according to the model recognition results. i and the water surface pixel P w , calculate the ice surface coverage C i :

[0035]

[0036] Beneficial effects: The fast ACE algorithm in data preprocessing of the present invention can efficiently perform dehazing, denoising and enhancement on foggy images at night; a semi-supervised annotation method based on SAM is used in data annotation to replace traditional manual annotation, thereby improving the annotation efficiency and the accuracy and consistency of the annotation data; the TS-MaskFormer model proposed in ice image recognition upgrades semantic segmentation from pixel-level classification to mask classification, integrates the DETR model and Transformer architecture, and adds a fusion module. By fusing the overall segmentation of the first stage with the results of the separate segmentation of details in the second stage, a more accurate semantic segmentation result is obtained, while retaining more detail information; the ice surface coverage rate is more accurately calculated through the segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 is a flow chart in an embodiment of the present invention;

[0038] Figure 2 This is a comparison chart of the defogging effect using the fast ACE algorithm in an embodiment of the present invention;

[0039] Figure 3 is a flow chart of a semi-supervised annotation method based on SAM in an embodiment of the present invention;

[0040] Figure 4 This is a data annotation effect diagram in an embodiment of the present invention;

[0041] Figure 5 It is a structural diagram of the TS-MaskFormer model in an embodiment of the present invention;

[0042] Figure 6 It is a comparison diagram of the recognition effect before and after the model is improved in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0044] like Figure 1As shown, the ice condition video image recognition method based on TS-MaskFormer in this embodiment includes the following steps:

[0045] Step S1, collecting video data from a surveillance camera, capturing image frames from the read video and classifying them;

[0046] Step S1.1, collect video data from the monitoring camera of the target watershed, set the time interval threshold t, divide the read video by the threshold t, and capture the image frame;

[0047] Step S1.2, the captured images are divided into five categories according to time periods and ice surface scenes: ice cover, beginning of ice melting, drifting ice, end of ice melting, and night;

[0048] Step S2, preprocessing ice condition image frames, using a SAM-based semi-supervised annotation method to annotate the preprocessed images, and constructing an ice condition image dataset, including a training set, a validation set, and a test set;

[0049] Step S2.1, performing data cleaning on the five types of images obtained in step 1, and removing images with similar ice conditions;

[0050] Step S2.2: To solve the problem of foggy and blurred night images, a fast ACE algorithm is proposed to achieve defogging and enhancement of ice images. Figure 2 As shown in the figure, the image enhancement implementation process of the fast ACE algorithm is as follows:

[0051] I. Input a color image I, and the dehazing parameters include the contrast enhancement factor ration and the convolution template radius radius.

[0052] II. Create a zero matrix res of the same size as the input image to store the dehazing result.

[0053] III. Loop through the three RGB channels (red, green, blue).

[0054] IV. For each channel, use the single-channel ACE algorithm to perform fast enhanced dehazing. The specific recursive calling process is:

[0055] m. If the minimum value of the image height and width is less than or equal to 2, return a matrix with the same size as the input image, with all elements set to 0.5, and stop the recursion;

[0056] n. If the image size is greater than 2, the input image is reduced by half, that is, the width and height are divided by 2 respectively;

[0057] o. Recursively call the single-channel ACE algorithm on the reduced image, passing in the reduced image, defogging parameters ratio and radius;

[0058] p. Enlarge the result of the recursive call and restore the width and height to the size of the original image;

[0059] q. Restore the reduced image to the original size.

[0060] r. Calculate the final result by adding the result of the recursive call to the result of the dehazing process of the original image by the conventional ACE algorithm, and then subtracting the result of the dehazing process of the reduced image.

[0061] V. Perform linear stretching on the processed image to map the pixel values ​​of the image to the range of [0, 1] to enhance the contrast of the image.

[0062] VI. Store the processed channel image in the corresponding channel of the result matrix res to obtain the dehazing result.

[0063] Step S2.3: Label the deduplication enhanced image, divide the image pixels into three categories: background, water, and ice. Use a semi-supervised labeling method based on SAM to label the preprocessed data. The labeling effect is as follows: Figure 4 As shown in Figure 1, an ice condition image dataset is constructed, including a training set, a validation set, and a test set. The implementation process of the semi-supervised annotation method based on SAM is as follows: Figure 3 As shown, input the pre-processed ice condition image data, use the pre-trained SAM (Segment Anything) model for reasoning and annotation, and judge whether the annotation error is less than the threshold. If the error is greater than the threshold, manually input prompt to assist in loop correction. If the error is less than the threshold, the final annotation result is obtained.

[0064] Step S3, constructing a TS-MaskFormer model, using the constructed ice condition image detection and recognition data set to train the model, and obtaining an ice condition image detection and recognition model based on TS-MaskFormer;

[0065] Step S3.1: Build the TS-MaskFormer model. The model structure is as follows: Figure 5 As shown, it consists of three parts: the first and second stage MaskFormer and the fusion module;

[0066] Step S3.2, the MaskFormer model of each stage is composed of a pixel-level module, a transformer module and a segmentation module. The pixel-level module contains a backbone network and a pixeldecoder pixel-level decoder. The backbone of the first stage model is the Swin-Transformer backbones, and the backbone of the second stage model is the ResNet backbones. The image input passes through the backbone network to generate a low-resolution image feature F, and then passes through the pixel decoder pixel-level decoder to gradually upsample the feature to generate a pixel-by-pixel embedding of the binary mask prediction. The Transformer module is a standard Transformer decoder that calculates the output from the image feature F and N learnable position queries. The segmentation module first passes the result of the Transformer module through an MLP multi-layer perceptron, and then uses a linear classifier to obtain a probability prediction for each class; the result of the MLP is converted into a mask embedding of dimension C and then dot-producted with the result of the previous pixel-by-pixel module to obtain a binary mask prediction probability. The results of the category prediction and the mask prediction are matrix multiplied to obtain the final segmentation result;

[0067] Step S3.3, the two-stage model recognition results are fused through the fusion module to obtain the final semantic segmentation result. After fusion, the recognition results of the two-stage models can be combined to improve the recognition accuracy. The recognition results of the single Maskformer model and the improved TS-MaskFormer model are compared. Figure 6 The specific process of the fusion module to achieve the weighted fusion of the confidence of the segmentation results of the two-stage model is as follows:

[0068] I. Set the weights of Model 1 and Model 2 to W1 and W2 respectively.

[0069] II. Set the prediction accuracy of the model for the three categories on the validation set to the confidence [α1, α2, α3] and [β1, β2, β3] of Model 1 and Model 2 for the three categories.

[0070] III. Calculate the weighted average [P1, P2, P3] based on the confidence and weight. For each category, multiply the confidence of model 1 and model 2 by the corresponding weight, then add the two results and divide by the sum of the weights.

[0071]

[0072] IV. Based on the weighted average, find the index of the maximum value in the weighted average and determine the final pixel prediction category. The final prediction category is the category with the highest confidence.

[0073] Step S4: Use the TS-MaskFormer-based ice condition image detection and recognition model trained in S3 to detect the ice condition image, and count the ice surface pixel points P according to the model recognition results. i and the water surface pixel P w , calculate the ice surface coverage C i :

[0074] .

Claims

1. A method for ice condition video image recognition based on TS-MaskFormer, characterized by: The following steps are involved: Step S1, collecting video data from surveillance cameras, capturing video image frames and dividing them into 5 types; Step S2, preprocessing ice image frames, performing data cleaning on the five types of images obtained in step 1, removing images with similar ice conditions, and using the fast ACE algorithm to achieve defogging and enhancement of ice images to address the problem of foggy and blurred night images. Annotating the deduplication-enhanced images, dividing the image pixels into three categories: background, water, and ice. Using a semi-supervised annotation method based on SAM to annotate the preprocessed data, constructing an ice image dataset, and dividing it into a training set, a validation set, and a test set according to a ratio of 7:2:

1. Step S3, constructing a TS-MaskFormer model, which consists of three parts: the first and second stage MaskFormer and the fusion module. Using the constructed ice condition image detection and recognition dataset, the model is trained to obtain an ice condition image detection and recognition model based on TS-MaskFormer; Step S4: Use the TS-MaskFormer-based ice condition image detection and recognition model trained in S3 to detect the ice condition image and calculate the ice surface coverage.

2. The ice condition video image recognition method based on TS-MaskFormer according to claim 1 is characterized in that: The detailed process of step S1 is as follows: Step S1.1, collect video data from the monitoring camera of the target watershed, set the time interval threshold t, divide the read video by the threshold t, and capture the image frame; Step S1.2, the captured images are divided into five categories according to time periods and ice surface scenes: ice cover, beginning of ice melting, drifting ice, end of ice melting, and night.

3. The ice condition video image recognition method based on TS-MaskFormer according to claim 1 is characterized in that: The image enhancement implementation process of the fast ACE algorithm in step S2 is as follows: I. Input a color image I, and the dehazing parameters include the contrast enhancement factor ration and the convolution template radius radius. II. Create a zero matrix res of the same size as the input image to store the dehazing result. III. Loop through the three RGB channels (red, green, blue). IV. For each channel, use the single-channel ACE algorithm to perform fast enhanced dehazing. The specific recursive calling process is: a. If the minimum value of the image height and width is less than or equal to 2, return a matrix of the same size as the input image, with all elements set to 0.5, and stop the recursion; b. If the image size is greater than 2, the input image is reduced by half, that is, the width and height are divided by 2 respectively; c. Recursively call the single-channel ACE algorithm on the reduced image, passing in the reduced image, defogging parameters ratio and radius; d. Enlarge the result of the recursive call and restore the width and height to the size of the original image; e. Restore the reduced image to the size of the original image. f. Calculate the final result by adding the result of the recursive call to the result of the defogging of the original image by the conventional ACE algorithm, and then subtracting the result of the defogging of the reduced image. V. Perform linear stretching on the processed image to map the pixel values ​​of the image to the range of [0, 1] to enhance the contrast of the image. VI. Store the processed channel image in the corresponding channel of the result matrix res to obtain the dehazing result. The process of the semi-supervised labeling method based on SAM in step S2 is as follows: I. Input pre-processed ice condition image data and use the pre-trained SAM (Segment Anything) model for inference annotation II. Determine whether the annotation error is less than the threshold. If the error is greater than the threshold, manually enter the prompt to assist in the loop correction. If the error is less than the threshold, the final annotation result is obtained.

4. The ice condition video image recognition method based on TS-MaskFormer according to claim 1 is characterized in that: The TS-MaskFormer model in step S3 specifically includes three parts: the first and second stage MaskFormer and fusion module. The MaskFormer model of each stage is composed of a pixel-level module, a transformer module and a segmentation module. The pixel-level module includes a backbone network and a pixel decoder pixel-level decoder. The backbone of the first stage model is the Swin-Transformer backbones, and the backbone of the second stage model is the ResNet backbones. The image input passes through the backbone network to generate a low-resolution image feature F, and then the feature is gradually upsampled through the pixel decoder pixel-level decoder to generate a pixel-by-pixel embedding of the binary mask prediction. The Transformer module is a standard Transformer decoder that calculates the output from the image feature F and N learnable position queries. The segmentation module first passes the result of the Transformer module through an MLP multi-layer perceptron, and then uses a linear classifier to obtain a probability prediction for each class; the result of the MLP is converted into a mask embedding of dimension C and then dot-producted with the result of the previous pixel-by-pixel module to obtain a binary mask prediction probability. The results of category prediction and mask prediction are matrix multiplied to obtain the final segmentation result. The specific process of the fusion module in step S3 realizing the weight fusion of the confidence of the segmentation results of the two-stage model is as follows: I. Set the weights of Model 1 and Model 2 to W1 and W2 respectively. II. Set the prediction accuracy of the model for the three categories on the validation set to the confidence [α1, α2, α3] and [β1, β2, β3] of Model 1 and Model 2 for the three categories. III. Calculate the weighted average [P1, P2, P3] based on the confidence and weight. For each category, multiply the confidence of model 1 and model 2 by the corresponding weight, then add the two results and divide by the sum of the weights. IV. Based on the weighted average, find the index of the maximum value in the weighted average and determine the final pixel prediction category. The final prediction category is the category with the highest confidence.

5. The ice condition video image recognition method based on TS-MaskFormer according to claim 1 is characterized in that: The ice surface coverage calculation method in step S4 is: according to the model recognition result, the ice surface pixel points P are counted. i and the water surface pixel P w , calculate the ice surface coverage C i : 。

Images