An ice condition video image recognition method based on TS-MaskFormer

Abstract

The application discloses an ice condition video image recognition method based on TS-MaskFormer. A fast ACE defogging algorithm is used to enhance night images, and a semi-supervised labeling method based on SAM is used to construct an ice condition image detection and recognition dataset covering different stages such as ice sealing, ice melting starting and ice floating. A TS-MaskFormer ice condition image recognition model is proposed, which integrates a MaskFormer model of two different backbone networks and a fusion module, and adjusts and optimizes the model parameters through training to realize accurate ice condition image detection. Based on the optimized model, the ice condition image is recognized, and the ice coverage is calculated through the recognition result. The method can retain more detailed information in ice condition image recognition, has higher ice surface recognition accuracy and faster calculation speed.

Description

TECHNICAL FIELD

[0001] The present application relates to a TS-MaskFormer-based ice condition video image recognition method, belonging to the field of computer vision image recognition. BACKGROUND

[0002] In recent years, sea ice and river / lake ice have been listed as "essential climate variables (ECV)" by the Global Climate Observing System (GCOS). Long-term monitoring of the freezing time, thawing time, duration, and pattern of river / lake ice can reflect changes in local climate. In cold regions of northern China, river ice and frozen water are produced in rivers every year, leading to ice jams, also known as "ice rows", which are a hydrological phenomenon caused by the blocking of water flow by ice jams, resulting in a significant rise in water level. In river channels with ice on the surface, upstream river ice melts first, while downstream river channels remain unfrozen, causing ice jams to block and cause water levels to rise. Large areas of floating ice can collide with and cut through water structures such as dikes, bridges, and piers, potentially causing dikes to collapse and floods to occur. When an ice jam occurs, timely icebreaking measures must be taken. Therefore, timely and accurate monitoring of ice condition information changes can effectively guide ice prevention and prevent disasters caused by ice jams. In summary, ice condition monitoring has become an important research direction.

[0003] Current ice condition monitoring data mainly comes from remote sensing satellite images, laser radars or infrared sensors, and manual field measurements. However, these data acquisition methods usually require high hardware costs and cannot guarantee real-time data. There are few studies on ice condition monitoring based on video data. In 2018, Xiao [Xiao, M., Rothermel, M., Tom, M., Galliani, S., Baltsavias, E., Schindler, K., 2018. Lake ice monitoring with webcams [J]. ISPRS Annals of Photogrammetry, Remote Sensing and Spatial Information Sciences, IV-2.] first proposed a method for lake ice detection using webcams. By setting up webcams around the lake and using convolutional neural networks (CNN), the morphology and changes of lake ice can be monitored in real time.

[0004] There are mainly three methods for processing and analyzing the obtained ice condition data: the method based on spectral characteristics, which utilizes the differences in spectral characteristics such as absorption rate and refractive index between ice surface and surrounding environment. However, this method usually requires repeated experiments to determine the threshold for differentiation, and its generalization performance is poor. The method based on traditional machine learning, such as support vector machine (SVM), random forest, and decision tree, has insufficient learning ability and low accuracy in ice surface extraction. The method based on deep learning, such as convolutional neural network (CNN) and residual network (ResNet), has been paid more attention and achieved better results. However, there are few studies on ice condition video image recognition using deep learning. Zhang et al. [Zhang H, Liu Y, Li X. Ice concentration estimation using deep learning from SAR imagery [J]. IEEE Geoscience and Remote Sensing Letters. 2018; 15(2): 238-242.] proposed a method based on deep learning in 2019, which used synthetic aperture radar (SAR) images to estimate ice concentration. Convolutional neural network (CNN) was used for feature extraction and classification of SAR images, achieving accurate ice concentration estimation. However, the direct application of ordinary deep learning models in ice condition recognition scenarios is not ideal, and further improvement and optimization of the model are still needed. SUMMARY

[0005] The purpose of the present application is to solve the problems existing in the prior art and propose an ice condition video image recognition method based on TS-MaskFormer. The proposed method acquires data through monitoring video, has low hardware cost and high real-time performance, and proposes a TS-MaskFormer model that can recognize ice condition video images with high accuracy and calculate ice surface coverage based on the recognition results.

[0006] Technical solution: The present application is an ice condition video image recognition method based on TS-MaskFormer, comprising the following steps:

[0007] S1, collect the original video data of the monitoring camera, and classify the read video image frames;

[0008] S2, ice condition image frame preprocessing, using a semi-supervised labeling method based on SAM to label the preprocessed images, constructing an ice condition image dataset including training set, validation set and test set;

[0009] S3, construct a TS-MaskFormer model, use the constructed ice condition image detection and recognition dataset to train the model, and obtain an ice condition image detection and recognition model based on TS-MaskFormer;

[0010] The TS-MaskFormer model in the step S3 includes three parts of MaskFormer1, MaskFormer2 and fusion module; the MaskFormer1 is composed of a pixel-level module, a transformer module and a segmentation module, wherein the pixel-level module includes a Swin-Transformer backbones network and a pixel decoder; the MaskFormer2 is composed of a pixel-level module, a transformer module and a segmentation module, wherein the pixel-level module includes a ResNet backbones network and a pixel decoder;

[0011] The fusion module in the step S3 realizes weight fusion of the segmentation result confidence of the MaskFormer1 and the MaskFormer2, and obtains a fusion segmentation result;

[0012] S4, using the ice condition image detection and recognition model based on TS-MaskFormer trained in S3, detecting the ice condition image, and calculating the ice surface coverage (C i ) according to the recognition result.

[0013] Further, the implementation process of the step S1 is as follows: collecting video data of a monitoring camera, cutting the read video image frame, and dividing the image into five categories: ice sealing, ice melting starting, ice drift, ice melting ending and night.

[0014] Further, the implementation process of the step S2 is as follows: ice condition image frame preprocessing, data cleaning and deduplication of the five categories of images, and for the problem of fog blur of night images, a fast ACE algorithm is proposed to realize the dehazing enhancement of the ice condition image. The implementation process of the fast ACE algorithm is as follows:

[0015] I. Input a color image I, and dehazing parameters including contrast enhancement factor ration and convolution template radius radius.

[0016] II. Create a zero matrix res with the same size as the input image, which is used to store the dehazed result.

[0017] III. Loop through the RGB three channels (red, green, blue).

[0018] IV. For each channel, use the single-channel ACE algorithm for fast enhanced defogging processing. The specific recursive call process is:

[0019] g. If the minimum value of the height and width of the image is less than or equal to 2, return a matrix with the same size as the input image, all elements are 0.5, and stop recursion;

[0020] h. If the size of the image is greater than 2, reduce the input image by half, i.e. divide the width and height by 2 respectively;

[0021] i. Recursively call the single-channel ACE algorithm on the reduced image, passing in the reduced image, the defogging parameters ratio and radius;

[0022] j. Enlarge the result of recursive call, restore the width and height to the original image size;

[0023] k. Restore the reduced image to the original image size.

[0024] l. Calculate the final result, add the result of recursive call to the result of the regular ACE algorithm for defogging processing on the original image, and subtract the result of defogging processing on the reduced image.

[0025] V. Perform linear stretching on the processed image to map the pixel values of the image to the range of [0, 1] to enhance the contrast of the image.

[0026] VI. Store the processed channel image in the corresponding channel of the result matrix res to get the defogging result.

[0027] According to the problem definition, a semi-supervised labeling method based on SAM is used to label the preprocessed data, and the image pixels are divided into three categories: background (Background), water (Water) and ice surface (Ice), to construct an ice condition image dataset, including training set, validation set and test set. The implementation process of the semi-supervised labeling method based on SAM is shown in Figure 3 , input the preprocessed ice condition image data, use the pre-trained SAM (Segment Anything) model for inference labeling, judge whether the labeling error is less than the threshold, if the error is greater than the threshold, manually input prompt auxiliary loop correction, if the error is less than the threshold, get the final labeling result.

[0028] Further, the implementation process of the step S3 is as follows: a TS-MaskFormer model is constructed, the model is trained by using the constructed ice condition image detection and recognition dataset, and an ice condition image detection and recognition model based on the TS-MaskFormer is obtained; wherein the TS-MaskFormer model is composed of three parts of MaskFormer1, MaskFormer2 and a fusion module.

[0029] Further, in the step S3, the steps of the input image passing through the MaskFormer1 are as follows:

[0030] V. The input image passes through a Swin-Transformer backbones backbone network to generate a low-resolution image feature F1, and then passes through a pixel decoder to gradually up-sample the feature to generate a pixel-by-pixel embedding of the binary mask prediction;

[0031] VI. The transformer calculates the output from the image feature F1 and N learnable position queries

[0032] T1;

[0033] VII. The segmentation module first passes the result T1 of the transformer module through an MLP, and then uses a linear classifier to obtain the probability prediction of the pixel point belonging to the background, water body and ice surface, converts the result of the MLP into a mask embedding of dimension C, and then performs dot product with the result of the aforementioned pixel-by-pixel module to obtain the binary mask prediction probability;

[0034] VIII. The results of the category prediction and the mask prediction are multiplied to obtain the final segmentation result of the pixel point;

[0035] Further, in the step S3, the steps of the input image passing through the MaskFormer2 are as follows:

[0036] V. The input image passes through a ResNet backbones backbone network to generate a low-resolution image feature F2, and then passes through a pixel decoder to gradually up-sample the feature to generate a pixel-by-pixel embedding of the binary mask prediction;

[0037] VI. The transformer calculates the output from the image feature F2 and N learnable position queries

[0038] T2;

[0039] VII. The segmentation module first passes the result T2 of the transformer module through an MLP, and then uses a linear classifier to obtain the probability prediction of the pixel belonging to the background, water body and ice surface. The result of the MLP is converted into a mask embedding of dimension C, and then multiplied with the result of the aforementioned pixel-by-pixel module to obtain the binary mask prediction probability.

[0040] VIII. The results of the category prediction and the mask prediction are multiplied to obtain the final segmentation result of the pixel.

[0041] Further, the fusion module in step S3 realizes the weight fusion of the confidence of the two-stage model segmentation result, and the specific process is as follows:

[0042] I. Set the weights of model 1 and model 2 in the final fusion result as W1 and W2, respectively.

[0043] II. Calculate the prediction accuracy of MaskFormer1 and MaskFormer2 for any pixel belonging to the background (Background), water body (Water) and ice surface (Ice) on the validation set, respectively, and set it as the confidence of MaskFormer1 and MaskFormer2 for the three categories; further, set the confidence of MaskFormer1 for the background (Background), water body (Water) and ice surface (Ice) as α1, α2 and α3, respectively; set the confidence of MaskFormer2 for the background (Background), water body (Water) and ice surface (Ice) as β1, β2 and β3, respectively.

[0044] III. According to the confidence of MaskFormer1 and MaskFormer2 multiplied by the corresponding weight, then add the two results and divide by the sum of the weights, to calculate the weighted average of the prediction results of MaskFormer1 and MaskFormer2.

[0045] For the background (Background), the calculation formula of the weighted average P1 of the prediction results of MaskFormer1 and MaskFormer2 is:

[0046]

[0047] For the water body (Water), the calculation formula of the weighted average P2 of the prediction results of MaskFormer1 and MaskFormer2 is:

[0048]

[0049] For ice surface (Ice), the weighted average P3 of the prediction results of MaskFormer1 and MaskFormer2 is calculated according to the following formula:

[0050]

[0051] IV. Calculate the maximum value of the weighted average P1, P2, P3, find the index of the maximum value in the weighted average, and determine the final pixel prediction category; the final prediction category is the category with the highest confidence.

[0052] Further, the implementation process of the step S4 is as follows: using the TS-MaskFormer-based ice condition image detection and recognition model trained in S3 to detect the ice condition image, and according to the model recognition result, the ice surface pixel points PN i and the water surface pixel points PN w , the ice surface coverage C i is calculated:

[0053]

[0054] Beneficial effects: the fast ACE algorithm in data preprocessing of the present application can efficiently dehaze, denoise and enhance the night fog image; in data labeling, a semi-supervised labeling method based on SAM is used to replace the traditional manual labeling, improve the labeling efficiency and the accuracy and consistency of the labeled data; in the ice condition image recognition, the TS-MaskFormer model is proposed, which improves the semantic segmentation from pixel-level classification to mask classification, integrates the DETR model and the Transformer architecture, adds the fusion module, fuses the results of the overall segmentation in the first stage and the separate segmentation for details in the second stage, obtains more accurate semantic segmentation results, and retains more detailed information; the ice surface coverage is calculated through the more accurate segmentation results. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 is a flowchart in the embodiment of the present application;

[0056] Figure 2 is a comparison chart of dehazing effects using the fast ACE algorithm in the embodiment of the present application;

[0057] Figure 3 is a flowchart of a semi-supervised labeling method based on SAM in the embodiment of the present application;

[0058] Figure 4 is a data labeling effect diagram in the embodiment of the present application;

[0059] Figure 5is a TS-MaskFormer model structure diagram in an embodiment of the present application;

[0060] Figure 6 is a comparison chart of recognition effects before and after model improvement in an embodiment of the present application. DETAILED DESCRIPTION

[0061] The technical solutions of the present application will be described in detail below, but the protection scope of the present application is not limited to the described embodiments.

[0062] As shown in the figure, the ice condition video image recognition method based on TS-MaskFormer in the embodiment includes the following steps: Figure 1

[0063] Step S1, collect video data of the monitoring camera, and classify the read video image frames;

[0064] Step S1.1, collect video data of the target river basin monitoring camera, set a time interval threshold t, and divide the read video into image frames with the threshold t;

[0065] Step S1.2, divide the cut images into 5 categories according to time period and ice surface scene: ice sealing, ice melting start, ice drift, ice melting end, and night;

[0066] Step S2, ice condition image frame preprocessing, using a semi-supervised labeling method based on SAM to label the preprocessed images, and constructing an ice condition image dataset including training set, validation set and test set;

[0067] Step S2.1, data cleaning is performed on the 5 types of images obtained in step 1 to remove images with similar ice conditions;

[0068] Step S2.2, to solve the problem of night image fogging and unclearness, a fast ACE algorithm is proposed to realize the dehazing enhancement of ice condition images. The dehazing effect example is shown in the figure Figure 2 The image enhancement implementation process of the fast ACE algorithm is as follows:

[0069] I. Input a color image I, and dehazing parameters including contrast enhancement factor ration and convolution template radius radius.

[0070] II. Create a zero matrix res with the same size as the input image, which is used to store the dehazed results.

[0071] III. Loop through the RGB three channels (red, green, blue).

[0072] IV. For each channel, use the single-channel ACE algorithm for fast enhancement dehazing processing. The specific recursive call process is as follows: ​

[0073] m. If the minimum of the height and width of the image is less than or equal to 2, return a matrix of the same size as the input image with all elements being 0.5, and stop the recursion;

[0074] n. If the size of the image is greater than 2, reduce the input image by half, i.e., divide the width and height by 2 respectively;

[0075] o. Recursively call the single-channel ACE algorithm on the reduced image, passing in the reduced image, the dehazing parameters ratio and radius;

[0076] p. Enlarge the result of the recursive call to restore the width and height to the original image size;

[0077] q. Also restore the reduced image to the original image size.

[0078] r. Calculate the final result by adding the result of the recursive call to the result of the regular ACE algorithm dehazing the original image, and subtracting the result of dehazing the reduced image.

[0079] V. Perform linear stretching on the processed image to map the pixel values of the image to the range [0, 1] to enhance the contrast of the image.

[0080] VI. Store the processed channel image in the corresponding channel of the result matrix res to obtain the dehazing result.

[0081] Step S2.3, label the de-duplication enhanced image, divide the image pixels into three categories: background (Background), water (Water) and ice surface (Ice), use a semi-supervised labeling method based on SAM to label the preprocessed data, the labeling effect is as shown in Figure 4 , construct an ice condition image dataset including training set, validation set and test set. The implementation process of the semi-supervised labeling method based on SAM is as shown in Figure 3 , input the preprocessed ice condition image data, use the pre-trained SAM (Segment Anything) model for inference labeling, judge whether the labeling error is less than the threshold value, if the error is greater than the threshold value, manually input prompt auxiliary loop correction, if the error is less than the threshold value, get the final labeling result;

[0082] Step S3, construct a TS-MaskFormer model, use the constructed ice condition image detection and recognition dataset to train the model, and obtain an ice condition image detection and recognition model based on TS-MaskFormer;

[0083] Step S3.1, construct a TS-MaskFormer model, the model structure is as shown inFigure 5 As shown, MaskFormer1, MaskFormer2 and fusion module three parts; MaskFormer1 is composed of pixel-level module, transformer module and segmentation module, wherein the pixel-level module contains Swin-Transformer backbones and pixel decoder; MaskFormer2 is composed of pixel-level module, transformer module and segmentation module, wherein the pixel-level module contains ResNet backbones and pixel decoder;

[0084] Step S3.2, the input image is input into MaskFormer1, and the classification prediction result of each pixel point of the image is obtained:

[0085] I. The input image is input into the Swin-Transformer backbones to generate a low-resolution image feature F1, and then the pixel decoder is used for step-by-step upsampling of the feature to generate a binary mask prediction pixel-by-pixel embedding;

[0086] II. The transformer calculates the output T1 from the image feature F1 and N learnable position queries;

[0087] III. The segmentation module first passes the result T1 of the transformer module through an MLP, and then uses a linear classifier to obtain the probability prediction of the pixel point belonging to the background, water body and ice surface. The result of the MLP is converted into a mask embedding with a dimension of C, and then multiplied with the result of the aforementioned pixel-by-pixel module to obtain the binary mask prediction probability;

[0088] IV. The results of the category prediction and the mask prediction are multiplied to obtain the final segmentation result of the pixel point;

[0089] Step S3.3, the input image is input into MaskFormer2, and the classification prediction result of each pixel point of the image is obtained:

[0090] I. The input image is input into the ResNet backbones to generate a low-resolution image feature F2, and then the pixel decoder is used for step-by-step upsampling of the feature to generate a binary mask prediction pixel-by-pixel embedding;

[0091] II. The transformer computes the output T2 from the image features F2 and N learnable positional queries;

[0092] III. The segmentation module first passes the result T2 of the transformer module through an MLP, then uses a linear classifier to get the probability prediction of the pixel belonging to background, water, and ice surface. The result of the MLP is converted into a mask embedding of dimension C and then dot product with the result of the aforementioned pixel-wise module to get the binary mask prediction probability;

[0093] IV. The results of the category prediction and the mask prediction are multiplied to get the final segmentation result of the pixel;

[0094] Step S3.4, the two-stage model recognition results are fused by the fusion module to get the final semantic segmentation result. After fusion, the recognition accuracy can be improved by combining the recognition results of the two-stage model. The comparison of the instance recognition results of the single Maskformer model and the improved TS-MaskFormer model is shown in the table as follows: Figure 6 The fusion module realizes the weight fusion of the confidence of the two-stage model segmentation result, and the specific process is as follows:

[0095] I. Set the weights of model 1 and model 2 in the final fusion result as W1 and W2 respectively;

[0096] II. Calculate the prediction accuracy of MaskFormer1 and MaskFormer2 for any pixel belonging to background (Background), water (Water), and ice surface (Ice) on the validation set respectively, and set it as the confidence of MaskFormer1 and MaskFormer2 for the three categories; further, set the confidence of MaskFormer1 for background (Background), water (Water), and ice surface (Ice) as α1, α2, α3 respectively; set the confidence of MaskFormer2 for background (Background), water (Water), and ice surface (Ice) as β1, β2, β3 respectively;

[0097] III. According to the confidence of MaskFormer1 and MaskFormer2 multiplied by the corresponding weight, then add the two results and divide by the sum of the weights, to calculate the weighted average value of the prediction results of MaskFormer1 and MaskFormer2;

[0098] For the background (Background), the weighted average P1 of the prediction results of MaskFormer1 and MaskFormer2 is calculated according to the formula:

[0099]

[0100] For the water body (Water), the weighted average P2 of the prediction results of MaskFormer1 and MaskFormer2 is calculated according to the formula:

[0101]

[0102] For the ice surface (Ice), the weighted average P3 of the prediction results of MaskFormer1 and MaskFormer2 is calculated according to the formula:

[0103]

[0104] IV. Calculate the maximum value of the weighted average P1, P2, P3, find the index of the maximum value in the weighted average, and determine the final pixel prediction category; the final prediction category is the category with the highest confidence.

[0105] Step S4, using the TS-MaskFormer-based ice condition image detection and recognition model trained in S3 to detect the ice condition image, and according to the model recognition result, counting the ice surface pixel points PN i and the water surface pixel points PN w , calculating the ice surface coverage C i :

[0106]

Claims

1. A method for ice condition video image recognition based on TS-MaskFormer, characterized in that: Includes the following steps: Step S1: Collect video data from the surveillance camera, extract video image frames and divide them into 5 types; Step S2: Ice condition image frame preprocessing. Data cleaning is performed on the five types of images obtained in step S1 to remove images with similar ice conditions. To address the problem of foggy and blurry nighttime images, the fast ACE algorithm is used to enhance the ice condition images by removing fog. The enhanced images are then labeled, and the image pixels are divided into three categories: background, water, and ice. A semi-supervised labeling method based on SAM is used to label the preprocessed data, constructing an ice condition image dataset, and dividing it into training, validation, and test sets in a 7:2:1 ratio. Step S3: Construct the TS-MaskFormer model, which consists of three parts: MaskFormer1, MaskFormer2, and fusion module. Use the constructed ice condition image detection and recognition dataset to train the model and obtain the ice condition image detection and recognition model based on TS-MaskFormer. The TS-MaskFormer model in step S3 includes three parts: MaskFormer1, MaskFormer2, and a fusion module. MaskFormer1 consists of a pixel-level module, a transformer module, and a segmentation module. The pixel-level module contains Swin-Transformer backbones and a pixel decoder. MaskFormer2 consists of a pixel-level module, a transformer module, and a segmentation module. The pixel-level module contains ResNet backbones and a pixel decoder. In step S3, the fusion module performs weighted fusion of the confidence scores of MaskFormer1 and MaskFormer2 to obtain the fused segmentation result. Step S4: Use the TS-MaskFormer-based ice condition image detection and recognition model trained in S3 to detect ice conditions and calculate the ice surface coverage.

2. The ice condition video image recognition method based on TS-MaskFormer according to claim 1, characterized in that: The detailed process of step S1 is as follows: I. Collect video data from the monitoring cameras in the target watershed, set a time interval threshold t, and segment the read video images according to the threshold t to extract image frames; II. The captured images are divided into 5 categories based on time period and ice surface scene: frozen, beginning of ice melt, floating ice drifting, end of ice melt, and night.

3. The ice condition video image recognition method based on TS-MaskFormer according to claim 1, characterized in that, The image enhancement implementation process of the fast ACE algorithm in step S2 is as follows: I. Input a color image I, along with dehazing parameters, contrast enhancement factor Ration, and convolution template radius radius; II. Create a zero matrix Res with the same size as the input image to store the dehazed result; III. Loop through the RGB (red, green, blue) channels of the image; IV. For each channel, a single-channel ACE algorithm is used for fast and enhanced dehazing; the specific recursive call process is as follows: a. If the minimum value of the height and width of the image is less than or equal to 2, return a matrix of the same size as the input image, with all elements being 0.5, and stop the recursion; b. If the image size is greater than 2, then the input image is halved, that is, the width and height are divided by 2 respectively; c. Recursively call the single-channel ACE algorithm on the scaled-down image, passing in the scaled-down image, the dehazing parameter ratio, and the convolution template radius radius; d. Enlarge the result of the recursive call and restore the width and height to the original image size; e. Restore the scaled-down image to its original size; f. Calculate the final result, add the result of the recursive call to the result of the regular ACE algorithm for dehazing the original image, and then subtract the result of the dehazing of the scaled-down image; V. Perform linear stretching on the processed image to map the pixel values ​​of the image to the range of [0,1] to enhance the contrast of the image; VI. Store the processed channel images in the corresponding channels of the result matrix Res to obtain the dehazed image; The SAM-based semi-supervised annotation method in step S2 is as follows: I. Input preprocessed ice condition image data and use a pre-trained SAM model for inference and annotation; II. Determine if the annotation error is less than the threshold. If the error is greater than the threshold, manually input prompt to assist in loop correction. If the error is less than the threshold, obtain the final annotation result.

4. The ice condition video image recognition method based on TS-MaskFormer according to claim 1, characterized in that, In step S3, the input image passes through MaskFormer1 as follows: I. The input image is processed by the Swin-Transformer backbone network to generate a low-resolution image feature F1. Then, the feature is upsampled step by step by the pixel decoder to generate a pixel-by-pixel embedding for binary mask prediction. II. The transformer computes output T1 from image features F1 and N learnable location queries; III. The segmentation module first passes the result T1 from the transformer module through an MLP (Multilayer Perceptron), then uses a linear classifier to obtain the probability prediction of whether the pixel belongs to the background, water, or ice. The MLP result is then converted into a C-dimensional mask embedding and multiplied by the result of the aforementioned pixel-by-pixel module to obtain the binary mask prediction probability. IV. Perform matrix multiplication on the results of category prediction and mask prediction to obtain the final segmentation result for the pixel; In step S3, the input image passes through MaskFormer2 as follows: I. The input image is processed by ResNet backbones to generate a low-resolution image feature F2. Then, the feature is upsampled step by step by the pixel decoder to generate a pixel-by-pixel embedding for binary mask prediction. II. The transformer computes output T2 from image features F2 and N learnable location queries; III. The segmentation module first passes the result T2 from the transformer module through an MLP (Multilayer Perceptron), then uses a linear classifier to obtain the probability prediction that the pixel belongs to the background, water, or ice. The MLP result is then converted into a C-dimensional mask embedding and multiplied by the result of the aforementioned pixel-by-pixel module to obtain the binary mask prediction probability. IV. Perform matrix multiplication on the results of category prediction and mask prediction to obtain the final segmentation result for the pixel; In step S3, the fusion module implements the weighted fusion of the confidence scores of MaskFormer1 and MaskFormer2 segmentation results. The specific process is as follows: I. Set the weights of Model 1 and Model 2 in the final fusion result as W1 and W2, respectively; II. Calculate the prediction accuracy of MaskFormer1 and MaskFormer2 for any pixel belonging to Background, Water, and Ice on the validation set, and set these accuracy as the confidence scores of MaskFormer1 and MaskFormer2 for the three categories. Further, set the confidence scores of MaskFormer1 for Background, Water, and Ice to α1, α2, and α3, respectively; and set the confidence scores of MaskFormer2 for Background, Water, and Ice to β1, β2, and β3, respectively. III. Multiply the confidence scores of MaskFormer1 and MaskFormer2 by their corresponding weights, then add the two results together and divide by the sum of their weights to calculate the weighted average of the prediction results of MaskFormer1 and MaskFormer2. For the background, the weighted average P1 of the prediction results from MaskFormer1 and MaskFormer2 is calculated using the following formula: For water bodies, the weighted average P2 of the prediction results from MaskFormer1 and MaskFormer2 is calculated using the following formula: For ice, the weighted average P3 of the predictions from MaskFormer1 and MaskFormer2 is calculated using the following formula: IV. Calculate the maximum value of the weighted averages P1, P2, and P3, find the index of the maximum value in the weighted averages, and determine the final pixel prediction category; the final prediction category is the category with the highest confidence.

5. The ice condition video image recognition method based on TS-MaskFormer according to claim 1, characterized in that, The method for calculating the ice surface coverage rate in step S4 is as follows: based on the model recognition results, the number of ice surface pixels PN is calculated. i PN with water surface w Calculate the ice surface coverage C i : 。

Images