Remote sensing cloud and fog image interactive labeling method and system based on multi-algorithm superpixel segmentation

Abstract

The invention discloses a remote sensing cloud image interactive labeling method and system based on multi-algorithm superpixel segmentation, and the system comprises an image loading module, a multi-algorithm superpixel segmentation module, a region-level interactive labeling module, a dual-mode visual verification module, a modification tracking and quality evaluation module, a data export and consistency evaluation module and the like. According to the method, pixel-by-pixel strokes of cloud, fog and background are converted into one-key marking according to regions, and only manual modification regions are displayed by using label / original image overlapping display and a difference mask, so that the marking efficiency and boundary smoothness are remarkably improved, the auditing cost is reduced, and the marking consistency can be quantitatively evaluated and improved through an IoU index; the method is suitable for cloud and fog identification data set construction and related business application.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote sensing cloud image annotation, and particularly relates to a remote sensing cloud image interactive annotation method and system based on multi-algorithm superpixel segmentation. BACKGROUND

[0002] Cloud and fog remote sensing images have important application value in the fields of weather forecasting, shipping safety, aviation support, and marine monitoring. In recent years, cloud and fog recognition and sea fog detection algorithms based on deep learning have emerged in large numbers, and their performance is highly dependent on large-scale, high-quality labeled data sets.

[0003] Currently, in the cloud and fog remote sensing scene, the labeled data mainly relies on manual use of general image annotation tools, such as pixel-by-pixel painting, lasso, and polygon drawing. Such methods have the following significant problems:

[0004] 1. Clouds and fog are often large-scale continuous regions with complex boundaries, and are often intertwined with backgrounds, land, and sea. When using pixel-level painting tools, the annotator needs to repeatedly trace the cloud and fog boundary, and it often takes tens of minutes or even longer to complete a high-resolution image, which is difficult to support the production of large quantities of data sets, resulting in low annotation efficiency.

[0005] 2. The cloud and fog boundary itself has fuzziness and gradualness, and there is no "absolutely correct" boundary line. The annotation of the same image by different annotators or the same annotator at different times has large differences, and even if the same tool is used, there will be boundary shift, local missing label, and cloud and fog confusion, resulting in unstable label data, high noise, poor annotation consistency, and strong subjectivity.

[0006] 3. Many general annotation software only supports separate display of the original image or separate display of the label image, or simple transparent superposition, lacks a dual-mode display and transparency fine-tuning mechanism designed for cloud and fog remote sensing, and the annotator often needs to switch between multiple windows, which makes it difficult to compare the boundary position and cloud and fog texture in time, affects the judgment accuracy, and has the disadvantages of single display mode and difficulty in accurate judgment.

[0007] 4. In cloud and fog remote sensing images, clouds and fog often present a blocky, band-like or filamentous structure, which is naturally suitable for "region-level annotation". However, existing tools mostly rely on pixel-level brushes or polygon tools, and do not fully utilize superpixel segmentation and other image content information, making it impossible to achieve efficient annotation by "clicking a region to assign a value", and lacking a region-level annotation method for cloud and fog structure.

[0008] 5. In actual business operations, it is often necessary to first generate initial labels using algorithms, and then manually correct them. General labeling tools typically only provide limited undo functionality and cannot intuitively show "which areas were manually modified and by what percentage." Reviewers find it difficult to quickly focus on key areas, resulting in high quality control costs and a lack of systematic modification tracking and quality review methods.

[0009] 6. Existing superpixel / region segmentation methods are scattered and lack collaborative design for cloud and fog applications. Although there are region segmentation algorithms such as SLIC, Felzenszwalb, and QuickShift, most applications are concentrated in algorithm research or general image segmentation fields, lacking system design for cloud and fog remote sensing application scenarios: that is, how to utilize the advantages and disadvantages of different algorithms to achieve a balance between overall structure, cloud contours, and local boundary accuracy, and combine it with manual annotation processes.

[0010] In summary, existing technologies lack the ability to combine multi-algorithm superpixel segmentation, region-level click annotation, dual-mode display, and incremental modification tracking. Summary of the Invention

[0011] The purpose of this invention is to design an interactive annotation method and system for remote sensing cloud and fog images based on multi-algorithm superpixel segmentation, so as to solve the problems of low efficiency, poor consistency, uneven boundaries and difficulty in quality review in the annotation of cloud and fog remote sensing images.

[0012] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0013] The first objective of this invention is to propose an interactive annotation method for remote sensing cloud and fog images based on multi-algorithm superpixel segmentation, the method of which is as follows:

[0014] S1, Image loading step: The remote sensing cloud and fog image to be labeled is read from a preset directory into memory; it is determined whether there is a label image with the same name as the remote sensing cloud and fog image. If there is a color label image, the color label image is converted into an internal grayscale label matrix according to the preset color-category mapping relationship. In the grayscale label matrix, 0 represents the background, 1 represents fog, and 2 represents cloud. The grayscale label matrix is ​​saved as the current label matrix and the original label matrix, respectively. If there is no label image, a grayscale label matrix with the same size as the remote sensing cloud and fog image and the initial value of each element is 0 is constructed as the current label matrix and the original label matrix.

[0015] S2, a multi-algorithm region segmentation step, constructing a segmentation input image based on the remote sensing cloud and fog image to be labeled, receiving user selection of a segmentation algorithm, at least one of SLIC rule superpixel segmentation algorithm, Felzenszwalb graph cut segmentation algorithm and QuickShift edge adaptive segmentation algorithm being called, region segmentation being performed on the segmentation input image to obtain a region number matrix consistent with the image size, each pixel position in the region number matrix corresponding to a region number, and pixels with the same region number constituting a superpixel region that can be labeled as a whole;

[0016] S3, a region-level interactive labeling step, displaying the remote sensing cloud and fog image and its corresponding region boundary in a graphical user interface, obtaining pixel coordinates of a clicked position in response to a user click operation on the image, obtaining a region number corresponding to the coordinates according to the region number matrix, constructing a pixel mask of the region, and updating pixel values of the mask position in the current label matrix to corresponding class encodings according to the currently selected class label, while feeding back the updated label result to the interface display in real time;

[0017] S4, a dual-mode display and labeling verification step, providing two views of a label display mode and an original image overlay mode in the graphical user interface, displaying different classes in the current label matrix in different colors in the label display mode, and optionally superimposing the region boundary; in the original image overlay mode, a pseudo-color image corresponding to the current label matrix and the remote sensing cloud and fog image are linearly mixed through an adjustable transparency parameter, and the mixed result is displayed on the interface, so that the user can compare and verify between the original texture and the label boundary;

[0018] S5, an incremental modification tracking step, comparing the original label matrix with the current label matrix pixel by pixel to generate a difference mask matrix, highlighting pixels marked as modified in the difference mask matrix, and counting the number of modified pixels and the proportion of the number of modified pixels in the total number of pixels of the image, for visual display of the modification range and labeling quality review;

[0019] S6, label export and consistency verification, after labeling is completed, exporting the current label matrix as a gray label image file, and generating a color visual label image file according to a preset color-class mapping relationship, while optionally exporting a difference mask image and a modification statistical information file for use in deep learning model training and manual review.

[0020] Further, the conversion of the color label image to the gray label matrix in S1 includes:

[0021] S11, reading a color label image corresponding to the image to be labeled in the same name or according to a preset naming rule;

[0022] S12, traversing the color label image pixel by pixel, matching the color value of each pixel with the preset category color table, setting the gray label value of the corresponding position to 0 when the color matches the background color, setting the gray label value of the corresponding position to 1 when the color matches the fog category color, and setting the gray label value of the corresponding position to 2 when the color matches the cloud category color;

[0023] S13, copying the converted gray label matrix to obtain a current label matrix and an original label matrix, respectively, for subsequent incremental modification tracking.

[0024] Further, the multi-algorithm region segmentation step in S2 further comprises:

[0025] S21, receiving the selected segmentation algorithm type and corresponding algorithm parameters in the interface by the user, wherein the algorithm type at least includes: the superpixel number and compactness parameter corresponding to the SLIC algorithm, the scale parameter, the smoothing parameter and the minimum region size parameter corresponding to the Felzenszwalb algorithm, and the kernel size, maximum distance and proportion parameter corresponding to the QuickShift algorithm;

[0026] S22, when the selected algorithm is SLIC, performing iterative clustering on the segmentation input image based on the joint measurement of color and spatial distance, and dividing the image into regular shape and uniform size superpixel regions; when the selected algorithm is Felzenszwalb, constructing a graph structure and performing region merging according to the weight between pixels to obtain irregular shape and close to cloud contour segmentation regions; when the selected algorithm is QuickShift, local clustering is performed based on color density and spatial distribution to obtain edge change sensitive fine-grained regions;

[0027] S23, mapping the region marking results obtained by any of the above algorithms into a region number matrix, and storing it as the region index of the current image for subsequent region-level interactive labeling.

[0028] Further, when SLIC segmentation is used in S2, the SLIC algorithm in S22 further comprises:

[0029] Determine the neighborhood pixel set for each initial clustering center in the feature space, calculate the distance between the pixels in the neighborhood and the clustering center in the color component and the spatial coordinates, weight the spatial distance in the distance measurement according to the compactness parameter, and iteratively update the clustering center until the clustering label converges, thereby obtaining a regular grid-shaped superpixel division suitable for large-scale sea fog and stratiform cloud scenes, to improve the uniformity of the initial region division and the labeling efficiency.

[0030] Further, when the Felzenszwalb segmentation is used in the S2, the Felzenszwalb algorithm in the S22 further comprises:

[0031] The edge weight is determined according to the brightness difference or texture difference between adjacent pixels in the image, a weighted undirected graph is constructed, and under the condition that the difference within a region is less than a given threshold and the difference between regions is greater than a given threshold, a region merging strategy based on a minimum spanning tree is used to obtain a plurality of irregular polygon regions, so that the contours of meteorological structures such as clouds and cloud bands are more naturally expressed in the region division.

[0032] Further, when the QuickShift segmentation is used in the S2, the QuickShift algorithm in the S22 further comprises:

[0033] The local density of each pixel is estimated in the feature space, the offset vector of the pixel to the high-density neighbor is calculated based on the kernel function, and the pixel is merged into the corresponding mode point along the direction of the density gradient, and finally the fine-grained region division is formed, which is close to the cloud-sea boundary, small-scale cloud filaments and fog tongue-shaped structures, so as to improve the fineness of the local boundary.

[0034] Further, the region-level interactive labeling step of the S3 further comprises:

[0035] Before each execution of the region label update, the current label matrix is saved to the history stack, when the user triggers the undo instruction, the previous version of the label matrix is restored from the history stack, when the user triggers the redo instruction, the next version of the label matrix is restored from the history stack, to support multi-step undo and redo; when it is detected that the user repeatedly labels the same superpixel region, a confirmation prompt is popped up to the user, so as to reduce the influence of misoperation on the label result.

[0036] Further, the incremental modification tracking step in the S5 comprises:

[0037] The original label matrix and the current label matrix are compared at each pixel position, when the label values of the two are different, the corresponding position in the difference mask matrix is marked as a modified pixel, otherwise it is marked as an unmodified pixel; the number of modified pixels is counted according to the difference mask matrix, and the modification ratio is obtained by dividing the total number of pixels in the image; when the interface is displayed, only the regions marked as modified are highlighted or displayed separately, so that the labeling auditing personnel can focus on the manually modified regions for quality inspection.

[0038] Further, it further comprises a consistency verification step:

[0039] For the same remote sensing cloud and fog image, at least two control label masks are obtained by using a traditional pixel-by-pixel labeling method, and at least two label masks generated by the method of the application are obtained by using the interactive labeling method based on multi-algorithm superpixel segmentation, the intersection over union index IoU between the label masks is calculated for the fog and cloud, respectively, and the IoU value difference between the traditional method and the method of the application is compared to verify the improvement effect of the application in labeling consistency.

[0040] A second object of the application is to provide an interactive labeling system for remote sensing cloud and fog images based on multi-algorithm superpixel segmentation, comprising:

[0041] An image loading module for loading an original cloud and fog remote sensing image;

[0042] A multi-algorithm superpixel segmentation module for selecting and switching between SLIC rule superpixel segmentation algorithm, Felzenszwalb graph cut segmentation algorithm and QuickShift edge adaptive segmentation algorithm, performing superpixel or region segmentation on the current cloud and fog remote sensing image, and generating a region number matrix for region-level click labeling;

[0043] A region-level interactive labeling module for responding to user click operations, performing one-key labeling on the whole region according to the region number matrix, and supporting undo and repeat labeling confirmation;

[0044] A dual-mode visual verification module for switching between a label display mode and an original cloud and fog remote sensing image superposition mode, judging the cloud, fog and background boundary positions, and completing detailed correction of the labeling results;

[0045] A modification tracking and quality evaluation module for calculating and visualizing the difference between the original label and the current label;

[0046] A data export and consistency evaluation module for exporting the current label results and related statistical information to an external file, and evaluating the consistency of different labeling results.

[0047] The above technical solutions can achieve the following beneficial effects:

[0048] Compared with the existing general labeling tool, the application has obvious comprehensive advantages in superpixel segmentation, interactive labeling method, display verification mechanism and engineering process design. The application combines SLIC, Felzenszwalb, QuickShift and other superpixel / region segmentation algorithms to realize multi-scale collaborative segmentation from overall structure to local boundary: SLIC is used to construct a global uniform basic grid to quickly divide large-scale sea fog and stratiform clouds; Felzenszwalb graph cut segmentation is used to adaptively fit irregular cloud contour to avoid the "block type" truncation of cloud boundary by regular grid; QuickShift edge adaptive segmentation is used to refine the details of cloud-sea interface, small-scale cloud filament and fog tongue, thereby forming a hierarchical collaborative segmentation process from "coarse to fine" and "overall to local". The existing general labeling tool usually only provides a single superpixel algorithm, which is difficult to realize the above-mentioned joint action for cloud and fog morphology.

[0049] In terms of interaction, the application adopts a click-type labeling method based on superpixel regions, which improves the labeling basic unit from "pixel" to "superpixel region". The labeling personnel only need to select the category and click the corresponding region on the image to complete the assignment of the entire block region at one time, and avoid large-area misoperation through multi-step undo, repeated labeling confirmation and other mechanisms. Compared with the traditional polygon edge or brush smearing pixel-by-pixel labeling method, the application greatly reduces the number of labeling actions and reduces the boundary jitter phenomenon, significantly improves the labeling efficiency, and naturally obtains smoother and more consistent label boundaries.

[0050] In terms of display and verification, the application introduces a double-mode display mechanism of "label mode + original image superposition mode", and separately controls the label transparency, boundary line transparency and label transparency under superposition mode. Compared with the existing tool which only provides simple "original image + mask" superposition and can only adjust a single transparency parameter, the application can flexibly balance between cloud and fog texture, label color and superpixel boundary, so that the labeling personnel can more accurately judge the cloud-fog-background boundary position when browsing globally and refining locally. In addition, the system supports one-key export of grayscale label map (for model training) and color visualized label map (for manual inspection and display), which is convenient for subsequent processing and display.

[0051] In terms of labeling process and quality control, the application constructs an engineered labeling process for batch cloud and fog samples, supports continuous labeling of thousands of cloud and fog images through engineering directory management and "save and next" functions, avoids the inefficient operation of importing images and repeatedly selecting the next image from the folder in the existing tools. At the same time, the system only highlights and counts the area of the region that changes relative to the initial label, so that the auditing personnel can focus on the incremental modification area to carry out spot checks, significantly reducing the quality auditing cost. Under the unified computing environment and directory structure, the same IoU calculation program is used to compare the labels generated by the tool of the application and the labels generated by the traditional labeling tool. The results show that while the labeling time is significantly reduced, the intersection over union (IoU) index of cloud and fog classes is improved, further proving the comprehensive advantages of the application in labeling efficiency, boundary smoothness and labeling consistency. BRIEF DESCRIPTION OF DRAWINGS

[0052] Figure 1 Fig. 1 is a schematic diagram of the overall architecture of the system of the application.

[0053] Figure 2 Fig. 2 is a schematic diagram of SLIC (regular grid) segmentation results.

[0054] Figure 3 Fig. 3 is a schematic diagram of Felzenszwalb (irregular polygon) segmentation results.

[0055] Figure 4 Fig. 4 is a schematic diagram of QuickShift (edge adaptive) segmentation results.

[0056] Figure 5 Fig. 5 is an interface schematic diagram of the label display mode.

[0057] Figure 6 Fig. 6 is a display schematic diagram of the original image overlay mode.

[0058] Figure 7 Fig. 7 is a schematic diagram of the color visualized label generated by the application.

[0059] Figure 8 Fig. 8 is a labeling flowchart of the system of the application. DETAILED DESCRIPTION

[0060] The application will be further described below in conjunction with the drawings:

[0061] As Figures 1-8As shown, an interactive labeling method and system for remote sensing cloud and fog images based on multi-algorithm superpixel segmentation, the system includes image loading, multi-algorithm superpixel segmentation, region-level interactive labeling, dual-mode visual verification, modification tracking and quality evaluation, data export and consistency evaluation, etc. modules. Through the cooperation of SLIC, Felzenszwalb and QuickShift segmentation, the pixel-by-pixel edge description of clouds, fog and background is converted into one-key labeling by region, and the label / original image overlay display and difference mask display of artificial modification area are used to significantly improve the labeling efficiency and boundary smoothness, reduce the review cost, and can be quantitatively evaluated and improved by IoU index. The consistency of labeling is suitable for cloud and fog recognition dataset construction and related business applications.

[0062] Embodiment 1: An interactive labeling system for remote sensing cloud and fog images based on multi-algorithm superpixel segmentation, comprising an image loading module, a multi-algorithm superpixel segmentation module, a region-level interactive labeling module, a dual-mode visual verification module, a modification tracking and quality evaluation module, and a data export and consistency evaluation module connected in turn. Each module interacts with data through original image data, label matrix and superpixel region number matrix in memory, forming a complete labeling process from "image reading → region segmentation → interactive labeling → visual verification → modification tracking → result export and consistency evaluation".

[0063] The image loading module is used to load the original cloud and fog remote sensing images. Specifically, the image loading module is used to complete the engineering-level management of cloud and fog remote sensing images and the reading of single image data, and is responsible for associating the existing label file, providing basic data support for subsequent multi-algorithm superpixel segmentation and interactive labeling.

[0064] In a preferred embodiment, the image loading module first reads the engineering configuration to obtain information such as the original image directory, the label directory and the output directory. If not preset, the user is allowed to manually specify the directory path through the interface. The image loading module scans the remote sensing image files that meet the preset suffix (such as ".png", ".jpg", etc.) under the original image directory, constructs a sample list according to the file name or time sequence, and records the original image path, the expected corresponding label image path and the labeling status of each sample in the memory.

[0065] When the user selects an image through the interface or clicks the "previous / next" button, the image loading module locates the target image from the sample list, calls the image reading interface to load the image into the memory as the original cloud and fog image to be labeled. Preferably, the original image can be a still meteorological satellite visible light composite image, an infrared brightness temperature pseudo-color image or other preprocessed cloud and fog image.

[0066] The multi-algorithm superpixel segmentation module is configured to select and switch among a SLIC regular superpixel segmentation algorithm, a Felzenszwalb graph cut segmentation algorithm, and a QuickShift edge adaptive segmentation algorithm, to perform superpixel or region segmentation on a current cloud and fog remote sensing image, and to generate a region numbering matrix for region-level click annotation.

[0067] In the preferred embodiment of the present application, the multi-algorithm superpixel segmentation module receives the original image data output by the image loading module as a segmentation input image. The user can select at least one of the SLIC regular superpixel segmentation algorithm, the Felzenszwalb graph cut segmentation algorithm, and the QuickShift edge adaptive segmentation algorithm through an algorithm selection control on the interface, and can set or adopt default algorithm parameters.

[0068] When the SLIC algorithm is selected, the multi-algorithm superpixel segmentation module performs iterative clustering on the image based on color-space joint features, aggregates pixels into superpixel regions with regular shapes and uniform sizes, and is suitable for fast blocking of large-scale sea fog and stratiform cloud scenes. In the region numbering matrix output by the multi-algorithm superpixel segmentation module, pixels with the same number constitute a regular grid-shaped superpixel region, as shown in FIG. 3. Figure 2

[0069] When the Felzenszwalb algorithm is selected, the multi-algorithm superpixel segmentation module regards the image as a weighted undirected graph, constructs edge weights based on the brightness difference and texture difference between pixels, and performs region merging through a graph cut strategy based on the minimum spanning tree, so that the segmentation result is closer to the shape of natural cloud system contours such as convective cloud clusters and cloud bands, as shown in FIG. 4. Figure 3

[0070] When the QuickShift algorithm is selected, the multi-algorithm superpixel segmentation module performs density estimation in the color-space feature space, clusters pixels along the density gradient direction, and merges pixels into local mode points, to obtain a fine-grained region division that is more sensitive to edge changes, which is suitable for local fine segmentation of cloud-sea junctions, small-scale cloud filaments, and fog tongue-shaped structures, as shown in FIG. 5. Figure 4

[0071] The multi-algorithm superpixel segmentation module uniformly encodes the segmentation results obtained by different algorithms into a region numbering matrix superpixels, and further calculates region boundary information for superimposed display. The multi-algorithm superpixel segmentation module allows the user to replace the segmentation algorithm or adjust the parameters multiple times on the same image, to regenerate superpixels to adapt to the annotation requirements of different cloud and fog morphologies.​​​

[0072] The region-level interactive labeling module is configured to perform one-key labeling on the whole region according to the region numbering matrix in response to a user click operation, and support undo and repeat labeling confirmation; and is configured to receive a user click or other interactive operation on the image, and in combination with the region numbering matrix generated by the multi-algorithm superpixel segmentation module, to realize one-key labeling on the whole superpixel region, and provide functions of preventing misoperation such as multi-step undo and repeat labeling confirmation.

[0073] In an embodiment of the present application, the region-level interactive labeling module synchronously displays the original image, the current label image and the region boundary superposition result on the interface. When the user clicks a point in any view, the region-level interactive labeling module captures the pixel coordinates (x, y) of the clicked position, reads the region number R of the position from the region numbering matrix superpixels, and regards the pixel set of all superpixels == R as a region to be operated.

[0074] The region-level interactive labeling module performs uniform assignment of all pixels in the region in the current label matrix current_label according to the currently selected category (background, fog or cloud), and updates the label value to the corresponding category code, so as to realize "one-click labeling of the whole region", and significantly reduce the amount of pixel-by-pixel edge drawing operation.

[0075] In order to reduce the risk of misoperation, the region-level interactive labeling module presses a copy of the current label matrix current_label into the undo stack before performing region label updating, to support multi-step undo and redo. When the user triggers the undo instruction, the region-level interactive labeling module restores the previous version of the label data from the undo stack, and when the user triggers the redo instruction, the next version of the label data is restored.

[0076] In addition, when the region-level interactive labeling module detects that the user performs repeated assignment or switches from one category to another on the same superpixel region, a confirmation prompt window can be popped up to prompt the user "whether to overwrite the original label of the region", and only when the user confirms, the overwriting operation is performed, so as to avoid mismodification of a large area.

[0077] The dual-mode visual verification module is configured to switch between a label display mode and an original cloud and fog remote sensing image superposition mode, to judge the cloud, fog and background boundary positions, and to complete detailed correction of the labeling result. Specifically, the dual-mode visual verification module is configured to provide two view display modes of the label display mode and the original image superposition mode, and allow the user to adjust the mixing transparency of the label and the original image, so as to compare the cloud and fog texture and the labeling boundary in the same interface, and improve the accuracy of labeling verification.

[0078] In label display mode, the dual-mode visual verification module maps the three category values ​​(0, 1, and 2) in the current label matrix (current_label) to three preset colors: background, fog, and cloud, generating a color label image. Optionally, the region boundaries output by the multi-algorithm superpixel segmentation module are superimposed on the label image to form a pattern such as... Figure 5 The label view shown makes it easy to quickly check the overall labeling situation.

[0079] In the original image overlay mode, the dual-mode visual verification module linearly blends the color label image with the original cloud and fog image provided by the image loading module according to the transparency parameter α to obtain the overlay image, such as... Figure 6 As shown. Preferably, the transparency parameter α ranges from [0,1]. When α is close to 0, the original cloud and fog texture is mainly displayed; when α is close to 1, the label result is mainly displayed. Users can adjust the value of α in real time using a slider. The dual-mode visual verification module 140 supports one-click switching between the two modes and keeps the zoom, pan, and other states of the view synchronized, allowing users to compare and observe the effects of the same area under different display modes, thereby more accurately judging the positions of clouds, fog, and background boundaries, and completing detailed correction of the annotation results.

[0080] The modification tracking and quality assessment module is used to calculate and visualize the differences between the original label and the current label. Specifically, it compares the differences between the initial label and the current label, generates a difference mask for the modified area, and statistically analyzes the modification ratio and category conversion to provide quantitative basis for quality review. In a preferred embodiment, the modification tracking and quality assessment module always holds both the original label matrix `original_label` and the current label matrix `current_label` during label initialization and subsequent labeling. At any given time, the modification tracking and quality assessment module can perform a pixel-by-pixel comparison between the two: when the label values ​​at a certain position are not equal, the corresponding position in the difference mask `diff_mask` is marked as 1, indicating that the pixel has been manually modified; when the label values ​​are equal, the corresponding position in the `diff_mask` is marked as 0.

[0081] The modification tracking and quality assessment module can visualize the difference mask (diff_mask) and overlay it onto the original image or label image in the interface, highlighting only the modified pixels or areas, thus enabling reviewers to quickly locate "which areas have been changed relative to the initial label".

[0082] In addition, the modification tracking and quality evaluation module accumulates the number of pixels in the difference mask with a value of 1 to obtain a total number of modified pixels M, and calculates a modification ratio R by a ratio of the total number of modified pixels M to the total number of pixels in the image, which is used to reflect the overall intensity of the current sample label modification. The modification tracking and quality evaluation module can further statistically analyze the conversion relationship between the original class and the current class, such as background->cloud, background->fog, fog->cloud, etc., to provide more detailed statistical information for subsequent quality control and project evaluation.

[0083] The data export and consistency evaluation module is configured to export the current label result and related statistical information to an external file, and to perform consistency evaluation on different annotation results. Specifically, the data export and consistency evaluation module is configured to export the current label result and related statistical information to an external file, and to perform consistency evaluation on different annotation results when needed, such as calculating the intersection over union (IoU) index of the cloud class and the fog class. In the embodiment of the present application, the data export and consistency evaluation module 160 exports the current label matrix current_label in grayscale form, retaining the class codes such as 0, 1, 2, for training and inference of a deep learning model; at the same time, the current_label is rendered into a color label image according to a preset color mapping, and exported as an image file, which is convenient for manual browsing and result display, as shown in FIG. 6. For projects that need to carry out quality tracking, the data export and consistency evaluation module 160 can also export the difference mask diff_mask generated by the modification tracking and quality evaluation module 150, as well as the modification ratio and class conversion statistics, to an image or text file, which is convenient for subsequent archiving and analysis. Figure 7

[0084] When performing annotation consistency evaluation, the data export and consistency evaluation module 160 can compare and analyze multiple independent annotation results of the same cloud and fog remote sensing image. Specifically, for the cloud class and the fog class, the pixel sets A and B of the corresponding classes in the two annotation results can be constructed respectively, the intersection over union IoU = |A∩B| / |A∪B| can be calculated, and the difference in the IoU index between the label generated by the system of the present application and the label generated by a traditional pixel-by-pixel annotation tool can be compared, so as to quantitatively evaluate the effect of the present application in improving annotation consistency.

[0085] Through the cooperative work of the above-mentioned modules, the system of the present application can realize efficient region-level annotation, double-mode visual verification, modification incremental visualization, and result export and consistency evaluation of cloud and fog remote sensing images, which has significant advantages in annotation efficiency, boundary smoothness, annotation consistency, and review cost compared with the prior art.

[0086] Annotation consistency IoU evaluation embodiment

[0087] ​In the embodiment of the application, to verify the effect of the multi-algorithm superpixel interactive labeling system in labeling consistency, the following labeling consistency evaluation experiment can be carried out in actual application. The specific numerical values in this embodiment are only illustrative, and a person skilled in the art can replace them according to actual data.

[0088] Firstly, a control labeling scheme is constructed.

[0089] Typical cloud and fog remote sensing image samples are selected, including large-scale sea fog scenes, stratiform cloud and convective cloud mixed scenes, and cloud-sea and cloud-land boundary scenes. For each image, two labeling schemes are used for labeling respectively:

[0090] Traditional pixel-by-pixel labeling scheme: using a general labeling software, a polygon or a brush tool is used to draw along the cloud and fog boundary pixel by pixel, and labels of three types of clouds, fogs and backgrounds are completed.

[0091] Superpixel interactive labeling scheme of the application: using the system of the application, on the basis of multi-algorithm superpixel segmentation, the same three types of labels are completed through region-level click labeling.

[0092] Secondly, multiple independent labeling is carried out under the same scheme.

[0093] To eliminate accidental factors, two independent labelings of the same image are carried out by the same labeling personnel with cloud and fog business experience at different times, and corresponding mask png images are obtained.

[0094] Thirdly, the intersection over union (IoU) of the cloud class and the fog class is calculated. For each type of target, the cloud class pixel set is extracted from the mask, and the IoU values under the traditional scheme and the scheme of the application are calculated respectively.

[0095] For example, for the cloud class, two samples are selected. The cloud class pixel sets A1, A2 and B1, B2 are extracted from the mask, and the IoU values under the traditional scheme and the scheme of the application are calculated respectively.

[0096] Traditional: IoU_cloud = |A1 ∩ A2| / |A1 ∪ A2|

[0097] The application: IoU_cloud = |B1 ∩ B2| / |B1 ∪ B2|

[0098] |A ∩ B| represents the number of pixels of the intersection of region A and region B

[0099] |A ∪ B| represents the area pixel number of the union of region A and region B

[0100] Step 4: result comparison and analysis. If the experimental results show that, in most samples, the cloud and fog IoU values obtained by the scheme of the application are significantly higher than those of the traditional pixel-by-pixel labeling scheme, it means that the multi-algorithm superpixel interactive labeling method of the application effectively reduces the subjective differences of labelers at different times for the same image, and improves the consistency and stability of the labeling results.

[0101] The calculated traditional scheme: fog (class 1) IoU ≈ 0.2387; cloud (class 2) IoU ≈ 0.1430.

[0102] The calculated application: fog (class 1) IoU ≈ 0.6598; cloud (class 2) IoU ≈ 0.8163.

[0103] Based on the experiment, randomly select multiple samples for miou (average iou of all samples) comparison, the application significantly improves the labeling consistency, and verifies the technical effect of the application in reducing artificial subjective error and improving label quality.

[0104] In addition, the IoU between different labeling personnel can also be compared to evaluate the effect of the application system in reducing the "difference between people". Through the above consistency evaluation embodiment, it can be further proved that the application has obvious advantages in cloud and fog remote sensing label quality compared with the traditional labeling method

[0105] Embodiment 2: a remote sensing cloud and fog image interactive labeling method based on multi-algorithm superpixel segmentation, the method is as follows:

[0106] S1, image loading step, read the remote sensing cloud and fog image to be labeled from the preset directory into the memory; judge whether there is a label image with the same name as the remote sensing cloud and fog image, if there is a color label image, convert the color label image to an internal gray label matrix according to the preset color-class mapping relationship, the gray label matrix is represented by 0 for background, 1 for fog, and 2 for cloud, and save the gray label matrix as the current label matrix and the original label matrix, if there is no label image, construct a gray label matrix with the same size as the remote sensing cloud and fog image, and the initial value of the element is 0 as the current label matrix and the original label matrix;

[0107] S2, a multi-algorithm region segmentation step, constructing a segmentation input image based on the remote sensing cloud and fog image to be labeled, receiving user selection of a segmentation algorithm, at least one of SLIC rule superpixel segmentation algorithm, Felzenszwalb graph cut segmentation algorithm and QuickShift edge adaptive segmentation algorithm being called, region segmentation being performed on the segmentation input image to obtain a region number matrix consistent with the image size, each pixel position in the region number matrix corresponding to a region number, and pixels with the same region number constituting a superpixel region that can be labeled as a whole;

[0108] S3, a region-level interactive labeling step, displaying the remote sensing cloud and fog image and its corresponding region boundary in a graphical user interface, obtaining pixel coordinates of a clicked position in response to a user click operation on the image, obtaining a region number corresponding to the coordinates according to the region number matrix, constructing a pixel mask of the region, and updating pixel values of the mask position in the current label matrix to corresponding class encodings according to a currently selected class label, while feeding back the updated label result to the interface display in real time;

[0109] S4, a dual-mode display and labeling verification step, providing two views of a label display mode and an original image overlay mode in the graphical user interface, displaying different classes in the current label matrix in different colors in the label display mode, and optionally superimposing the region boundary; in the original image overlay mode, performing linear mixing of a pseudo-color image corresponding to the current label matrix and the remote sensing cloud and fog image through an adjustable transparency parameter, and displaying the mixing result on the interface, so that the user can compare and verify between the original texture and the label boundary;

[0110] S5, an incremental modification tracking step, performing pixel-by-pixel comparison between the original label matrix and the current label matrix to generate a difference mask matrix, highlighting pixels marked as modified in the difference mask matrix, and counting the number of modified pixels and the proportion of the number of modified pixels in the total number of pixels of the image, for visual display of the modification range and labeling quality review;

[0111] S6, label export and consistency verification, after labeling is completed, exporting the current label matrix as a gray label image file, and generating a color visual label image file according to a preset color-class mapping relationship, while optionally exporting a difference mask image and a modification statistical information file for use in deep learning model training and manual review.

[0112] The specific method is as follows:

[0113] S1: image loading

[0114] S11: Read the engineering configuration, determine the original image directory, label directory and output directory, scan the remote sensing images that meet the naming rules, and construct the sample list to be labeled;

[0115] S12: Read the original cloud and fog image of the current sample according to the user selection or default order, and display it in the interface;

[0116] S13: Detect whether the current sample has a color label image, if so, read the color label image; otherwise, read other grayscale images.

[0117] S14 Label matrix initialization: For samples with color labels, parse the color value pixel by pixel, map the background color to 0, the fog color to 1, and the cloud color to 2, and generate a grayscale label matrix for internal use.

[0118] S2: Multi-algorithm superpixel segmentation

[0119] S21: Construct a segmentation input image based on the original image, which can directly use an RGB composite image or a grayscale brightness temperature image or other images;

[0120] S22: The user selects one of the three algorithms SLIC, Felzenszwalb, and QuickShift through the algorithm selection control, and sets the superpixel number, scale, smoothness, and other parameters of the algorithm;

[0121] S23: According to the user selection, call the corresponding segmentation algorithm to divide the segmentation input image into regions, and get a region number matrix superpixels consistent with the size of the original image, where pixels with the same number constitute a region that can be operated as a whole;

[0122] S24: Calculate the region boundary information according to superpixels, generate a boundary overlay map, and use it for subsequent display.

[0123] S3: Region-level interactive labeling

[0124] S31: Display the original image view and the label view on the interface at the same time, and the user can click the image in any view;

[0125] S32: Capture the mouse click coordinates (x, y), and read the region number R of this position from superpixels;

[0126] S33: According to the current selected category C (background / fog / cloud), assign all pixels in current_label with superpixels == R to C, realizing "one-click labeling of the entire region";

[0127] S34: Before updating the label each time, backup the current_label into the undo stack to support multi-step undo / redo;

[0128] S35: When detecting repeated assignment to the same region, pop up a confirmation dialog to avoid accidental large-area coverage.

[0129] S4: Dual-mode display and label verification

[0130] S41: In label mode, map the current_label to a pseudo-color label map, and can superimpose region boundary lines, for quick browsing of the overall labeling situation;

[0131] S42: In the original image superposition mode, linearly mix the pseudo-color label map and the original image according to the transparency parameter α, and display the cloud and fog texture and the label boundary in the same view;

[0132] S43: Users can adjust α in real time through the slider bar to achieve continuous transition from "only looking at the original image" to "only looking at the label", and fine-tune the boundary according to the cloud and fog texture.

[0133] S5: Incremental modification tracking

[0134] S51: Compare original_label and current_label pixel by pixel, when the label value is not equal, set the difference mask diff_mask corresponding position to 1, otherwise set to 0;

[0135] S52: Provide "only look at the modified area" view in the interface, only display the area where diff_mask==1 or highlight it, so that the labeler and reviewer can quickly focus on "where to change";

[0136] S53: Count the total number of pixels with value 1 in diff_mask and the proportion relative to the total number of pixels in the image to form the "modification ratio" index;

[0137] S54: Further statistics of the category conversion matrix, such as the number of pixels changed from background to cloud, from cloud to fog, etc., for quality assessment.

[0138] S6: Label export and consistency verification

[0139] S61: Export current_label as a grayscale label map file for model training;

[0140] S62: Generate a color label map file according to the preset color mapping for visualization and report display;

[0141] S63: Optionally derive diff_mask and modify statistics for quality tracking;

[0142] S64: When it is necessary to carry out annotation consistency experiments, the same image is repeatedly independently annotated using the method of the present application, cloud class and fog class IoU indexes are calculated and compared with traditional pixel-by-pixel annotation methods, and the improvement of the present application in consistency is quantitatively verified.

[0143] It should be noted that the present application has the following optimization design and coordination mechanism on the basis of the above basic process:

[0144] Firstly, in step S1, the image loading module not only supports the reading of a single image, but also provides management functions for engineering batch data. Specifically, the user can specify the original sample directory through the "one-click selection of picture folder" method, the system automatically traverses all cloud and fog remote sensing images under the directory that meet the naming rules, constructs a list of samples to be annotated, and provides control buttons such as "save and next" and "jump to specified image" in the interface, so that the annotation process can be continuously performed in the same project, avoiding the inefficient operation of manually selecting the next image in the folder frequently required by existing general annotation software (such as tools that can only import images one by one).

[0145] Secondly, in steps S2 to S4, the multi-algorithm superpixel segmentation module and the dual-mode visual verification module work together to realize hierarchical segmentation and annotation process from global to local and from coarse to fine. The system preferentially adopts the SLIC rule superpixel segmentation algorithm by default to perform initial blocking on the whole cloud and fog image, and obtains a relatively coarse regular grid division by setting a small superpixel number, which is used to quickly delineate large-scale cloud and fog areas. When the cloud system presents complex morphologies such as clump, hook or strip, the regular small blocks obtained by SLIC may contain both cloud and non-cloud areas, resulting in the phenomenon of internal class mixing. At this time, the user can switch to the Felzenszwalb graph cut segmentation algorithm in step S2 to use the original image (such as RGB composite image or other multi-spectral composite image) as the segmentation input, and adjust the label transparency and region boundary transparency under dual-mode display to re-segment the mixed area, so that the generated irregular polygon region is more consistent with the natural cloud contour.

[0146] Then, in the cloud-sea, cloud-land interface area and the local area where there are fine cloud filaments, fog tongue and small low clouds, even if the Felzenszwalb algorithm is used, the situation that a single area contains cloud, sea or other ground object types at the same time may occur. For this purpose, the user can continue to select the QuickShift edge adaptive segmentation algorithm in step S2, and by appropriately reducing the kernel size and other parameters, the segmentation result is more automatically along the edge of the brightness and texture gradient distribution, so as to obtain a small scale area closely following the cloud-sea boundary. Then in step S3, the annotator only needs to locally refine these fine-grained areas, for example, only modify a circle of pixel areas near the boundary without changing the large area label inside the cloud body. If the labeling effect still does not meet the quality requirements, the user can iterate between S2-S4 multiple times until the boundary position is obtained which is visually highly consistent with the cloud and fog texture by adjusting the "original image contrast" switch, label transparency, boundary line transparency and label transparency in the original image superposition mode and other parameters under the double mode view. The combination of the above multi-algorithm switching and multi-dimensional transparency adjustment is not possessed by the existing general annotation software.

[0147] Again, in step S5, the present application uses an incremental modification tracking mechanism to only mark and highlight the "areas that change relative to the initial label". The system simultaneously retains the original label matrix and the current label matrix during the entire labeling process, and generates a difference mask by comparing pixels one by one, so that the auditor does not need to recheck the entire image, but only needs to check the pixel set marked as the modified area to complete the quality audit. When the modification range and quality reflected by the difference mask meet the requirements, step S6 can be entered to export the label file, and the labeling result is good for model training after the intersection consistency evaluation; if the audit result does not meet the requirements, the previous step can be returned to continue to modify the label according to the area indicated by the difference mask, forming a closed loop process of "labeling-verification-incremental modification-again verification".

[0148] The above described are preferred embodiments of the present application, and for those skilled in the art, various equivalent forms of modification of the present application without departing from the principles of the present application are within the protection scope of the appended claims of the present application.

Claims

1. An interactive annotation system for remote sensing cloud and fog images based on multi-algorithm superpixel segmentation, characterized in that: The system includes: The image loading module is used to load raw cloud and fog remote sensing images; The multi-algorithm superpixel segmentation module is used to select and switch between the SLIC regular superpixel segmentation algorithm, the Felzenszwalb graph cut segmentation algorithm, and the QuickShift edge adaptive segmentation algorithm to perform superpixel or region segmentation on the current cloud and fog remote sensing image and generate a region number matrix for region-level click annotation. The regional interactive annotation module is used to respond to user click operations, and to annotate the entire region with one click according to the region number matrix, supporting cancellation and confirmation of repeated annotation; The dual-mode visual verification module switches between label display mode and original cloud and fog remote sensing image overlay mode to determine the position of cloud, fog and background boundaries and complete detailed correction of the annotation results. Modify the tracking and quality assessment module to check and visualize the differences between the original label and the current label; The data export and consistency assessment module is used to export the current label results and related statistical information to an external file and to perform consistency assessment on different label results.

2. The interactive annotation system for remote sensing cloud and fog images based on multi-algorithm superpixel segmentation according to claim 1, characterized in that: The multi-algorithm superpixel segmentation module allows you to select at least one of the following: SLIC regular superpixel segmentation algorithm, Felzenszwalb graph cut segmentation algorithm, and QuickShift edge adaptive segmentation algorithm. The SLIC regular superpixel segmentation algorithm generates regular grid-like superpixel regions, the Felzenszwalb graph cut segmentation algorithm generates irregular polygonal regions to fit the outline of natural clouds, and the QuickShift edge adaptive segmentation algorithm generates edge adaptive regions to capture local details.

3. An interactive annotation method for remote sensing cloud and fog images based on multi-algorithm superpixel segmentation, characterized in that: The method is as follows: S1: Image loading: Read the remote sensing cloud and fog image to be labeled. If a corresponding color label image exists, convert it into a grayscale label matrix according to the color-category mapping relationship, where the background, fog, and cloud correspond to grayscale values ​​0, 1, and 2, respectively. If no label image exists, create an all-zero grayscale label matrix as the initial label. Save the grayscale label matrix as the current label matrix and the original label matrix respectively; S2: Multi-algorithm region segmentation: Construct a segmentation input image based on the remote sensing cloud and fog image, receive at least one segmentation algorithm selected by the user, perform region segmentation on the segmentation input image, and generate a region numbering matrix with the same image size, wherein pixels with the same number constitute a superpixel region; S3: Regional interactive annotation: Display the remote sensing cloud and fog image and regional boundaries in the graphical user interface, respond to user click operations, obtain the corresponding regional number from the regional number matrix according to the click position, and update the label values ​​of all pixels in the current label matrix of the region to the currently selected category; S4: Dual-mode display and verification: Provides label display mode and original image overlay mode; in label display mode, labels of each category are displayed in different colors; in original image overlay mode, pseudo-color label images and original images are mixed and displayed with adjustable transparency for boundary comparison and annotation verification. S5: Incremental Modification Tracking: Compare the original label matrix with the current label matrix, generate a difference mask matrix, count the number and proportion of modified pixels, and highlight the modified area; S6: Label Export and Consistency Verification: Export the current label matrix as grayscale label images and color visualization label images. Optional export of difference masks and modification of statistical information are also available. Calculate the intersection-union index for multiple annotation results of the same image to verify label consistency.

4. The interactive annotation method for remote sensing cloud and fog images based on multi-algorithm superpixel segmentation according to claim 3, characterized in that: The specific steps for converting the color label image into a grayscale label matrix in S1 include: S11, reading the color label image with the same name as the image to be labeled or corresponding to it according to a preset naming rule; S12, traversing the color label image pixel by pixel, matching the color value of each pixel with a preset category color table, setting the grayscale label value at the corresponding position to 0 when the color matches the background color, setting the grayscale label value at the corresponding position to 1 when the color matches the fog category color, and setting the grayscale label value at the corresponding position to 2 when the color matches the cloud category color; S13, copying the converted grayscale label matrix to the current label matrix and the original label matrix.

5. The interactive annotation method for remote sensing cloud and fog images based on multi-algorithm superpixel segmentation according to claim 3, characterized in that: The specific method for multi-algorithm region segmentation in S2 is as follows: S21: Receive the segmentation algorithm type and corresponding algorithm parameters selected by the user in the interface. The algorithm type includes at least the following: superpixel count and compactness parameters for the SLIC algorithm, scale parameters, smoothing parameters and minimum region size parameters for the Felzenszwalb algorithm, and kernel size, maximum distance and scale parameters for the QuickShift algorithm. S22: When the selected algorithm is SLIC, iterative clustering is performed on the segmented input image based on the joint metric of color and spatial distance, dividing the image into superpixel regions with regular shapes and uniform sizes; when the selected algorithm is Felzenszwalb, a graph structure is constructed and region merging is performed according to the weights between pixels to obtain segmented regions with irregular shapes that are close to the outline of clouds; when the selected algorithm is QuickShift, local clustering is performed based on color density and spatial distribution to obtain fine-grained regions that are sensitive to edge changes. S23: Map the region labeling results obtained by any of the above algorithms into a region number matrix and store it as the region index of the current image for subsequent region-level interactive annotation.

6. The interactive annotation method for remote sensing cloud and fog images based on multi-algorithm superpixel segmentation according to claim 3, characterized in that: The specific method for regional interactive annotation in S3 is as follows: Before each region label update, the current label matrix is ​​saved to the history stack. When the user triggers an undo command, the previous version of the label matrix is ​​restored from the history stack. When the user triggers a redo command, the next version of the label matrix is ​​restored from the history stack to support multi-step undo and redo. When the system detects that a user has repeatedly labeled the same superpixel region, a confirmation prompt will pop up for the user to reduce the impact of accidental operations on the labeling results.

7. The interactive annotation method for remote sensing cloud and fog images based on multi-algorithm superpixel segmentation according to claim 3, characterized in that: The incremental modification tracking method in S5 is as follows: The original label matrix and the current label matrix are compared at each pixel position. If the label values ​​of the two are different, the corresponding position in the difference mask matrix is ​​marked as a modified pixel; otherwise, it is marked as an unmodified pixel. The number of pixels modified is counted based on the difference mask matrix, and the modification ratio is obtained by dividing by the total number of pixels in the image. When displayed on the interface, only the areas marked as modified are highlighted or displayed separately, so that the annotation reviewers can focus on the manually modified areas for quality inspection.

8. The interactive annotation method for remote sensing cloud and fog images based on multi-algorithm superpixel segmentation according to claim 3, characterized in that: The consistency verification method is as follows: For the same remote sensing cloud and fog image, at least two reference label masks are obtained using the traditional pixel-by-pixel annotation method, and at least two label masks generated by the method of the present invention are obtained using the interactive annotation method based on multi-algorithm superpixel segmentation. The intersection-over-union ratio (IoU) between the label masks is calculated for fog and cloud categories respectively. The difference in IoU values ​​between the traditional method and the method of the present invention is compared to verify the improvement effect of the present invention in annotation consistency.

Images