A boundary-enhanced weakly supervised ocean remote sensing image semantic segmentation method and system

Abstract

The invention belongs to the field of image processing technology, and discloses a boundary-enhanced weakly supervised marine remote sensing image semantic segmentation method and system, comprising the following steps: step S1, image feature extraction: for the input global image, the final layer features and the intermediate layer features are extracted by the ViT model; step S2, cluster word comparison: the intermediate layer features are used to guide the final layer features used to generate pseudo labels, step S3, random uncertainty region comparison: in the intermediate layer random uncertainty region selection unit, two uncertain words of the intermediate layer output features are randomly selected, and in the cluster center replacement uncertainty region unit, the cluster center closest to the above two words is selected by comparison for replacement, and the replaced cluster center is used to supervise the uncertain region corresponding to the final layer; step S4, pseudo labels are generated. The invention improves the semantic segmentation effect of marine remote sensing images.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a boundary-enhanced weakly supervised ocean remote sensing image semantic segmentation method and system. Background Art

[0002] Marine remote sensing technology plays a key role in marine environmental monitoring and provides an important means for disaster forecasting, resource exploration, marine protection, etc. Semantic segmentation is the core technology for processing marine remote sensing images. The goal of semantic segmentation is to perform pixel-by-pixel prediction on the input image to achieve regional segmentation, and then identify the semantic information of the category as the segmentation result and superimpose high-level semantic information. This provides an effective tool for analyzing the marine environment and helps to deeply understand the dynamic changes of marine structure, characteristics and ecosystems.

[0003] Semantic segmentation mainly includes two important steps: first, mining features of the input image through the model (encoding area); second, segmenting by predicting the category pixel by pixel based on the extracted features (decoding area). The cutting-edge semantic segmentation model uses FCN to mine features through multi-scale and graph convolution, or uses Transformer to cut remote sensing images into small image blocks to reduce the amount of model calculation, and captures contextual information through attention mechanisms to model the relationship between target objects. However, these methods have the following problems:

[0004] First, the elements of marine remote sensing images are diverse, and the labeling cost of fully supervised models is high. As the scale of marine remote sensing image data continues to expand, the expensive labeling process of traditional fully supervised methods at the pixel level has become a bottleneck restricting the development of semantic segmentation. In particular, the pixel-level fully supervised methods based on CNN and Transformer cannot meet the feasibility of large-scale applications in terms of label requirements. The primary problem faced today is how to solve the semantic segmentation task of marine remote sensing images in a semi-supervised framework.

[0005] Second, the weakly supervised semantic segmentation model based on Transformer has the problem of overly smooth features, which leads to misjudgment of boundary information. In the weakly supervised semantic segmentation method, Transformer is used as the core structure. Due to the self-attention mechanism of Transformer, the model pays global attention to each position of the input image, resulting in overly smooth features and loss of some important boundary information. This smoothing phenomenon affects the model's accurate understanding of the image boundaries and may result in poor quality of the generated pseudo-labels. For example, when processing marine remote sensing images, if the model pays too much attention to areas unrelated to ships, the shape and position of the ship in the generated pseudo-label will deviate significantly from the actual annotation. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides a boundary-enhanced weakly supervised marine remote sensing image semantic segmentation method and system. The method is also based on the Transformer structure and uses Segmenter as the backbone, but on this basis, a new pseudo-label generation process is introduced to achieve semi-supervised learning by making pseudo-labels. While retaining the global context information, the method solves the problems of over-smoothing and poor pseudo-label quality through the clustering token comparison module and the random uncertain region comparison module. Specifically, the clustering token comparison module uses the intermediate layer features and image-level labels to cluster the determined areas and generate a clustering relationship diagram, thereby generating high-quality pseudo-labels. The random uncertain region comparison module further improves the representation consistency, reduces the number of uncertain regions, and thus improves the quality and accuracy of the pseudo-labels. Therefore, compared with the traditional fully supervised method, the semi-supervised method of the present invention can not only effectively reduce the annotation cost, but also improve the generalization ability and segmentation performance of the model, bringing a new solution to the task of semantic segmentation of marine remote sensing images.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0008] A boundary-enhanced weakly supervised ocean remote sensing image semantic segmentation method comprises the following steps:

[0009] Step S1, image feature extraction: for the input global image, the final layer features and the intermediate layer features are extracted through the ViT model;

[0010] Step S2, clustered word-unit comparison: use the intermediate layer features to guide the final layer features used to generate pseudo labels. Specifically, use the intermediate layer features to divide the word-units into certain areas and uncertain areas using double thresholds, then cluster the certain areas under the guidance of labels that only mark category information, generate a cluster relationship graph after clustering, and use the cluster relationship graph to guide the final layer output features to generate pseudo labels;

[0011] Step S3: Random uncertainty region comparison:

[0012] In the random uncertainty region selection unit of the middle layer, the uncertain words of the output features of two middle layers are randomly selected. In the cluster center replacement uncertainty region unit, the cluster center closest to the uncertain words of the output features of the two middle layers is selected by comparison for replacement, and the replaced cluster center is used to supervise the uncertainty region corresponding to the final layer.

[0013] Step S4, generate pseudo labels: Use the class activation map method to make the final layer output features into pseudo labels as supervision labels for the segmentation task in Segmenter.

[0014] Furthermore, step 1 is as follows: the input image is transformed into Divided into scales We obtain N image blocks of the same size, where H, W, and C are the height, width, and number of channels of the image, respectively, and map all image blocks into a linear mapping embedding sequence, expressed as: , the embedding sequence E is input into the Transformer module iteration of the ViT model, where is the image embedding, is the position embedding of the image.

[0015] Furthermore, the embedding sequence E is input into the Transformer module to obtain the intermediate layer features and the final layer features , , N is the number of image blocks, d is the dimension of the image block to the word unit, where the Transformer module includes a multi-head self-attention MSA module and a multi-layer perceptron MLP. The layer norm LN is applied before each block, and the residual connection is performed after each block. The multi-head self-attention MSA module is composed of multiple self-attention mechanisms. The input of the self-attention mechanism includes three vectors , as shown in formula (1):

[0016] (1);

[0017] in, , , is a learnable parameter, E is an embedding sequence;

[0018] The self-attention mechanism consists of the calculation between three vectors Q, K, and V, which are calculated as follows:

[0019] (2);

[0020] in is the dimension of vector K;

[0021] Assume there are L layers of Transformer modules. Output of the multi-head attention mechanism It is sent to the multi-layer perceptron MLP, and the layer norm LN is applied before each block, followed by residual connection. The intermediate layer features of the encoding region can be obtained by multiple cycles. and the final layer features , the calculation formula is as follows:

[0022] (3);

[0023] (4);

[0024] in express The output features of the multi-head self-attention MSA module, express The input features of the layer Transformer module, express The output features of the multi-layer perceptron MLP or Input features of the layer Transformer module.

[0025] Furthermore, the specific steps in step S2 are as follows:

[0026] Production of intermediate layer clustering relationship diagram: After obtaining the intermediate layer features Then, according to the second dimension Perform the addition operation and get , N is the number of image blocks or words; then, two background thresholds are used , Will Features divided into definite and uncertain regions , The determined area is represented by 1, and the uncertain area is represented by 99. ; Due to the weak supervision of the label that only marks the category information, we can know the number of categories num of the original input image, and use this to compare Corresponding The determined area in the clustering operation is The deterministic regions in are divided into num categories, The corresponding determined area position in is set to the number corresponding to the category, that is, the digital code 1 representing the determined area is replaced by the specific category code 0, 1, 2, ..., num-1, and the result is , on this basis, a cluster relationship diagram of the pairwise relationship is made : If two tokens share the same semantic label, they are labeled as a positive pair; otherwise, they are labeled as a negative pair;

[0027] The clustering relationship graph guides the final layer output features: The clustering relationship graph R supervises the final layer output features F; the final layer features F are composed of num tokens, and the tokens are compared two by two. Only two tokens that belong to the deterministic area in the clustering relationship graph R are considered, and the uncertain area is ignored; if the semantic labels are the same, that is, marked as positive pairs in the clustering relationship graph R, the similarity of the two final layer image block tokens is maximized, otherwise the similarity is minimized; the image block tokens of the last layer are used as the final layer output features , then the loss function used in step 2 is constructed as:

[0028] (5);

[0029] in represents the cosine similarity, and Represent the number of positive and negative pairs respectively; It represents the first traversal of F according to the first dimension feature, with a shape of d×1. It represents the second traversal of F according to the first dimension feature, with a shape of d×1. and Indicates the category code of each determined region word;

[0030] A classifier is added after the final layer feature F to generate pseudo labels.

[0031] Furthermore, in step S3, the input is also taken from the features of the middle layer , in the area marked with uncertainty and certainty Under the guidance of Two word units belonging to the uncertain regions, such as uncertain region 1 and uncertain region 2; next, in the cluster center replacement uncertainty region unit, the Euclidean distances between the two random uncertain regions and the num cluster centers generated in the clustering operation of step S2 are calculated respectively. , k=1,2, the formula is as follows:

[0032] (6);

[0033] Pick The minimum distance in the record, the two cluster centers corresponding to the records are, and , replace the uncertain regions 1 and 2 with them; then process the final layer output feature F in the corresponding uncertain region extraction unit of the final layer, and select the two image block words corresponding to the positions of the random uncertain regions 1 and 2 selected in the intermediate layer in the final layer feature, and , prompting it to and Approaching, in this way, under the guidance of the middle layer, the goal of aligning the uncertainty area of ​​the final layer with the certainty area is achieved.

[0034] Furthermore, the loss function used in step S3 is the MSE function, as follows:

[0035] (7);

[0036] in, represents the contrast loss of random uncertain regions, k represents the index of the selected random uncertain region, represents the cluster center closest to the uncertainty region k, Represents the final layer features corresponding to the random uncertainty region with the intermediate layer index value k.

[0037] The present invention also provides a boundary-enhanced weakly supervised marine remote sensing image semantic segmentation system, which is used to implement the boundary-enhanced weakly supervised marine remote sensing image semantic segmentation method as described above, including an image feature extraction module, a clustered word unit comparison module, a random uncertain region comparison module, and a pseudo label generation module;

[0038] The image feature extraction module is used to extract intermediate layer features and final layer features through the ViT model;

[0039] The cluster word comparison module is used to use the intermediate layer features to guide the final layer features for generating pseudo labels; it includes an intermediate layer cluster relationship map making unit and a cluster relationship map guiding the final layer output feature unit; the intermediate layer cluster relationship map making unit divides each image block into category areas using the intermediate layer output features, and makes a cluster relationship map according to the category relationship; the cluster relationship map guiding the final layer output feature unit uses the cluster relationship map to guide the differences of image blocks in the same category area, and amplifies the differences of image blocks in different category areas;

[0040] The random uncertain region comparison module is used to align the final layer uncertain region with the cluster center of the nearest determined region in the intermediate layer; it includes a final layer uncertain region extraction unit, an intermediate layer random uncertain region selection unit and a cluster center replacement uncertain region unit; the intermediate layer random uncertain region selection unit randomly selects two uncertain regions from the intermediate layer output features; the cluster center replacement uncertain region unit calculates the cluster center closest to the selected uncertain region and replaces it; the final layer uncertain region extraction unit is used to select two image block word units corresponding to the positions of the two random uncertain regions selected in the intermediate layer in the final layer features, so as to force the two word units of the final layer uncertainty region to align with the replaced cluster center;

[0041] The pseudo-label generation module uses the class activation map method to convert the final layer output features into pseudo-labels to serve as supervisory labels for segmentation tasks in Segmenter.

[0042] Compared with the prior art, the present invention has the following advantages:

[0043] 1. The problem of high labeling cost of fully supervised models has been improved. The present invention proposes a weakly supervised semantic segmentation method, which significantly reduces the dependence on a large amount of labeled data, thereby greatly reducing the labeling cost. In traditional fully supervised learning models, a large amount of labeled data is often required to train a high-performance model. The labeling process of these data is not only time-consuming and labor-intensive, but also costly, especially in the field of marine remote sensing images. The labeling work needs to be completed by experts with professional knowledge, which further increases the cost. The present invention reduces the original pixel-level label requirements to image-level labels by introducing weakly supervised learning technology. This method can still train a model with good performance when labeled data is scarce.

[0044] 2. This method aims to improve the quality of pseudo-labels generated by weakly supervised semantic segmentation methods, especially to solve the problem of inaccurate boundary information of pseudo-labels caused by excessive smoothing of Transformer. The characteristics of the intermediate layer of Transformer with rich semantic information and low similarity between tokens are used to guide the smooth features of the final layer, so that it can better retain boundary information and align with the differentiated information of the intermediate layer. Compared with traditional remote sensing image semantic segmentation methods, such as Segmenter, this method has made targeted optimizations in dealing with the problem of excessive smoothing of tokens in the later stage of Transformer, thereby effectively avoiding inaccurate segmentation results. By emphasizing the retention of boundary information, the invention makes the categories assigned by the segmentation pseudo-labels more accurate, especially in dealing with the edge positions of tokens.

[0045] In summary, compared with the traditional fully supervised method, the semi-supervised method provided by the present invention can not only effectively reduce the annotation cost, but also improve the generalization ability and segmentation performance of the model, bringing a new solution to the semantic segmentation task of marine remote sensing images. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0047] Figure 1 is a system structure diagram of the present invention;

[0048] Figure 2 It is a schematic diagram of the clustered word unit comparison module in the present invention;

[0049] Figure 3 It is a schematic diagram of the structure of the clustering method in the present invention;

[0050] Figure 4 It is a schematic diagram of the random uncertain region comparison module in the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0052] See also Figures 1 to 4 The present invention provides a boundary-enhanced weakly supervised marine remote sensing image semantic segmentation method and system, with Segmenter as the backbone. On this basis, the present invention adds a process for making pseudo labels, which still uses the traditional Transformer module to ensure the interaction of global context information, and uses the pseudo labels finally generated to supervise the prediction results of the Segmenter model.

[0053] It mainly includes the following two points:

[0054] First, based on the original image feature extraction module and pseudo-label generation module, a cluster token comparison module (CTC) is proposed. Through experimental observation, it is found that with the stacking of Transformer blocks, the output features of the final layer of Transformer have the problem of over-smoothing. The pseudo-labels generated cannot accurately classify the remote sensing image categories. The intermediate layer features of Transformer do not have this problem of over-smoothing while having semantic information. Therefore, the goal of the cluster token comparison module is to use the intermediate layer features of Transformer to guide the final layer features used to generate pseudo-labels. The intermediate layer features are used to divide the tokens into deterministic areas and uncertain areas using dual thresholds. Then, the determined areas are clustered under the guidance of image-level labels. After clustering, a cluster relationship diagram is generated to supervise the patch tokens of the final layer to solve the problem of over-smoothing and generate higher-quality pseudo-labels.

[0055] Second, a random uncertainty region comparison module (RURC) is proposed. Using the clustered word comparison module to deal with the over-smoothing problem can prompt Transformer to generate high-quality pseudo-labels. However, there is still a problem. Those tokens marked as uncertainty regions cannot provide guidance for our final layer. Therefore, in order to improve this method, a random uncertainty region comparison module is proposed to reduce the number of uncertainty regions. Two middle-layer uncertainty regions are randomly selected, and the cluster centers closest to them are selected by comparison. The cluster centers are used to supervise the corresponding final-layer uncertainty regions to promote the consistency of their representations and reduce the number of uncertainty regions.

[0056] The structure of each module and the data processing process within the module will be introduced in detail later.

[0057] Combine the following Figure 1 , introduces a boundary-enhanced weakly supervised ocean remote sensing image semantic segmentation method provided in this embodiment.

[0058] The following steps are involved:

[0059] Step S1, image feature extraction (implemented by image feature extraction module):

[0060] For the input global image, the final layer features and the intermediate layer features are extracted through the ViT model. In this embodiment, a 12-layer ViT model is used to process image features, the 10th layer of ViT is selected as the intermediate layer (auxiliary layer), the 12th layer is selected as the final layer, and the output features of the intermediate layer and the final layer are extracted respectively.

[0061] Step 1 is as follows: Input image is transformed into Divided into scales image blocks, and obtain N image blocks of the same size. ,in H, W, and C are the length and width of each image block, respectively. The image block is mapped into a linear mapping embedding sequence, expressed as , the embedding sequence E is input into the Transformer module iteration of the ViT model, where is the image embedding, is the position embedding of the image.

[0062] Input the embedding sequence E into the Transformer module to obtain the intermediate layer features and the final layer features , , N is the number of image blocks, d is the dimension of the image block to the word unit, where the Transformer module includes a multi-head self-attention MSA module and a multi-layer perceptron MLP. The layer norm LN is applied before each block, and the residual connection is performed after each block. The multi-head self-attention MSA module is composed of multiple self-attention mechanisms. The input of the self-attention mechanism includes three vectors , as shown in formula (1):

[0063] (1);

[0064] in, , , is a learnable parameter, E is an embedding sequence;

[0065] The self-attention mechanism consists of the calculation between three vectors Q, K, and V, which are calculated as follows:

[0066] (2);

[0067] in is the dimension of vector K.

[0068] Assume there are L layers of Transformer modules. Output of the multi-head attention mechanism It is sent to the multi-layer perceptron MLP, and the layer norm LN is applied before each block, followed by residual connection. The intermediate layer features of the encoding region can be obtained by multiple cycles. and the final layer features , the calculation formula is as follows:

[0069] (3);

[0070] (4);

[0071] in express The output features of the multi-head self-attention MSA module, express The input features of the layer Transformer module, express The output features of the multi-layer perceptron MLP or Input features of the layer Transformer module.

[0072] Step S2: Cluster word comparison (implemented by cluster word comparison module):

[0073] The intermediate layer features are used to guide the final layer features used to generate pseudo labels. Specifically: the intermediate layer features are used to divide the word units into certain areas and uncertain areas using double thresholds. The certain areas are then clustered under the guidance of labels that only mark category information. After clustering, a clustering relationship graph is generated. The clustering relationship graph is used to guide the final layer output features to generate pseudo labels.

[0074] like Figure 1-2 As shown, the specific steps in step S2 are as follows:

[0075] Intermediate layer clustering relationship diagram production: Obtain the intermediate layer features in the previous step Then, according to the second dimension Perform the addition operation and get , N is the number of image blocks or words. Then, two background thresholds are used , Will Features divided into certain areas (such as word 1 and word 3) and uncertain areas (such as word 2 and word N) , The determined area is represented by 1, and the uncertain area is represented by 99. ; Due to the weak supervision of the label that only marks the category information, the number of categories num of the original input image can be known, which is used to compare Corresponding The determined area in the clustering operation is The deterministic regions in are divided into num categories, The corresponding determined area position in is set to the number corresponding to the category, that is, the digital code 1 representing the determined area is replaced by the specific category code 0, 1, 2, ..., num-1, and the result is On this basis, a cluster relationship diagram of the pairwise relationship is made , to supervise the features of the final layer; if two tokens share the same semantic label, they are labeled as a positive pair; otherwise, they are labeled as a negative pair. Figure 3 To perform clustering operations on the determined areas, the method used is the K-means clustering method, which is a widely used unsupervised learning algorithm that aims to divide the points in the data set into K clusters (Cluster) so that the similarity of points within the same cluster is high, while the similarity of points between different clusters is low.

[0076] The clustering relationship graph guides the final layer output features: The clustering relationship graph R supervises the final layer output features F. The final layer feature F consists of num tokens. In this unit, tokens need to be compared two by two. It should be noted that in order to ensure reliability, only two tokens that belong to the deterministic area in the clustering relationship graph R are considered, and the uncertain area is ignored.

[0077] To solve the problem of over-smoothing, if the semantic labels are the same, that is, marked as positive pairs in the clustering relationship graph R, the similarity of the two final layer image block words is maximized, otherwise the similarity is minimized. Let The image block word of the last layer (that is, the image block word of the last layer is used as the final layer output feature ), then the loss function used in step 2 is constructed as:

[0078] (5);

[0079] in represents the cosine similarity, and Represent the number of positive and negative pairs respectively; It represents the first traversal of F according to the first dimension feature, with a shape of d×1. It represents the second traversal of F according to the first dimension feature, with a shape of d×1. and Indicates the category code of each determined region word;

[0080] A classifier is added after the final layer feature F to generate pseudo labels.

[0081] Step S3, random uncertain region comparison (implemented by random uncertain region comparison module):

[0082] In the random uncertainty region selection unit of the middle layer, uncertain word units of two middle layer output features are randomly selected. In the cluster center replacement uncertainty region unit, the cluster center closest to the uncertain word units of the two middle layer output features is selected for replacement by comparison. The replaced cluster center is used to supervise the uncertain region corresponding to the final layer to promote the consistency of their representations and reduce the number of uncertain regions.

[0083] like Figure 1 , Figure 4 As shown, in step S3, the input is also taken from the features of the middle layer , in the area marked with uncertainty and certainty Under the guidance of The two words belonging to the uncertain region, such as (uncertainty region 1) and (uncertain region 2); Next, in the cluster center replacement uncertainty region unit, the Euclidean distances between the two random uncertain regions and the num cluster centers generated in the clustering operation in step S2 are calculated respectively. , k=1,2, the formula is as follows:

[0084] (6);

[0085] Pick The minimum distance in the record, the two cluster centers corresponding to the records are, and , replace the uncertain regions 1 and 2 with them; then process the final layer output feature F in the corresponding uncertain region extraction unit of the final layer, and select the two image block words corresponding to the positions of the random uncertain regions 1 and 2 selected in the intermediate layer in the final layer feature, and , prompting it to and In this way, under the guidance of the middle layer, the uncertainty region of the final layer is aligned with the certainty region, thereby reducing the number of uncertainty regions. Figure 4 middle For processing and The linear mapping head of Processing and The linear mapping header will be processed and and As Input to the loss function.

[0086] The loss function used in step S3 is the MSE function, as follows:

[0087] (7);

[0088] in, represents the contrast loss of random uncertain regions, k represents the index of the selected random uncertain region, represents the cluster center closest to the uncertainty region i, represents the final layer feature corresponding to the random uncertainty region with the intermediate layer index value i;

[0089] Finally, Segmenter is used as the backbone, and the generated pseudo labels are used to supervise the segmentation task.

[0090] Step S4, generate pseudo labels: Use the class activation map method to make the final layer output features into pseudo labels as supervision labels for the segmentation task in Segmenter.

[0091] As another embodiment of the present invention, a boundary-enhanced weakly supervised ocean remote sensing image semantic segmentation system is provided, the structure of which is as follows: Figure 1-4 As shown, it includes an image feature extraction module, a clustered word unit comparison module, a random uncertain region comparison module, and a pseudo label generation module.

[0092] The image feature extraction module is used to extract intermediate layer features and final layer features through the ViT model.

[0093] The cluster word comparison module is used to use the intermediate layer features to guide the final layer features for generating pseudo labels; it includes an intermediate layer cluster relationship map production unit and a cluster relationship map guidance final layer output feature unit; the intermediate layer cluster relationship map production unit divides each image block into category areas using the intermediate layer output features, and produces a cluster relationship map according to the category relationship; the cluster relationship map guidance final layer output feature unit uses the cluster relationship map to guide the differences of image blocks in the same category area and amplify the differences of image blocks in different category areas.

[0094] The random uncertain region comparison module is used to align the final layer uncertain region with the cluster center of the determined region closest to the intermediate layer; it includes a final layer uncertain region extraction unit, an intermediate layer random uncertain region selection unit and a cluster center replacement uncertain region unit; the intermediate layer random uncertain region selection unit randomly selects two uncertainty regions from the intermediate layer output features; the cluster center replacement uncertain region unit calculates the cluster center closest to the selected uncertainty region and replaces it; the final layer uncertain region extraction unit is used to select two image block word elements corresponding to the positions of the two random uncertain regions selected in the intermediate layer in the final layer features, so as to force the two word elements of the final layer uncertainty region to be aligned with the replaced cluster center.

[0095] The pseudo-label generation module uses the class activation map method to convert the final layer output features into pseudo-labels to serve as supervisory labels for segmentation tasks in Segmenter.

[0096] The system is used to implement the weakly supervised semantic segmentation method of marine remote sensing images with boundary enhancement as described above. The functional implementation of each module and the data processing process can be found in the previous records and will not be repeated here.

[0097] In summary, the present invention proposes a boundary-enhanced weakly supervised marine remote sensing image semantic segmentation method and system, which is a new marine remote sensing image semantic segmentation method based on semi-supervised learning and Transformer structure. Similar to the existing method, the present invention is also based on the Transformer structure and uses Segmenter as the backbone, but on this basis, the present invention introduces a new pseudo-label generation process, and realizes semi-supervised learning by making pseudo-labels. The method of the present invention retains the global context information while solving the problems of over-smoothing and poor pseudo-label quality through technical means such as clustered word comparison module (CTC) and random uncertain region comparison module (RURC); specifically, the clustered word comparison module of the present invention uses intermediate layer features and image-level labels to cluster the determined areas and generate a cluster relationship graph, thereby generating high-quality pseudo-labels, while the random uncertain region comparison module further improves the representation consistency and reduces the number of uncertain areas, thereby improving the quality and accuracy of the pseudo-labels. Therefore, compared with the traditional fully supervised method, the semi-supervised method of the present invention can not only effectively reduce the annotation cost, but also improve the generalization ability and segmentation performance of the model, bringing a new solution to the task of semantic segmentation of marine remote sensing images.

[0098] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Any changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A boundary-enhanced weakly supervised ocean remote sensing image semantic segmentation method, characterized in that: The following steps are involved: Step S1, image feature extraction: for the input global image, the final layer features and the intermediate layer features are extracted through the ViT model; Step S2, clustered word-unit comparison: use the intermediate layer features to guide the final layer features used to generate pseudo labels. Specifically, use the intermediate layer features to divide the word-units into certain areas and uncertain areas using double thresholds, then cluster the certain areas under the guidance of labels that only mark category information, generate a cluster relationship graph after clustering, and use the cluster relationship graph to guide the final layer output features to generate pseudo labels; Step S3: Random uncertainty region comparison: In the random uncertainty region selection unit of the middle layer, the uncertain words of the output features of two middle layers are randomly selected. In the cluster center replacement uncertainty region unit, the cluster center closest to the uncertain words of the output features of the two middle layers is selected by comparison for replacement, and the replaced cluster center is used to supervise the uncertainty region corresponding to the final layer. Step S4, generate pseudo labels: Use the class activation map method to make the final layer output features into pseudo labels as supervision labels for the segmentation task in Segmenter.

2. The method for semantic segmentation of weakly supervised ocean remote sensing images with boundary enhancement according to claim 1, characterized in that: Step 1 is as follows: Input image is transformed into Divided into scales We obtain N image blocks of the same size, where H, W, and C are the height, width, and number of channels of the image, respectively, and map all image blocks into a linear mapping embedding sequence, expressed as: , the embedding sequence E is input into the Transformer module iteration of the ViT model, where is the image embedding, is the position embedding of the image.

3. The method for semantic segmentation of weakly supervised ocean remote sensing images with boundary enhancement according to claim 2, characterized in that: Input the embedding sequence E into the Transformer module to obtain the intermediate layer features and the final layer features , , N is the number of image blocks, d is the dimension of the image block to the word unit, where the Transformer module includes a multi-head self-attention MSA module and a multi-layer perceptron MLP. The layer norm LN is applied before each block, and the residual connection is performed after each block. The multi-head self-attention MSA module is composed of multiple self-attention mechanisms. The input of the self-attention mechanism includes three vectors , as shown in formula (1): (1)； in, , , is a learnable parameter, E is an embedding sequence; The self-attention mechanism consists of the calculation between three vectors Q, K, and V, which are calculated as follows: (2)； in is the dimension of vector K; Assume there are L layers of Transformer modules. Output of the multi-head attention mechanism It is sent to the multi-layer perceptron MLP, and the layer norm LN is applied before each block, followed by residual connection. The intermediate layer features of the encoding region can be obtained by multiple cycles. and the final layer features , the calculation formula is as follows: (3)； (4)； in express The output features of the multi-head self-attention MSA module, express The input features of the layer Transformer module, express The output features of the multi-layer perceptron MLP or Input features of the layer Transformer module.

4. The method for semantic segmentation of weakly supervised ocean remote sensing images with boundary enhancement according to claim 1, characterized in that: The specific steps in step S2 are as follows: Production of intermediate layer clustering relationship diagram: After obtaining the intermediate layer features Then, according to the second dimension Perform the addition operation and get , N is the number of image blocks or words; then, two background thresholds are used , Will Features divided into definite and uncertain regions , The determined area is represented by 1, and the uncertain area is represented by 99. ; Due to the weak supervision of the label that only marks the category information, we can know the number of categories num of the original input image, and use this to compare Corresponding The determined area in the clustering operation is The deterministic regions in are divided into num categories, The corresponding determined area position in is set to the number corresponding to the category, that is, the digital code 1 representing the determined area is replaced by the specific category code 0, 1, 2, ..., num-1, and the result is , on this basis, a cluster relationship diagram of the pairwise relationship is made : If two tokens share the same semantic label, they are marked as a positive pair; Otherwise, they are marked as negative pairs; The clustering relationship graph guides the final layer output features: The clustering relationship graph R supervises the final layer output features F; the final layer features F are composed of num tokens, and the tokens are compared two by two. Only two tokens that belong to the deterministic area in the clustering relationship graph R are considered, and the uncertain area is ignored; If the semantic labels are the same, that is, marked as positive pairs in the clustering relationship graph R, the similarity of the two final layer image block words is maximized, otherwise the similarity is minimized; the image block word of the last layer is used as the output feature of the final layer , then the loss function used in step 2 is constructed as: (5)； in represents the cosine similarity, and Represent the number of positive and negative pairs respectively; It represents the first traversal of F according to the first dimension feature, with a shape of d×1. It represents the second traversal of F according to the first dimension feature, with a shape of d×1. and Indicates the category code of each determined region word; A classifier is added after the final layer feature F to generate pseudo labels.

5. The method for semantic segmentation of weakly supervised ocean remote sensing images with boundary enhancement according to claim 4, characterized in that: In step S3, the input is also taken from the middle layer. , in the area marked with uncertainty and certainty Under the guidance of Two word units belonging to the uncertain regions, such as uncertain region 1 and uncertain region 2; next, in the cluster center replacement uncertainty region unit, the Euclidean distances between the two random uncertain regions and the num cluster centers generated in the clustering operation of step S2 are calculated respectively. , k=1,2, the formula is as follows: (6)； Pick The minimum distance in the record, the two cluster centers corresponding to the records are, and , replace the uncertain regions 1 and 2 with them; then process the final layer output feature F in the corresponding uncertain region extraction unit of the final layer, and select the two image block words corresponding to the positions of the random uncertain regions 1 and 2 selected in the intermediate layer in the final layer feature, and , prompting it to and Approaching, in this way, under the guidance of the middle layer, the goal of aligning the uncertainty area of ​​the final layer with the certainty area is achieved.

6. The method for semantic segmentation of weakly supervised ocean remote sensing images with boundary enhancement according to claim 4, characterized in that: The loss function used in step S3 is the MSE function, as follows: (7)； in, represents the contrast loss of random uncertain regions, k represents the index of the selected random uncertain region, represents the cluster center closest to the uncertainty region k, Represents the final layer features corresponding to the random uncertainty region with the intermediate layer index value k.

7. A boundary-enhanced weakly supervised ocean remote sensing image semantic segmentation system, characterized in that: A weakly supervised marine remote sensing image semantic segmentation method for implementing boundary enhancement as described in any one of claims 1 to 6, comprising an image feature extraction module, a clustered word unit comparison module, a random uncertain region comparison module, and a pseudo label generation module; The image feature extraction module is used to extract intermediate layer features and final layer features through the ViT model; The clustered word unit comparison module is used to use the intermediate layer features to guide the final layer features used to generate pseudo labels; It includes a middle-layer clustering relationship graph production unit and a clustering relationship graph-guided final-layer output feature unit; The intermediate layer cluster relationship map making unit divides each image block into category regions using the intermediate layer output features, and makes a cluster relationship map according to the category relationship; the cluster relationship map guides the final layer output feature unit to use the cluster relationship map to guide the differences of image blocks in the same category region, and amplify the differences of image blocks in different category regions; The random uncertain region comparison module is used to align the final layer uncertain region with the nearest determined region cluster center in the intermediate layer; It includes a final layer uncertainty region extraction unit, an intermediate layer random uncertainty region selection unit and a cluster center replacement uncertainty region unit; The random uncertainty region selection unit in the middle layer randomly selects two uncertainty regions from the output features of the middle layer; The cluster center replacement uncertainty area unit calculates the cluster center closest to the selected uncertainty area and replaces it; the final layer uncertainty area extraction unit is used to select two image block words corresponding to the positions of the two random uncertainty areas selected in the middle layer in the final layer feature, forcing the two words in the final layer uncertainty area to align with the replaced cluster center; The pseudo-label generation module uses the class activation map method to convert the final layer output features into pseudo-labels to serve as supervisory labels for segmentation tasks in Segmenter.

Images