A Multi-Target Extraction Method for Whole-Scene SAR Marine Aquaculture Based on Semantic Enhancement Feature Fusion and Self-Supervised Transformer

Abstract

A method for extracting multiple targets in whole-scene SAR marine aquaculture using a self-supervised transformer based on semantic enhancement feature fusion is proposed. The steps are as follows: 1) For massive amounts of unlabeled remote sensing monitoring data on marine aquaculture, a framework for extracting multiple targets in whole-scene marine aquaculture is established, including an input embedding module, a transformer encoder, a multi-feature fusion module, and a high-level semantic enhancement module; 2) Based on self-distillation learning (knowledge distillation in self-supervised learning), combined with global contrastive learning loss and local image masking loss, the encoder is trained; 3) A semantic enhancement feature fusion decoder is proposed to perform deep semantic enhancement, effectively covering shallow shape features and deep semantic features. This invention can achieve accurate extraction of multiple targets in whole-scene marine aquaculture, alleviating the problems of over-reliance on labels, imbalanced sample data, and misjudgment caused by large intra-class differences and small inter-class differences among multiple targets in supervised learning methods. It also enhances the anti-interference ability against speckle noise in synthetic aperture radar images and improves the accuracy of multi-target extraction in marine aquaculture.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of marine remote sensing and artificial intelligence, and provides a method for extracting multiple targets in marine aquaculture using a whole-scene SAR method that integrates semantic enhancement features with a self-supervised transformer. Background Technology

[0002] Marine aquaculture refers to the cultivation of marine organisms such as fish, shellfish, and seaweed in the ocean or other saline environments. As a way to meet the growing demand for seafood, alleviate pressure on wild fish populations, and promote sustainable food security, it has received increasing attention. While marine aquaculture brings significant economic benefits, it also leads to pollution of the marine ecosystem. Monitoring marine aquaculture activities is crucial for the macro-management of marine resources. Synthetic Aperture Radar (SAR) imagery, due to its advantages such as all-day, all-weather imaging, has become an important tool for monitoring marine aquaculture. On the sea surface, the scattering characteristics of marine aquaculture cages and rafts differ from those of seawater; extracting marine aquaculture information from SAR imagery is beneficial for estimating area and yield.

[0003] Existing methods for extracting marine aquaculture data are mostly designed for cropped images containing only aquaculture targets and seawater backgrounds. For example, the literature [Wang Xinzhe, Deng Qiwen, Wang Jichao, et al. Extraction of marine raft aquaculture information using deep semantic segmentation MRF model [J]. Journal of Shandong University (Engineering Science), 2022, 52(2): 89-98] does not consider whole-scene segmentation. However, marine aquaculture is widely distributed near the coast of the mainland or around islands. When monitoring with whole-scene satellite remote sensing images, it is easily affected by interference from land islands and data imbalance between classes. In addition, marine targets will drift and deform with changes in sea conditions, and their own imaging characteristics will also change, resulting in the ground truth label samples not being able to generalize to other images. The patent [Wang Chengyi, Guo Yanjun, Chen Jiansheng, et al. Classification method of marine raft aquaculture area based on panoramic segmentation of optical remote sensing images [P]. Beijing: CN115908894A, 2023-04-04] extracts panoramic marine aquaculture data based on supervised methods, which requires a large number of training data samples. To address the aforementioned issues, a method for extracting multiple targets in whole-scene SAR marine aquaculture using a self-supervised transformer based on semantic enhancement features is proposed. This method overcomes the technical challenges of mutual interference and data imbalance between classes in whole-scene remote sensing images with a limited number of labels, and achieves accurate extraction of multiple targets in whole-scene marine aquaculture with a limited number of labels. Summary of the Invention

[0004] This invention primarily addresses how to obtain salient features from a large amount of unlabeled data through self-supervised learning, mitigating the misjudgment problem caused by large intra-class differences and small inter-class differences. Furthermore, it proposes a whole-scene SAR multi-target extraction method for marine aquaculture based on a small number of labeled samples and a self-supervised transformer for semantic enhancement feature fusion. The method uses a transformer model as the encoder, trains it through self-distillation to obtain salient features, and applies these features to the semantic enhancement feature fusion decoder. With fine-tuning using a small number of labels, a high-precision marine aquaculture extraction result using self-supervised transformer is obtained.

[0005] To achieve the aforementioned technical requirements, the technical solution adopted by this invention is as follows:

[0006] A method for multi-target extraction in whole-scene SAR marine aquaculture using semantically enhanced feature fusion and self-supervised transformer includes the following steps:

[0007] The first step involves collecting a large amount of unlabeled remote sensing monitoring data for marine aquaculture, slicing it into 512×512 images, and then using a marine aquaculture transformer encoder with self-distillation training to discover salient features and obtain long-term dependent features. This effectively addresses the sample imbalance caused by the large proportion of seawater area in the overall remote sensing image during marine aquaculture monitoring, reducing information loss. The marine aquaculture transformer encoder consists of an input embedding module and an encoding module, as detailed below:

[0008] The input embedding described in 1.1) is the process of encoding an image into a one-dimensional vector. Each two-dimensional image is first divided into n smaller image blocks. Through convolution operations with kernel sizes the same as the image block sizes, the n image blocks are embedded into n values ​​and arranged into a one-dimensional vector. To prevent information loss, feature upscaling is performed simultaneously with the transformation into a one-dimensional vector. In addition, to prevent the loss of positional information during the embedding process, positional embedding is introduced, as shown in equations (1) and (2).

[0009]

[0010]

[0011] Where pos represents the position of the image patch in the image, i represents the dimension index value of the image patch, and d model This represents the total dimension of the image patch. Location embeddings are added to the input embedding to capture positional information within the input sequence. Furthermore, the input embedding is an operation that divides the image into patches and flattens them.

[0012] The encoding module described in 1.2) consists of N sub-modules, the core of which is a multi-head attention module. In the multi-head attention module, it is a combination of various self-attention mechanisms, as shown in Equation (3). The input vector X, which has been converted into a one-dimensional vector in step 1.1), is input into the encoding module. The input vector X is first converted into three different vectors, namely the query vector Q, the key vector K, and the numerical vector V, as shown in Equation (3).

[0013]

[0014] in, W q W k W v These are the optimization parameters. QK T Obtain attention weights. k The dimension is K to ensure the gradient stability of Softmax. V is a weighted sum, while remaining consistent with the input. Multi-head attention modules allow the model to focus on different aspects from multiple subspaces. A transformer encoder is constructed by concatenating N multi-head attention modules. A total of N sub-modules are constructed to form the transformer encoder.

[0015] The second step involves introducing global contrastive learning loss and local image masking loss as surrogate tasks for self-distillation training. This approach focuses on both the global differences between different targets in marine aquaculture, land, and seawater to facilitate subsequent multi-target differentiation, and the local details of each target to facilitate accurate multi-target segmentation.

[0016] First, construct two identical transformer encoders, as described in the first step, one of which is called the teacher network. Another one is called the student network. The student network is trained and optimized by using view changes and random masks, combined with global contrastive learning loss and local image mask loss. Backpropagation is prohibited to optimize the teacher network. However, since the network structures of the teacher network and the student network are the same, the parameter updates of the teacher network are dynamically updated based on the parameters of the student network. See Section 2.3 for details.

[0017] 2.1) Global contrastive loss is based on contrastive learning, which maximizes the similarity features of the same image under different views.

[0018] The 512×512 image collected in the first step is used as the input image x. Random image augmentation is then performed to obtain two different views, u and v. Both u and v are input into the teacher network and the student network, respectively. After the input embedding, a random mask is applied to the embedded image again (i.e., a portion of the n values ​​obtained in Section 1.1 is randomly extracted and forced to zero), resulting in the masked embeddings of the two views. (Results obtained from student network input embedding) and unmasked embedding u c ,v c (Results obtained from teacher network input embedding), from teacher network and student network The results are input into the softmax function, as shown in equations (4) and (5).

[0019]

[0020] Where i and k are the probability distributions of the network output along the K dimensions. C is the center value, which is the average value calculated from the batches of teacher network outputs to avoid collapse. τ t This represents a temperature parameter of the teacher network.

[0021]

[0022] Where i and k are the network output probability distributions along the K dimensions. τ s This represents the temperature parameter of the student network.

[0023] The input to the student network is different mask views. and These two networks have the same structure but different parameters. Under the gradient-stopped teacher network parameters, a global contrastive learning loss is used to expect the outputs of the teacher and student networks to match, as shown in Equation (6), to optimize the student network parameters.

[0024]

[0025] Among them, L CLS It is the contrastive learning loss function. Specifically, t c1 It is a teacher network view u c Contrastive learning features SF t (u c ,v c ), t c2 It is a teacher network view v c Contrastive learning features SF t (u c ,v c ), s c1 It is a student network mask view Contrastive learning features s c2 It is a student network mask view Contrastive learning features

[0026] 2.2) Local image mask loss is based on mask reconstruction and restoration, which restores the unmasked image from the masked image and obtains the essential local features of the image.

[0027] Different views u and v are input into the network, and the images are converted into one-dimensional vectors through an input embedding operation. and To obtain image masks for different views, where i∈{1,..,n}, and n is the number of embeddings (same as the input of n embedding values ​​in Section 1.1). Additionally, generate random masks m∈{0,1}. n The number of elements in the batch, the same as the number of input embeddings, consists of 0s and 1s and is used to record the positions of randomly selected portions of the n values ​​in the input embedding. Each view... and The input embedding is achieved through the corresponding random mask m u and m v Obtain mask view and mask image and unmasked image u m ,v m Input two networks (student network) Teacher Network The results are fed into the Softmax function, as shown in formulas (7) and (8).

[0028]

[0029]

[0030] Where i and k are the network output probability distributions in the K dimensions, τ′ t It is the teacher network temperature parameter, τ′ s C′ is the temperature parameter of the student network, and C′ is the center value.

[0031] Subsequently, based on the idea of ​​image masking, a loss function L was defined. MIS As shown in formula (9).

[0032]

[0033] Among them, t m1 It is a teacher network separation view u m The obtained mask image features SF t ′(u m ,v m );s m1 It is a student network separation mask view Mask image features t m2 It is a teacher network separation view v mThe obtained mask image features SF t ′(u m ,v m );s m2 It is a student network separation mask view Mask image features m∈{0,1} n Let m be a random mask. u,v For a random mask, m u A random mask for view u; m v This is a random mask for view v.

[0034] 2.3) Encoder network parameter update: Based on the contrastive learning loss function and image mask loss function obtained above, backpropagation optimization is performed on the student network. Backpropagation does not optimize the teacher network, but since the teacher network and student network have the same structure, the teacher network parameter θ is updated. t Based on student network parameters θ s The exponential moving average method is used for updating, as shown in equation (10).

[0035] θ t ←λθ t +(1-λ)θ s (10)

[0036] Here, λ is a momentum parameter ranging from 0.996 to 1. Contrastive learning loss and image masking loss are used as objective functions. Backpropagation optimizes the student network, pausing the backpropagation of the teacher network. Then, the teacher network parameters are updated in real-time using the parameters optimized by the student network in each round. Finally, the trained teacher network is used as a transformer encoder. This network-to-network learning approach allows the network to learn more essential features, transforming labels from either 0 or 1 to soft labels in decimal form. Furthermore, through multi-objective loss functions, the network can focus on different types of information. Image masking loss focuses on uncovering local image information, reconstructing original features from locally important features, resulting in better performance in distinguishing target local information. Contrastive learning loss focuses on uncovering global image information, treating an image as a class and using different views of the entire image for global comparison, making the same category more similar and reducing intra-class differences, resulting in better performance in multi-objective distinction. The third step proposes a semantic enhancement feature fusion module, which can enhance deep semantic information while preserving shallow shape information, including a deep semantic enhancement module and a multi-layer feature fusion module. To reduce misclassifications caused by large intra-class differences and small inter-class differences among multiple targets, the deep semantic enhancement module focuses on deep semantic continuity, improving intra-class continuity of the segmented image by fusing high-level features. The multi-layer feature fusion module utilizes shape features extracted from low-level layers and features extracted from different layers to preserve target edges and reduce inter-class misclassifications.

[0037] In the second step, a trained teacher network was obtained as a transformer encoder. However, the encoder only obtains the essential features of the image, so a decoder is needed to perform feature decoding and generate semantic segmentation results. The transformer encoder structure and its weight parameters were fixed, and a semantic enhancement feature fusion decoder was built to make reasonable use of the transformer encoder features. Based on the different characteristics of shallow and deep features, a deep semantic enhancement module and a multi-layer feature fusion module were designed.

[0038] 3.1) Deep semantic enhancement relies on the higher layers of the transformer encoder (i.e., the last three sub-modules of the N=12 sub-modules in Section 1.2). STFF enhances the high-level semantic information to include more of the higher layers of the transformer encoder. Features E of different layers... m It is obtained from the transformer encoder in the previous step and contains rich semantic information. Deep semantic enhancement includes E m The feature of m≥10 utilizes the fully connected FC(·) and concatenation concat(·) operations, as shown in equations (11)-(14).

[0039] C 10 =FC(concat(E) 10 ,FC(concat(E 11 E 12 (11)

[0040] C 11 =FC(concat(C 10 ,FC(concat(E 11 E 12 (12)

[0041] C 12 =FC(concat(C 11 E 12 (13)

[0042] F high =FC(concat(C 10 C 11 C 12 (14)

[0043] Where C is an enhanced intermediate variable; C 10 This is an enhanced intermediate variable for the 10th submodule; C 11 C 10 Enhance intermediate variables to higher levels E 11 E 12Enhancement intermediate variables of diffusion; C 12 C 11 To higher level E 12 Enhancement intermediate variables of diffusion; F high This is a feature for enhancing deep semantics.

[0044] The enhanced high-level features are used as the final deep semantic enhancement features F. high The data is then fed into the Pyramid Pooling Module (PPM), where it is diffused to lower-level features. The PPM utilizes different pooling sizes to obtain the extensive high-level semantic information F from the previous step. PPM As shown in equation (15).

[0045] F PPM =concat(ξ(Pool(F) high ,size=1,2,3,6))) (15)

[0046] Here, Pool(·) is the pooling operation, and ξ(·) is a bilinear upsampling function.

[0047] 3.2) Multi-layer feature fusion diffuses deep semantic enhancement into low-level features to reduce noise in these features. First, the low-level features are upsampled individually; the number of upsampling operations varies depending on the specific low-level feature. Different layer features E are obtained from the transformer encoder. m It contains both low-level and high-level information. Multi-level feature fusion includes low-level features E obtained from the low-level layers of the transformer encoder. m m = {4, 6, 8} and from deep semantic enhancement F high The obtained high-level features F PPM As shown in equations (16) and (17).

[0048] F4=ξ′(BN(σ(ξ′(E4)))) (16)

[0049] F6=ξ′(E6) (17)

[0050] Where σ(·) is the activation function, ξ′(·) is the upsampling of the transposed convolution, and BN is batch normalization. Unlike the previous bilinear upsampling, since the feature vector is obtained from the transformer encoder, it needs to be adapted to the downstream visual task through transposed convolution. After obtaining features of different layers and sizes, the decoder needs to diffuse the deep features to the low layers and fuse the diffused low-layer features with the high-layer features, as shown in Equations (18) to (21).

[0051] A8=φ(ξ(φ(F PPM))+E8) (18)

[0052] A6=φ(ξ(A8)+F6) (19)

[0053] A4=φ(ξ(A6)+F4) (20)

[0054] Y out =σ(BN(φ(concat(A4,A6,A8,φ(F PPM ))))) (twenty one)

[0055] Here, φ(·) is a 3×3 convolution filter.

[0056] Finally, the transformer encoder parameters obtained in the second step are fixed (i.e., the teacher network weights optimized through backpropagation to achieve stability). Since the teacher network, acting as the transformer encoder, has already obtained its trained weight parameters in the second step, the transformer encoder needs to be fixed when optimizing the encoder weight parameters in the third step. When optimizing the decoder weight parameters, since the essential image features have been obtained from the second step, the decoder only needs to be optimized through backpropagation using a small number of labeled samples, similar to supervised learning. Cross-entropy loss is applied using labels and segmentation results, and the semantic enhancement feature fusion decoder is fine-tuned using a small number of labeled samples. The trained decoder weight parameters are then fixed, and the accurate extraction results of mariculture from SAR images of the entire scene with multiple targets are obtained through testing.

[0057] The beneficial effects of this invention are as follows:

[0058] This invention provides a method for multi-target extraction in Maritime Aquaculture using a self-supervised transformer based on semantically enhanced feature fusion. Addressing challenges such as the difficulty in obtaining labeled samples from Maritime Aquaculture remote sensing images, the large size of the entire image and imbalanced target samples, intra-class discontinuities among multiple targets, and inter-class misclassification, this invention utilizes a transformer encoder to obtain long-term dependent features, alleviating the sample imbalance problem. Self-distillation learning is employed to extract salient features from a large amount of unlabeled data, mitigating the difficulty in obtaining labeled samples. A semantically enhanced feature fusion decoder, including deep semantic enhancement, low-level information diffusion, and multi-level feature fusion operations, effectively improves discrimination capabilities, accurately extracts target edge information, avoids speckle noise interference, reduces inter-class misclassification, and improves intra-class continuity segmentation. The proposed method offers high accuracy and is feasible for Maritime Aquaculture monitoring with a large number of targets, a limited number of labels, and the ability to capture multiple targets within a single Maritime Aquaculture image. Attached Figure Description

[0059] Figure 1 This is a general block diagram of a whole-scene SAR multi-target extraction method for marine aquaculture based on semantic enhancement feature fusion and self-supervised transformer.

[0060] Figure 2 The images show the results of aquaculture extraction from the entire SAR imagery: (a) is the original image of the entire scene, (b) is the ground truth image of the entire scene, and (c) is the final image of the entire scene.

[0061] Figure 3 For the extracted detail images, (a)-(d) are the original images of the detail region AD, (e)-(h) are the ground truth images of the detail region AD, and (i)-(l) are the extracted result images of the detail region AD. Detailed Implementation

[0062] To make the problem solved by the present invention, the method adopted, and the effect achieved by the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present invention are shown in the accompanying drawings, not all of them.

[0063] like Figure 1 As shown in the embodiment of the present invention, a method for extracting multiple targets in marine aquaculture using semantically enhanced feature fusion and self-supervised transformer for whole-scene SAR includes:

[0064] Compiled on a Windows 10 system using Python 3.6.12, PyTorch 1.7.1, and CUDA 11.0, and run on an RTX 3080 GPU.

[0065] The first step involves collecting a large amount of unlabeled remote sensing monitoring data for marine aquaculture, slicing it into 512×512 images, and then using a marine aquaculture transformer encoder to discover salient features through self-distillation training and obtain long-term dependent features. This effectively addresses the sample imbalance caused by the large proportion of seawater area in the overall remote sensing image during marine aquaculture monitoring, reducing information loss. The marine aquaculture transformer encoder consists of an input embedding module and an encoding module, as detailed below:

[0066] The input embedding described in 1.1) is the process of encoding an image into a one-dimensional vector. Each two-dimensional image is first divided into n smaller image blocks. Considering the size of a marine aquaculture image, the number of image blocks n is 196. Through convolution operation with a kernel size the same as the image block size, the n image blocks are embedded into n values ​​and arranged into a one-dimensional vector. To prevent information loss, feature upsizing is performed simultaneously with the transformation into a one-dimensional vector. In addition, to prevent the loss of positional information during the embedding process, positional embedding is introduced, as shown in equations (1) and (2).

[0067]

[0068]

[0069] Where pos represents the position of the image patch in the image, i represents the dimension index value of the image patch, and d model This represents the total dimension of the image patch, where the dimension is 768. Location embeddings are added to the input embedding to capture location information in the input sequence. Furthermore, the input embedding is an operation that divides the image into patches and flattens them.

[0070] The encoding module described in 1.2) consists of N sub-modules, the core of which is a multi-head attention module. In the multi-head attention module, it is a combination of various self-attention mechanisms, as shown in Equation (3). The input vector X, which has been converted into a one-dimensional vector in step 1.1), is input into the encoding module. The input vector X is first converted into three different vectors, namely the query vector Q, the key vector K, and the numerical vector V, as shown in Equation (3).

[0071]

[0072] in, W q W k W v These are the optimization parameters. QK T Obtain attention weights. k =384 is the dimension of K to ensure the gradient stability of Softmax. V is weighted and remains consistent with the input. The multi-head attention module allows the model to focus on different aspects from multiple subspaces. A transformer encoder is constructed by concatenating N multi-head attention modules, resulting in a total of N sub-modules forming the transformer encoder. The transformer encoder converts the image into a one-dimensional vector and, through a self-attention mechanism, discovers long-term dependent features, effectively alleviating the label sample imbalance problem caused by the majority of seawater samples in whole-scene remote sensing image segmentation.

[0073] The second step involves introducing global contrastive learning loss and local image masking loss as surrogate tasks for self-distillation training. This approach focuses on both the global differences between different targets in marine aquaculture, land, and seawater to facilitate subsequent multi-target differentiation, and the local details of each target to facilitate accurate multi-target segmentation.

[0074] First, construct two identical transformer encoders, as described in the first step, one of which is called the teacher network. Another one is called the student network. The student network is trained and optimized by using view changes and random masks, combined with global contrastive learning loss and local image mask loss. Backpropagation is prohibited to optimize the teacher network. However, since the network structures of the teacher network and the student network are the same, the parameter updates of the teacher network are dynamically updated based on the parameters of the student network. See Section 2.3 for details.

[0075] 2.1) Global contrastive loss is based on contrastive learning, maximizing the similarity features of the same image under different views. The 512×512 image collected in the first step is used as the input image x. Random image augmentation is then performed to obtain two different views u and v. Both u and v are input into the teacher network and student network, respectively. After embedding, a random mask is applied to the embedded image again (i.e., a portion of the n values ​​obtained in Section 1.1 is randomly extracted and forced to zero), resulting in the masked embeddings of the two views. (Results obtained from student network input embedding) and unmasked embedding u c ,v c (Results obtained from teacher network input embedding), from teacher network and student network The results are input into the softmax function, as shown in equations (4) and (5).

[0076]

[0077] Where i and k are the probability distributions of the network output along the K dimensions. C is the center value, which is the average value calculated from the batches of teacher network outputs to avoid collapse. τ t =0.04 represents a temperature parameter of the teacher network.

[0078]

[0079] Where i and k are the network output probability distributions along the K dimensions. τ s =0.1 represents the temperature parameter of the student network. The input to the student network is different mask views. and These two networks have the same structure but different parameters. Under the gradient-stopped teacher network parameters, a global contrastive learning loss is used to expect the outputs of the teacher and student networks to match, as shown in Equation (6), to optimize the student network parameters.

[0080]

[0081] Among them, L CLS It is the contrastive learning loss function. Specifically, t c1 It is a teacher network view u c Contrastive learning features SF t (uc ,v c ), t c2 It is a teacher network view v c Contrastive learning features SF t (u c ,v c ), s c1 It is a student network mask view Contrastive learning features s c2 It is a student network mask view Contrastive learning features

[0082] 2.2) Local image masking loss is based on mask reconstruction and restoration, which restores the unmasked image from the masked image and obtains the essential features of the local image. The aquaculture area is randomly masked and input into two networks respectively with the unmasked image. Local image masking loss expects the masked features and the unmasked features to be as similar as possible to obtain the essential features of aquaculture.

[0083] Different views u and v are input into the network, and the images are converted into one-dimensional vectors through an input embedding operation. and To obtain image masks for different views, where i∈{1,..,n}, and n is the number of embeddings (same as the input of n embedding values ​​in Section 1.1). Additionally, generate random masks m∈{0,1}. n The number of elements in the batch, the same as the number of input embeddings, consists of 0s and 1s and is used to record the positions of randomly selected portions of the n values ​​in the input embedding. Each view... and The input embedding is achieved through the corresponding random mask m u and m v Obtain mask view and mask image and unmasked image u m ,v m Input two networks (student network and teacher network), and the results are fed into the Softmax function, as shown in formulas (7) and (8).

[0084]

[0085]

[0086] Where i and k are the network output probability distributions in the K dimensions, τ′ t It is the teacher network temperature parameter, τ′ s C′ is the temperature parameter of the student network, and C′ is the center value.

[0087] Subsequently, based on the idea of ​​image masking, a loss function L was defined. MIS As shown in formula (9).

[0088]

[0089] Among them, t m1 It is a teacher network separation view u m The obtained mask image features SF t ′(u m ,v m ), s m1 It is a student network separation mask view Mask image features t m2 It is a teacher network separation view v m The obtained mask image features SF t ′(u m ,v m );s m2 It is a student network separation mask view Mask image features m∈{0,1} n For a random mask, m u,v For a random mask, m u A random mask for view u; m v This is a random mask for view v.

[0090] 2.3) Encoder network parameter update: Based on the contrastive learning loss function and image mask loss function obtained above, backpropagation optimization is performed on the student network. Backpropagation does not optimize the teacher network, but since the teacher network and student network have the same structure, the teacher network parameter θ is updated. t Based on student network parameters θ s The exponential moving average method is used for updating, as shown in equation (10).

[0091] θ t ←λθ t +(1-λ)θ s (10)

[0092] Here, λ = 0.996 is the momentum parameter ranging from 0.996 to 1. Contrastive learning loss and image masking loss are used as objective functions. Backpropagation optimizes the student network, while the backpropagation of the teacher network is paused. Then, the teacher network parameters are updated in real-time using the parameters optimized by the student network in each round. Finally, the trained teacher network is used as the transformer encoder. This network-to-network learning approach allows the network to learn more fundamental features, transforming labels from purely zero-to-one to soft labels in decimal form. Furthermore, through multi-objective loss functions, the network can focus on different types of information. Image masking loss emphasizes uncovering local image information, reconstructing original features from locally important features, resulting in better performance in distinguishing target local information. Contrastive learning loss emphasizes uncovering global image information, treating an image as a class and using different views of the entire image for global comparison, making the same category more similar and reducing intra-class differences, resulting in better performance in multi-objective distinction.

[0093] Global contrast loss and local mask loss can effectively uncover the essential features of marine aquaculture. When distinguishing between marine aquaculture, seawater, and land targets, more attention should be paid to global differences. That is, global contrast loss can effectively distinguish different targets. When accurately segmenting and extracting marine aquaculture, local mask loss can obtain essential features that are more conducive to local information, providing a foundation for the subsequent decoder to accurately extract aquaculture edges and enhance semantic continuity.

[0094] The third step proposes a semantic enhancement feature fusion module, which enhances deep semantic information while preserving shallow shape information. This module includes a deep semantic enhancement module and a multi-layer feature fusion module. To reduce misclassifications caused by large intra-class differences and small inter-class differences among multiple targets, the deep semantic enhancement module focuses on deep semantic continuity by fusing high-level features to improve intra-class continuity of the segmented image. The multi-layer feature fusion module utilizes shape features extracted from low-level layers and features extracted from different layers to preserve target edges and reduce inter-class misclassifications.

[0095] In the second step, a trained teacher network was obtained as a transformer encoder. However, the encoder only obtains the essential features of the image, so a decoder is needed to perform feature decoding and generate semantic segmentation results. The transformer encoder structure and its weight parameters were fixed, and a semantic enhancement feature fusion decoder was built to make reasonable use of the transformer encoder features. Based on the different characteristics of shallow and deep features, a deep semantic enhancement module and a multi-layer feature fusion module were designed.

[0096] 3.1) Deep semantic enhancement relies on the higher layers of the transformer encoder (i.e., the last three sub-modules of the N=12 sub-modules in Section 1.2). STFF enhances the high-level semantic information to include more of the higher layers of the transformer encoder. Features E of different layers... m It is obtained from the transformer encoder in the previous step and contains rich semantic information. Deep semantic enhancement includes E m The feature of m≥10 utilizes the fully connected FC(·) and concatenation concat(·) operations, as shown in equations (11)-(14).

[0097] C 10 =FC(concat(E) 10 ,FC(concat(E 11 E 12 (11)

[0098] C 11 =FC(concat(C 10 ,FC(concat(E 11 E 12 (12)

[0099] C 12 =FC(concat(C 11 E 12 (13)

[0100] F high =FC(concat(C 10 C 11 C 12 (14)

[0101] Where C is an enhanced intermediate variable; C 10 This is an enhanced intermediate variable for the 10th submodule; C 11 C 10 Enhance intermediate variables to higher levels E 11 E 12 Enhancement intermediate variables of diffusion; C 12 C 11 To higher level E 12 Enhancement intermediate variables of diffusion; F high This is a feature for enhancing deep semantics.

[0102] The enhanced high-level features are used as the final deep semantic enhancement features F. highThe data is then fed into the Pyramid Pooling Module (PPM), where it is diffused to lower-level features. The PPM utilizes different pooling sizes to obtain the extensive high-level semantic information F from the previous step. PPM As shown in equation (15).

[0103] F PPM =concat(ξ(Pool(F) high ,size=1,2,3,6))) (15)

[0104] Here, Pool(·) is the pooling operation, and ξ(·) is a bilinear upsampling function.

[0105] 3.2) Multi-layer feature fusion diffuses deep semantic enhancement into low-level features to reduce noise in these features. First, the low-level features are upsampled individually; the number of upsampling operations varies depending on the specific low-level feature. Different layer features E are obtained from the transformer encoder. m It contains both low-level and high-level information. Multi-level feature fusion includes low-level features E obtained from the low-level layers of the transformer encoder. m m = {4, 6, 8} and from deep semantic enhancement F high The obtained high-level features F PPM As shown in equations (16) and (17).

[0106] F4=ξ′(BN(σ(ξ′(E4)))) (16)

[0107] F6=ξ′(E6) (17)

[0108] Where σ(·) is the activation function, ξ′(·) is the upsampling of the transposed convolution, and BN is batch normalization. Unlike the previous bilinear upsampling, since the feature vector is obtained from the transformer encoder, it needs to be adapted to the downstream visual task through transposed convolution. After obtaining features of different layers and sizes, the decoder needs to diffuse the deep features to the low layers and fuse the diffused low-layer features with the high-layer features, as shown in Equations (18)-(21).

[0109] A8=φ(ξ(φ(F PPM ))+E8) (18)

[0110] A6=φ(ξ(A8)+F6) (19)

[0111] A4=φ(ξ(A6)+F4) (20)

[0112] Y out=σ(BN(φ(concat(A4,A6,A8,φ(F PPM ))))) (twenty one)

[0113] Here, φ(·) is a 3×3 convolution filter.

[0114] Finally, the transformer encoder parameters obtained in the second step are fixed (i.e., the teacher network weights optimized through backpropagation to achieve stability). Since the teacher network, acting as the transformer encoder, has already obtained its trained weight parameters in the second step, the transformer encoder needs to be fixed when optimizing the encoder weight parameters in the third step. When optimizing the decoder weight parameters, since the essential image features have been obtained from the second step, the decoder only needs to be optimized through backpropagation using a small number of labeled samples, similar to supervised learning. Cross-entropy loss is applied using labels and segmentation results, and the semantic enhancement feature fusion decoder is fine-tuned using a small number of labeled samples. The trained decoder weight parameters are then fixed, and the accurate extraction results of mariculture from SAR images of the entire scene with multiple targets are obtained through testing.

[0115] The semantic enhancement feature fusion module's decoder expands the receptive field, obtaining information with different emphases from different encoder layers. Lower layers focus more on information such as farming edges and shapes, while higher layers focus more on farming semantic information—key information for determining farming—to enhance the continuity of the farming area. Furthermore, in whole-scene segmentation, due to computational limitations, the input is cropped into smaller images, making the semantic continuity of the target area equally important. Therefore, before multi-layer fusion, high-level semantics are enhanced to obtain deep semantic enhancement features. (Whole-scene segmentation image shown below.) Figure 2 Detailed segmentation diagram as follows Figure 3 As shown, from Figure 2 As can be seen, the overall scene segmentation effect is excellent. Figure 2 (c) The extracted result is a mask of the original image, which largely matches the ground truth image. From Figure 3 This shows that when facing the problem of goal continuity, such as Figure 3 As shown in (b), it can handle dark areas on land quite well. Extraction is performed facing aquaculture areas, such as... Figure 3 As shown in (c), the edges of the aquaculture area can be accurately extracted. Figure 3 As shown in (d), the land edge also shows good extraction results, such as Figure 3 As shown in (a), the model also performs well when faced with multi-target (seawater, land, aquaculture) segmentation, and can accurately identify land areas that exist at the edge of the image.

[0116] This model can obtain long-term temporal dependency features from a large amount of unlabeled data. While ensuring high-level semantics, it learns target shape information by fusing with low-level features, accurately extracts target edges, avoids mutual misjudgment between multiple targets, and enhances intra-class spatial continuity. Thus, it realizes the extraction of marine aquaculture information from a self-supervised transformer model for multiple targets in the whole scene. It can quickly obtain accurate spatial distribution information of marine aquaculture without relying on a large number of samples, alleviate the problem of data imbalance between classes, and provide valuable resources for the research on marine aquaculture evaluation and management and its interaction mechanism with the environment.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the method solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications to the method solutions described in the foregoing embodiments, or equivalent substitutions for some or all of the method features, do not cause the essence of the corresponding method solutions to deviate from the scope of the method solutions of the embodiments of the present invention.

Claims

1. A method for extracting multiple targets in marine aquaculture using whole-scene SAR based on semantic enhancement feature fusion and self-supervised transformer, characterized in that, It includes the following steps: The first step involves collecting a large amount of unlabeled remote sensing monitoring data for marine aquaculture, slicing it into multiple 2D images, and then using a marine aquaculture transformer encoder trained via self-distillation to discover salient features and obtain long-term dependency features. The marine aquaculture transformer encoder includes input embedding and encoding modules, specifically: Step 1.1) describes the input embedding process, which encodes an image into a one-dimensional vector. The encoding module mentioned in step 1.2) consists of N It consists of several sub-modules, the core of each of which is a multi-head attention module. The multi-head attention module allows the model to pay attention to information from different aspects of multiple subspaces, through... N A multi-head attention module is connected in series to construct a transformer encoder; a total of [number] modules are constructed. N The transformer encoder is composed of several sub-modules. The second step is to train the marine aquaculture transformer encoder based on self-distillation learning of knowledge distillation in self-supervised learning, combined with global contrastive learning loss and local image mask loss. The third step involves introducing a semantic enhancement feature fusion module to enhance deep semantic information while preserving shallow shape information. This includes a deep semantic enhancement module and a multi-layer feature fusion module. The semantic enhancement feature fusion decoder is fine-tuned using a small number of labeled samples. The weight parameters of the trained decoder are then fixed, and tests are conducted to obtain accurate extraction results for mariculture from SAR images of the entire scene with multiple targets. Specifically: Step 3.1) Deep semantic enhancement relies on the higher levels of the transformer encoder, which refers to the concatenation in step 1.2). N The last three sub-modules of the STFF sub-modules incorporate more high-level semantic information from the transformer encoder through feature enhancement; features from different layers E m Obtained by the transformer encoder, deep semantic enhancement includes Its features utilize fully connected splicing The operation is shown in equations (11) to (14); (11) (12) (13) (14) in, C To enhance intermediate variables; C 10 This is an enhanced intermediate variable for the 10th submodule; C 11 C 10 Enhance intermediate variables to higher levels E 11 E 12 Enhancement intermediate variables of diffusion; C 12 C 11 To higher level E 12 Enhancement intermediate variables of diffusion; F high For deep semantic enhancement features; The enhanced high-level features are used as the final deep semantic enhancement features. The data is then fed into the pyramid pooling module (PPM) and subsequently diffused to low-level features. PPM utilizes different pooling sizes to obtain a wide range of high-level semantic information from the previous step. As shown in equation (15); (15) in, For pooling operations, It is a bilinear upsampling function; Step 3.2) Multi-layer feature fusion diffuses deep semantic enhancement into low-level features to reduce noise in these features. First, the low-level features are upsampled individually, with the number of upsampling operations varying depending on the specific low-level feature. Different layer features are then obtained from the transformer encoder. E m It contains both low-level and high-level information; multi-level feature fusion includes low-level features obtained from the low-level layers of the transformer encoder. and from deep semantic enhancement High-level features obtained As shown in equations (16) and (17); (16) (17) in, It is an activation function. It is upsampling of transposed convolution, and BN is batch normalization; After obtaining features of different layers and sizes, the decoder needs to diffuse the deep features to the lower layers and fuse the diffused lower-layer features with the higher-layer features, as shown in Equations (18) to (21). (18) (19) (20) (21) in, It is a 3×3 convolutional filter; Step 3.3) Finally, fix the transformer encoder parameters obtained in the second step, optimize the decoder by backpropagation with a small number of sample labels, perform cross-entropy loss with labels and segmentation results, fine-tune the semantic enhancement feature fusion decoder with a small number of labeled samples, fix the trained decoder weight parameters, and test to obtain accurate extraction results of marine aquaculture from SAR images of the whole scene with multiple targets.

2. The method for extracting multiple targets in marine aquaculture using a self-supervised transformer based on semantic enhancement feature fusion for whole-scene SAR, as described in claim 1, is characterized in that... The specific details of the marine aquaculture transformer encoder in the first step are as follows: Step 1.1) Input embedding: Each two-dimensional image is first segmented into n Smaller image patches are processed through convolution operations with kernel sizes the same as the image patch size. n Each image patch is embedded as n The values ​​are arranged into a one-dimensional vector. To prevent information loss, feature dimensionality is increased while transforming into a one-dimensional vector. In addition, in order to prevent the loss of location information during the embedding process, location embedding is introduced, and the formulas are shown in equation (1) and equation (2); (1) (2) in, Indicates the position of the image patch in the image. This represents the dimension index value of the image patch. The total dimension of the image patch is represented; positional embedding is added to the input embedding to capture positional information in the input sequence; furthermore, the input embedding is the operation of dividing the image into image patches and flattening them; The encoding module described in step 1.2): The core of each sub-module is a multi-head attention module; the input vector, which has been converted into a one-dimensional vector in step 1.1), is... X Input encoding module, input vector X First, it is converted into three different vectors, namely the query vector. Q Key vectors K and numerical vectors V The formula is shown in equation (3); (3) in, , , , These are the optimization parameters; Obtain attention weights; yes K The dimension is determined to ensure the gradient stability of Softmax; V Obtain the weighted sum while maintaining consistency with the input; The multi-head attention module allows the model to focus on different aspects of information from multiple subspaces, through N A multi-head attention module is connected in series to construct a transformer encoder; a total of [number] modules are constructed. N The transformer encoder is composed of several sub-modules.

3. The method for extracting multiple targets in marine aquaculture using a self-supervised transformer based on semantic enhancement feature fusion for whole-scene SAR, as described in claim 1, is characterized in that... In the second step, two identical transformer encoders are first constructed, as described in the first step, one of which is called the teacher network. Another one is called the student network. The student network is trained and optimized using view variations and random masks, combined with global contrastive learning loss and local image masking loss, while the teacher network is optimized by prohibiting backpropagation; details are as follows: Step 2.1) Global contrastive loss is based on contrastive learning, which maximizes the similarity features of the same image under different views; The multiple 2D images collected in the first step are used as input images. x Then, random image enhancement is performed to obtain two different views. u and v ,Will u and v Both the teacher network and the student network are input. After the input is embedded, a random mask is applied to the embedded image again to obtain the masked embeddings of the two views. and unmasked embedding ,in The results obtained by embedding the student network input. Input the embedded results into the teacher network; then input the teacher network... and student network The results are input into the softmax function, as shown in equations (4) and (5); (4) in, i and k yes K Network output probability distribution in dimensionality; C It is the central value, that is, the average value calculated from the batch output by the teacher network, in order to avoid collapse; This represents a temperature parameter in the teacher network. (5) in, i and k yes K Network output probability distribution in dimensionality; This indicates the temperature parameter of the student network; The teacher network and the student network have the same structure but different parameters; under the gradient-stopped teacher network parameters, the global contrastive learning loss is used to expect the outputs of the teacher and student networks to match, as shown in Equation (6), in order to optimize the student network parameters; (6) in, It is a contrastive learning loss function; specifically, It is a teacher network view Contrastive learning features , It is a teacher network view Contrastive learning features , It is a student network mask view Contrastive learning features ; It is a student network mask view Contrastive learning features ; Step 2.2) Local image mask loss is based on mask reconstruction and restoration, which restores the unmasked image from the masked image and obtains the essential local features of the image; Different views u and v The input network converts the image into a one-dimensional vector through an input embedding operation. and To obtain image masks under different views, where , n The number of embeddings; in addition, a random mask is generated. The number of batches is the same as the number of input embeddings, consisting of 0s and 1s, used to record the input embeddings. n The position of a randomly selected portion of each value; each view and The input embedding is achieved through a corresponding random mask. m u and m v Obtain mask view and ; mask image and unmasked images Input two networks, including the student network. Teacher Network The results are fed into the Softmax function, as shown in formulas (7) and (8); (7) (8) in, i and k yes K Network output probability distribution in dimensionality It is a parameter related to the teacher's network temperature. This is the student network temperature parameter. It is the central value; Subsequently, based on the idea of ​​image masking, a loss function is defined. As shown in formula (9); (9) in, It is a teacher network separation view u m The obtained mask image features ; It is a student network separation mask view Mask image features ; It is a teacher network separation view v m The obtained mask image features ; It is a student network separation mask view Mask image features ; For a random mask, where, m u,v For random mask, m u For view u A random mask; m v For view v A random mask; Step 2.3) Encoder network parameter update: Based on the contrastive learning loss function and image mask loss function obtained above, backpropagation optimization is performed on the student network. Backpropagation does not optimize the teacher network, but since the teacher network and student network have the same structure, the teacher network parameters are updated accordingly. θ t Based on student network parameters θ s Update using the exponential moving average method, as shown in equation (10); (10) Where λ is a momentum parameter ranging from 0.996 to 1; By comparing the learning loss and the image mask loss as objective functions, backpropagation is used to optimize the student network. Backpropagation of the teacher network is paused. Then, the parameters of the teacher network are updated in real time using the parameters optimized by the student network in each round. Finally, the trained teacher network is used as the transformer encoder.

Images