SAR (Synthetic Aperture Radar) mariculture information extraction method of completely unsupervised Transform network

Abstract

The invention provides an SAR (Synthetic Aperture Radar) mariculture information extraction method of a completely unsupervised Transform network, and belongs to the technical field of crossing of marine remote sensing and artificial intelligence. In the first stage, general features of a target are extracted from a large number of unmarked SAR mariculture images, and an initialization weight is provided for a downstream segmentation network. In the second stage, a global optimal pseudo-label generator and a fine granularity discrimination and completion module are designed: the global optimal pseudo-label generator captures global similar semantic features, generates initial pseudo-labels, and eliminates interference of confusion targets while enhancing semantic correlation; and the local continuity of the pseudo tag is improved, and the detail and continuity semantic features of the target are complemented by regions. According to the method, the problems that massive unmarked mariculture intrinsic information cannot be deeply mined, and the segmentation performance of a traditional unsupervised method is limited in a complex scene can be solved, and a completely unsupervised mariculture segmentation task can be realized by taking continuously optimized pseudo labels as true values to train the segmentation network.

Description

Technical Field

[0001] The invention belongs to the cross-technical field of marine remote sensing and artificial intelligence, and relates to a SAR seawater aquaculture information extraction method of a completely unsupervised Transformer network. Background Art

[0002] China is rich in marine fishery resources, and its marine aquaculture output accounts for more than 50% of the world's total marine aquaculture output. However, unreasonable marine aquaculture planning may lead to serious water quality deterioration, causing significant losses to the marine ecological balance and aquaculture industry. Therefore, it is crucial to obtain accurate and comprehensive marine aquaculture location information, which provides a scientific basis for monitoring. Synthetic Aperture Radar (SAR) is widely used in marine monitoring tasks because of its ability to work continuously during the day and high resolution, and can obtain a large amount of marine remote sensing data. The scattering characteristics of marine aquaculture rafts are significantly different from those of seawater: marine aquaculture rafts mainly exhibit double-bounce scattering characteristics, while seawater mainly exhibits surface scattering characteristics. Therefore, in SAR images, marine aquaculture targets are usually brighter than the seawater background.

[0003] Although existing supervised methods can extract marine aquaculture targets more accurately, they usually rely on a large amount of labeled data. For example, the Chinese invention patent designs a method for extracting marine aquaculture areas based on deep learning (Chinese invention patent CN112766155A), which requires manual production of true value maps for each remote sensing image to train the network, and obtaining high-quality labeled data is time-consuming and laborious. At the same time, traditional unsupervised methods are affected by coherent speckle noise in SAR images, making it difficult to accurately extract aquaculture areas in complex marine aquaculture images. For example, the Chinese invention patent designs a SAR image semantic segmentation method based on deep convolutional networks and weakly supervised learning (Chinese invention patent CN109344837A), which uses the target features extracted by convolutional neural networks to generate pseudo labels as true value training networks, while convolutional neural networks ignore the global semantic information of the target. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention proposes a SAR marine aquaculture information extraction method based on a completely unsupervised transformer network, which can solve the problems of being unable to deeply mine the intrinsic information of massive unlabeled marine aquaculture and the limited segmentation performance of traditional unsupervised methods in complex scenarios. In the first stage, the present invention extracts the common features of the target from a large number of unlabeled SAR marine aquaculture images to provide initialization weights for the downstream segmentation network. In the second stage, a global optimal pseudo-label generator and a fine-grained discrimination and completion module are designed: the former captures globally similar semantic features and generates initial pseudo-labels, eliminating the interference of confusing targets while enhancing semantic relevance; the latter improves the local continuity of pseudo-labels and completes the details and continuity semantic features of the target by region. Finally, the segmentation network is trained with continuously optimized pseudo-labels as the true value to achieve a completely unsupervised marine aquaculture segmentation task.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A SAR marine aquaculture information extraction method using a completely unsupervised Transformer network includes two stages: upstream self-supervised pre-training and downstream unsupervised pseudo-label training. First, in the upstream stage, a large amount of unlabeled SAR image data is input into a self-supervised Transformer network for pre-training to obtain the trained network weights. Here, the self-supervised transformer network is defined as a self-supervised Transformer network. Next, in the downstream stage, a global optimal pseudo-label generator and a fine-grained discrimination and completion module are designed. Specifically: First, the network weights obtained by the upstream self-supervised pre-training are used to initialize the Teacher network, and the downstream training data is input into the Teacher network, and the initial pseudo-labels of the training images are generated by the global optimal pseudo-label generator. Secondly, the initial pseudo-labels are used as constraints to guide the training of the Student network. The image is segmented by the Student network, and its output is combined with the features learned by the Teacher network and input into the fine-grained discrimination and completion module. The initial pseudo-label is fine-grainedly optimized through the fine-grained discrimination and completion module, the detailed features of the aquaculture target are refined and its continuity information is improved, thereby generating an optimized pseudo-label. Finally, the optimized pseudo-label is used to replace the initial pseudo-label to constrain the segmentation network, and the output of the segmentation network in turn optimizes the pseudo-label. Through the iterative optimization mechanism of the pseudo-label, the quality of the pseudo-label and the performance of the segmentation network are continuously improved. Through this mutual optimization method, completely unsupervised SAR marine aquaculture information extraction is finally achieved. Specifically, the following steps are included:

[0007] The first step is to build an unlabeled marine aquaculture dataset, which includes the following steps:

[0008] Step 1.1, obtain SAR remote sensing image data from the satellite resource platform and preprocess the SAR remote sensing image, including radiation calibration, image enhancement and terrain correction.

[0009] Step 1.2, filtering the SAR remote sensing image processed in step 1.1 to reduce the influence of coherent speckle noise.

[0010] Step 1.3, crop the filtered SAR remote sensing image to a fixed size, generate a SAR marine aquaculture dataset, and divide it into an upstream self-supervised pre-training dataset and a downstream unsupervised pseudo-label training dataset in proportion. The downstream unsupervised pseudo-label training dataset is further divided into a training set and a test set, and the test set is labeled.

[0011] The second step is the upstream pre-training stage, where the self-supervised Transformer network is used to train the upstream self-supervised pre-training dataset, as follows:

[0012] Step 2.1, the self-supervised Transformer network consists of and Combination of The network parameters are , The network parameters are First, for each image in the upstream self-supervised pre-training dataset Perform random cropping to get a global view and partial views . The global view Input to Train the local view Input to Train in and then get and The two output probabilities are and .

[0013] Step 2.2, use formula (1) to calculate and The cross entropy loss , and as shown in formula (2), the calculated cross entropy loss is used as The loss function is back-propagated to update Parameters , formula (1) and formula (2) are as follows:

[0014] (1);

[0015] (2);

[0016] in, Indicates the length of the probability distribution vector output by the upstream pre-training model; express exist The predicted probability of each category; express In the The predicted probability of each category; express In the The predicted probability of the categories The logarithm of Represents the cross entropy loss function relative to parameter The gradient of Parameters; is the cross entropy loss pair Output Probability The gradient of Probability and Probability The error between yes Output Probability right parameter The partial derivative of .

[0017] Step 2.3, use the Exponential Moving Average (EMA) method to Network parameters renew Network parameters , as shown in formula (3):

[0018] (3);

[0019] in, is the smoothing factor of EMA, used to control The degree of influence of historical parameters.

[0020] Step 2.4: After the upstream self-supervised Transformer network training is completed, freeze Parameters .

[0021] The third step is to conduct downstream unsupervised pseudo-label training. The downstream training network consists of a Teacher network and a Student network. First, freeze the second step. Parameters As the initial weight of the Teacher network, the weight of the Teacher network , , and then use the Teacher network to extract features from the images in the downstream unsupervised pseudo-label training data set obtained in step 1.3. After that, the features output by the Teacher network are input into the global optimal pseudo-label generator to generate initial pseudo-labels, and the initial pseudo-labels are used as constraints to train the Student network, as follows:

[0022] Step 3.1, use the frozen upstream pre-training Parameters Initialize the parameters of the Teacher network. The Teacher network consists of N multi-head self-attention modules. Each multi-head self-attention module consists of H self-attention heads, which can extract features and perform global modeling on the input data. In the multi-head self-attention module, it contains query (Query, ), Key ) and Values ​​(Query, ), in the last layer of multi-head self-attention module, there is a special classification vector , used to mark the overall classification information.

[0023] Step 3.2, define the training images of the downstream unsupervised pseudo-label training dataset obtained in step 1.3 as ,Will Divide into P image blocks, input into the Teacher network, and extract features through N multi-head self-attention modules to obtain the final layer of multi-head self-attention feature representation , , image patch features Corresponding to the first The features of the image blocks, ,in and Respectively represent The image block is The query and key in each self-attention head.

[0024] Step 3.3: The last layer of multi-head self-attention features obtained in step 3.2 Input into the global optimal pseudo-label generator and use formula (4) to calculate the classification vector With image patch features The affinity vector between , use formula (5) to calculate the image block features With image patch features The affinity matrix between , formula (4) and formula (5) are as follows:

[0025] (4);

[0026] (5);

[0027] in, Indicates that in the self-attention head Down-classification vector With image patch features The affinity vector between ; is the dimension space of the calculation results, where is the number of image blocks, ; Represents the self-attention head Classification vector The query vector ; Indicates image patch in the self-attention head Keys in Take the transpose; Represents the currently calculated self-attention head; Indicates that in the self-attention head Below, the similarity matrix between image patch features; is the dimension space of the calculation results; Indicates image patch in the self-attention head Keys in And take the transpose of it.

[0028] Step 3.4, then add the affinity vector The image block with the highest affinity is defined as the key image block , . Perform calculations on each self-attention head in turn to obtain a set of key image blocks , is the number of self-attention heads. Among them, the key image block contains the most discriminative features of the training target. Next, for each key image block of the self-attention head Diffusion to the global set of patches In the global image block set Indicates In the self-attention heads, all image blocks with high similarity to the key image blocks are as shown in formula (6):

[0029] (6);

[0030] in, Indicates the image block index currently being calculated; represents a collection of image block indices, is the number of image blocks; Indicated in In the self-attention head, the key image block With image blocks The similarity of Extracted from.

[0031] Step 3.5, then, using the semantic similarity calculation - K-Means Clustering, for the global image patch set of each self-attention head obtained in step 3.4 The clusters are divided into two groups: aquaculture and seawater. The image block set of aquaculture is defined as , the set of image patches classified as seawater is defined as , use formula (7) to calculate the pseudo label of each self-attention head image block level ,in Indicates image blocks, After each block-level pseudo-label is generated, it can be spliced ​​into a pseudo-label at the whole image level according to the position of the image block in the whole image. , For the The pseudo labels of the whole image are obtained by the self-attention heads. Then the initial pseudo label set is obtained , , is the number of self-attention heads, formula (7) is as follows:

[0032] (7);

[0033] Step 3.6: For the initial pseudo-label set obtained in step 3.5 Filter and use formula (8) to calculate the entire image-level pseudo label corresponding to each attention head The density value , and then use formula (9) to calculate the image-level pseudo-label with the largest density value as the final initial pseudo-label , formulas (8) and (9) are as follows:

[0034] (8);

[0035] (9);

[0036] in, is the number of image blocks; For image blocks The average distance of image patches within the class to which they belong; For image blocks The average distance to its different category image patches; Indicates taking and The maximum value in The value range is [-1, 1].

[0037] In the fourth step, the segmentation results of the Student network and the feature maps output by the Teacher network are input into the fine-grained discrimination and completion module to further optimize the pseudo labels. The Student network consists of an encoder and a decoder. The encoder is based on a multi-scale Transformer architecture to extract the features of the input data. The decoder adopts a Unified Perceptual Parsing Head (UPerHead) structure, including feature fusion and multi-scale information aggregation units. The initial parameters of the Student network are , as follows:

[0038] Step 4.1: The downstream unsupervised pseudo-label training dataset images obtained in step 1.3 The enhanced image is obtained by data enhancement through rotation, scaling and linear stretching , the enhanced image The multi-scale features are input into the encoder of the Student network to extract the multi-scale features. The extracted multi-scale features are input into the decoder, and after step-by-step upsampling and feature fusion operations, the feature map is output. , then, the feature map is passed through After convolution processing, binarization is performed to finally obtain the output segmentation result. At the same time, the enhanced image Input into the Teacher network, the Teacher network learns the correlation between different spatial positions based on the multi-head self-attention module to generate a salient feature map , the salient feature map is passed through Convolution processing and binarization are performed to finally output the significant feature results .

[0039] Step 4.2, then, the segmentation result of the Student network And the significant feature results output by the Teacher network Input to the fine-grained discrimination and completion module. First, use the Canny edge detection operator to extract the segmentation results Boundary information of the target , boundary information Represented as a set, each element in the set Represents an independent target boundary in the segmentation result, . According to each boundary The coordinates of the upper left corner, upper right corner, lower left corner, and lower right corner of the four points generate the minimum bounding box set , where each minimum bounding box As shown in formula (10):

[0040] (10);

[0041] in, Indicates The minimum bounding box of the object boundary; bounding box Contains all pixels that meet the conditions ; Represents a pixel in the image, and its horizontal coordinate is , the vertical axis is ; Representing Boundaries The minimum value of the horizontal coordinates of all pixels in ; Representing Boundaries The maximum value of the horizontal coordinates of all pixels in ; Representing Boundaries The minimum value of the ordinate of all pixels in ; Representing Boundaries The maximum value of the horizontal coordinates of all pixels in ; It indicates the number of breeding areas in the image, which is also the number of boundaries.

[0042] Step 4.3: The number of boundaries and their corresponding bounding boxes in each image is determined by the number of objects in the image and is expressed in the form of a set. Using formula (11), we get In the bounding box Features within , using formula (12) we get In the bounding box Features within , formulas (11) and (12) are as follows:

[0043] (11);

[0044] (12);

[0045] in, Contains all the Pixel value ; Contains all satisfaction Pixel value ;

[0046] Step 4.4, then, the set of feature extraction results can be obtained from step 4.3, specifically: the feature set of the segmentation result output by the Student network in the bounding box The significant results of the Teacher network output are in the feature set of the bounding box ,in is the number of bounding boxes obtained in step 4.2. In the discrimination stage of the fine-grained discrimination and completion module, by inputting and For features within the same bounding box, the IoU (Intersection over Union) is calculated as shown in formula (13):

[0047] (13);

[0048] in, Express and Perform IoU calculation; Representation calculation and The intersection between pixels; Representation calculation and The union between pixels;

[0049] Step 4.4, finally, the fine-grained discrimination and completion module operates independently on the features within each bounding box. and The IoU between them exceeds the IoU threshold When and Otherwise, keep until each box is completed, and finally the optimized pseudo label is obtained. , as shown in formulas (14) and (15):

[0050] (14);

[0051] (15);

[0052] in, Represents a bounding box The characteristic results obtained in Indicates fusion and Features between; Represents the IoU threshold, ranging from [0, 1].

[0053] The fifth step is to establish the objective function of the downstream unsupervised pseudo-label training network and the network iterative optimization process. The details are as follows:

[0054] Step 5.1: Use formula (16) to calculate the initial pseudo-label generated by the global optimal pseudo-label Segmentation results output by Student network The cross entropy loss between . Formula (17) is also used to calculate the optimized pseudo label Segmentation results output by Student network The cross entropy loss between , formulas (16) and (17) are as follows:

[0055] (16);

[0056] (17);

[0057] in, Represents downstream unsupervised pseudo-label training images The total number of pixels.

[0058] Step 5.2, the overall objective function of the Student network is , and is the coefficient. and Number of training rounds with downstream unsupervised pseudo-label training network About, when hour, ;when hour, After that, the objective function As the loss of the Student network, the weight parameters of the Student network are reversely updated using formula (18): ; At the same time, formula (19) is used to use the weight parameter of the Student network through the exponential moving average (EMA) method Update the weight parameters of the Teacher network , formula (18) and formula (19) are as follows:

[0059] (18);

[0060] (19);

[0061] in, is the learning rate; is the parameter of the loss function relative to the network; is the smoothing factor of EMA, used to control The degree of influence of historical parameters.

[0062] Step 5.3, when steps 3.1 to 5.2 are completed, a round of training is completed. The training of the downstream unsupervised pseudo-label training network affects the Student network and the Teacher network through pseudo-labels, and the pseudo-labels are optimized by the prediction results of the Student network and the Teacher network, so as to achieve iterative updates of pseudo-labels. When the number of training rounds is reached, the downstream unsupervised pseudo-label training stops training and the segmentation output by the Student network is used as the final output of the network.

[0063] The beneficial effects of the present invention are:

[0064] (1) The present invention includes two stages: upstream self-supervised pre-training and downstream unsupervised pseudo-label training. The upstream self-supervised pre-training can extract intrinsic features from massive unlabeled marine aquaculture data sets. The downstream unsupervised pseudo-label training generates pseudo-labels with high-quality semantic features for training images through iterative optimization of the Teacher network and the Student network.

[0065] (2) We design a globally optimal semantic pseudo-label generator. It exploits the feature learning capabilities of different attention heads to diffuse from the most discriminative region of the target to the global context. This process captures similar semantic features across the entire region and generates initial pseudo-labels that enhance semantic relevance while eliminating interference that confuses the target.

[0066] (3) The present invention constructs a fine-grained discrimination and completion module to improve the local continuity of pseudo-labels. In the discrimination stage, the fine-grained discrimination and completion module calculates the overlapping part of the segmentation result of the Student network and the significance result of the Teacher network, thereby identifying the area that needs to be optimized. In the supplementation stage, the pseudo-label is enhanced by using the target details and continuity information in the sub-region. The experimental data are two SAR satellite marine aquaculture data sets, which verify the effectiveness of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a flow chart of a SAR marine aquaculture information extraction method based on a fully unsupervised Transformer network.

[0068] Figure 2 Schematic diagram of the fine-grained discrimination and completion module.

[0069] Figure 3SAR image marine aquaculture extraction result diagram; (a) column is SAR image, (b) column is true value label, (c) column is network prediction result; the first line of each column is selected Gaofen-3 SAR satellite image sample 1, the second line of each column is selected Gaofen-3 SAR satellite image sample 2, the third line of each column is selected Radarsat-2 SAR satellite image sample 1, and the fourth line of each column is selected Radarsat-2 SAR satellite image sample 2.

[0070] Figure 4 This is a visualization result diagram of the change of pseudo labels with the number of network training rounds; (a) is the initial pseudo labels; (b) is the first round of optimized pseudo labels; (c) is the final pseudo labels; (d) is the true value labels; the first row of each column is the selected Gaofen-3 SAR satellite image sample 1, the second row of each column is the selected Gaofen-3 SAR satellite image sample 2, the third row of each column is the selected Gaofen-3 SAR satellite image sample 3, the fourth row of each column is the selected Gaofen-3 SAR satellite image sample 4, and the fifth row of each column is the selected Gaofen-3 SAR satellite image sample 5. DETAILED DESCRIPTION

[0071] In order to make the method problems solved by the present invention, the method solutions adopted and the method effects achieved clearer, the present invention is further described in detail below with reference to the accompanying drawings and examples. It is understood that the specific examples described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for the convenience of description, only the parts related to the present invention are shown in the accompanying drawings, rather than all the contents.

[0072] like Figure 1 As shown, a completely unsupervised Transformer network SAR marine aquaculture information extraction method provided by an example of the present invention includes:

[0073] Compile with python3.8, pytorch1.8.1 and cuda11.1 under windows11 system, run with NVIDIA GeForce RTX 3090 GPU, and the input data size is SAR imagery.

[0074] The first step is to obtain SAR remote sensing image data on the satellite resource website, pre-process the remote sensing images, perform radiation calibration, image enhancement, and terrain correction on the SAR data, and produce a data set in the study area.

[0075] Radiometric calibration: convert digital image data into actual physical quantities such as reflectivity or radiation brightness temperature to eliminate the radiation deviation of SAR images caused by the sensor.

[0076] Image enhancement: Use a window size of The Refined Lee filtering method is used to reduce the coherent speckle noise in SAR images.

[0077] Terrain correction: SAR remote sensing images have certain geometric distortions, so they need to be registered and corrected.

[0078] Make a data set: For the filtered SAR remote sensing image, crop it into We used images of different sizes to create more than 13,000 images as the upstream self-supervised pre-training dataset, 369 images as the training set for downstream unsupervised pseudo-label training, and 160 images as the test set. We also recorded the latitude and longitude and shooting time corresponding to each image, and finally created the test set label.

[0079] The second step is the upstream pre-training stage, where the self-supervised Transformer network is used to train the data. Figure 1 In the upper part, the network consists of and First, the unlabeled marine aquaculture upstream self-supervised pre-training dataset produced in the first step The global view is obtained by randomly cropping according to the area ratio [0.4, 1] , and crop the local view according to the area ratio [0.05, 0.4] . The global view Input to Train the local view Input to Then, The output result is processed by the center bias (Center) operation, and then the probability result is output by Softmax ,at the same time, The output result is output through Softmax to output the probability result .calculate and The cross entropy loss , and the cross entropy loss after calculation is used as The loss function is back-propagated and updated Parameters , and its calculation formula is as follows:

[0080] (1);

[0081] (2);

[0082] in, Indicates the length of the probability distribution vector output by the upstream pre-training model; express exist The predicted probability of each category; express In the The predicted probability of each category; express In the The predicted probability of the categories The logarithm of Represents the cross entropy loss function relative to parameter The gradient of Parameters; is the cross entropy loss pair Output Probability The gradient of Probability and Probability The error between yes Output Probability right parameter The partial derivative of .

[0083] At the same time, the Exponential Moving Average (EMA) method is used to Network parameters renew Network parameters , the calculation formula is as follows:

[0084] (3)

[0085] in, is the smoothing factor of EMA, used to control The degree of influence of historical parameters.

[0086] The third step is to freeze the upstream self-supervised Transformer network after training is completed. Parameters . Perform downstream unsupervised pseudo-label training, such as Figure 1 As shown in the lower part, the downstream unsupervised pseudo-label training network consists of a Teacher network and a Student network. Parameters Initialize Teacher network parameters , The Teacher network consists of 6 multi-head self-attention modules, which can extract features and perform global modeling on the input data. ), Key ) and Values ​​(Query, ), in the last layer of multi-head self-attention module, there is a special classification vector , used to mark the overall classification information. First, the training images of the downstream unsupervised pseudo-label training dataset produced in the first step are defined as , divided into 1024 image blocks, input into the Teacher network, and extracted features through 12 multi-head self-attention modules to obtain the final multi-head self-attention feature representation , , p=1024, image patch features Corresponding to the first The features of the image blocks, ,in and Respectively represent The image block is Then, the last layer of multi-head self-attention features Input into the global optimal pseudo-label generator.

[0087] The classification vector is calculated using formula (4) With image patch features The affinity vector between , use formula (5) to calculate the image block features With image patch features The affinity matrix between , formula (4) and formula (5) are as follows:

[0088] (4);

[0089] (5);

[0090] in, Indicates that in the self-attention head Down-classification vector With image patch features The affinity vector between ; is the dimension space of the calculation results, where is the number of image blocks, ; Represents the self-attention head Classification vector The query vector ; Indicates image patch in the self-attention head Keys in Take the transpose; Represents the currently calculated self-attention head; Indicates that in the self-attention head Below, the similarity matrix between image patch features; is the dimension space of the calculation results; Indicates image patch in the self-attention head Keys in And take the transpose of it.

[0091] Then the affinity vector The image block with the highest affinity is defined as the key image block , . Perform calculations on each self-attention head in turn to obtain a set of key image blocks , Among them, the key image block contains the most discriminative features of the training target. Next, for each key image block of the self-attention head Diffusion to the global set of patches In the global image block set Indicates In the self-attention heads, all image blocks with high similarity to the key image blocks are as shown in the following formula:

[0092] (6)

[0093] in, Indicates the image block index currently being calculated; represents a collection of image block indices, is the number of image blocks; Indicated in In the self-attention head, the key image block With image blocks The similarity of Extracted from.

[0094] Then, using the semantic similarity - K-Means Clustering, for each self-attention head's global image patch set The clusters are divided into two groups: aquaculture and seawater. The image block set of aquaculture is defined as , the set of image patches classified as seawater is defined as , the pseudo label of each self-attention head image block level is calculated using formula (7), where Indicates image blocks, After each block-level pseudo-label is generated, it can be spliced ​​into a pseudo-label at the whole image level according to the position of the image block in the whole image. , For the The pseudo labels of the whole image are obtained by the self-attention heads. Then the initial pseudo label set is obtained , , formula (7) is as follows:

[0095] (7);

[0096] Then, the initial pseudo-label set Filter and use formula (8) to calculate the entire image-level pseudo label corresponding to each attention head The density value , and then use formula (9) to calculate the image-level pseudo-label with the largest density value as the final initial pseudo-label , formulas (8) and (9) are as follows:

[0097] (8);

[0098] (9);

[0099] in, is the number of image blocks; For image blocks The average distance of image patches within the class to which they belong; For image blocks The average distance to its different category image patches; Indicates taking and The maximum value in The value range is [-1, 1].

[0100] In the fourth step, the segmentation results of the Student network and the feature maps output by the Teacher network are input into the fine-grained discrimination and completion module to further optimize the pseudo labels. The Student network consists of an encoder and a decoder. The encoder is based on a multi-scale Transformer architecture to extract the features of the input data. The decoder adopts a Unified Perceptual Parsing Head (UPerHead) structure, including feature fusion and multi-scale information aggregation units. The initial parameters of the Student network are . The downstream unsupervised pseudo-label training dataset images obtained in the first step The enhanced image is obtained by data enhancement through rotation, scaling and linear stretching , the enhanced image The multi-scale features are input into the encoder of the Student network to extract the multi-scale features. The extracted multi-scale features are input into the decoder, and after step-by-step upsampling and feature fusion operations, the feature map is output. Then, the feature map is passed through After convolution processing, binarization is performed to finally obtain the output segmentation result. At the same time, the enhanced image Input into the Teacher network, the Teacher network learns the correlation between different spatial positions based on the multi-head self-attention module to generate a salient feature map , the salient feature map is passed through Convolution processing and binarization are performed to finally output the significant feature results .like Figure 2 For the specific architecture of fine-grained discrimination and completion, the segmentation results of the Student network are And the significant feature results output by the Teacher network Input to the fine-grained discrimination and completion module. First, use the Canny edge detection operator to extract the segmentation results Boundary information of the target , boundary information Represented as a set, each element in the set Represents an independent target boundary in the segmentation result, . According to each boundary The coordinates of the upper left corner, upper right corner, lower left corner, and lower right corner of the four points generate the minimum bounding box set , where each minimum bounding box As shown in formula (10):

[0101] (10);

[0102] in, Indicates The minimum bounding box of the object boundary; bounding box Contains all pixels that meet the conditions ; Represents a pixel in the image, and its horizontal coordinate is , the vertical axis is ; Representing Boundaries The minimum value of the horizontal coordinates of all pixels in ; Representing Boundaries The maximum value of the horizontal coordinates of all pixels in ; Representing Boundaries The minimum value of the ordinate of all pixels in ; Representing Boundaries The maximum value of the horizontal coordinates of all pixels in ; It indicates the number of breeding areas in the image, which is also the number of boundaries.

[0103] The number of boundaries and their corresponding bounding boxes in each image is determined by the number of objects in the image and is expressed in the form of a set. Using formula (11), we get In the bounding box Features within , using formula (12) we get In the bounding box Features within , formulas (11) and (12) are as follows:

[0104] (11);

[0105] (12);

[0106] in, Contains all the Pixel value ; Contains all satisfaction Pixel value ;

[0107] Then, the above operation can obtain a set of feature extraction results, specifically: the feature set of the segmentation result output by the Student network in the bounding box The significant results of the Teacher network output are in the feature set of the bounding box ,in is the number of bounding boxes obtained in step 4.2. In the discrimination stage of the fine-grained discrimination and completion module, by inputting and For features within the same bounding box, IoU (Intersection over Union) is calculated as shown in formula (13):

[0108] (13);

[0109] in, Express and Perform IoU calculation; Representation calculation and The intersection between pixels; Representation calculation and The union between pixels;

[0110] Finally, the fine-grained discrimination and completion module operates independently on the features within each bounding box. and The IoU between them exceeds the IoU threshold When and Otherwise, keep until each box is completed, and finally the optimized pseudo label is obtained. , as shown in formulas (14) and (15):

[0111] (14);

[0112] (15);

[0113] in, Represents a bounding box The characteristic results obtained in Indicates fusion and Features between; It represents the threshold of IoU, and its value is 0.3.

[0114] The fifth step is Figure 1 The two cross entropy losses in the middle and lower parts optimize the Student network and establish the objective function of the downstream network and the network iterative optimization process. The initial pseudo-label generated by the global optimal pseudo-label is calculated using formula (16) Segmentation results output by Student network The cross entropy loss between . Formula (17) is also used to calculate the optimized pseudo label Segmentation results output by Student network The cross entropy loss between , formulas (16) and (17) are as follows:

[0115] (16);

[0116] (17);

[0117] in, Representative image The total number of pixels, =262144.

[0118] Therefore, the overall objective function of the Student network is , and is the coefficient. and Number of training rounds with downstream unsupervised pseudo-label training network About, when hour, ;when hour, After that, the objective function As the loss of the Student network, the weight parameters of the Student network are reversely updated using formula (18): ; At the same time, formula (19) is used to use the weight parameter of the Student network through the exponential moving average (EMA) method Update the weight parameters of the Teacher network , formula (18) and formula (19) are as follows:

[0119] (18);

[0120] (19);

[0121] in, is the learning rate; is the parameter of the loss function relative to the network; is the smoothing factor of EMA, used to control The degree of influence of historical parameters.

[0122] Step 5: When steps 2 to 4 are completed, a round of training is completed. The training of the downstream unsupervised pseudo-label training network affects the Student network and the Teacher network through pseudo-labels. At the same time, the pseudo-labels are optimized by the prediction results of the Student network and the Teacher network, so as to achieve iterative updates of pseudo-labels. When the number of training rounds is reached, the downstream unsupervised pseudo-label training stops and the segmentation output by the Student network is used as the final output of the network.

[0123] Figure 4The following is a visualization of the pseudo-labels changing with the number of network training rounds; (a) is the initial pseudo-label, (b) is the first round of optimized pseudo-label, (c) is the final pseudo-label, and (d) is the true value label; the first row of each column is the selected Gaofen-3 SAR satellite image sample 1, the second row of each column is the selected Gaofen-3 SAR satellite image sample 2, the third row of each column is the selected Gaofen-3 SAR satellite image sample 3, the fourth row of each column is the selected Gaofen-3 SAR satellite image sample 4, and the fifth row of each column is the selected Gaofen-3 SAR satellite image sample 5. Analysis Figure 4 From columns (a) to (d) in the figure, we can see that: (a) indicates that the initial pseudo-labels are generated by the model under unsupervised conditions, which contain a lot of noise and misclassification; (b) shows that after one round of training optimization, the quality of pseudo-labels has improved, and the misclassification of some areas has been corrected, but there are still some errors; (c) indicates that after multiple rounds of optimization, the pseudo-labels are closer to the true labels, the classification boundaries are more accurate, and the overall prediction quality is improved; (d) is the true value label, which is used as a reference standard to evaluate the effect of pseudo-label optimization and can intuitively compare the matching degree between the final pseudo-label and the true value. Finally, the pseudo-label is used as the true value label to train the network, achieving completely unsupervised training.

[0124] Figure 3 The results of SAR image marine aquaculture extraction are shown in Figure 1. (a) is the SAR image as the original image, (b) is the true value label, and (c) is the network prediction result. The first line of each column is the selected Gaofen-3 SAR satellite image sample 1, the second line of each column is the selected Gaofen-3 SAR satellite image sample 2, the third line of each column is the selected Radarsat-2 SAR satellite image sample 1, and the fourth line of each column is the selected Radarsat-2 SAR satellite image sample 2. Analysis Figure 3 From columns (a) to (c) in the figure, we can see that the segmentation results obtained after using pseudo labels as true values ​​to train the network show the segmentation effects of different SAR satellite image samples. The first and second rows of each column are high-resolution images, and the third and fourth rows of each column are lower-resolution images. Through pseudo-label optimization training, the model can more effectively extract target area features and reduce misjudgments and missed judgments. The optimized pseudo-labels enhance the model's adaptability to SAR images of different resolutions, making the segmentation results more accurate and the boundaries clearer.

[0125] Finally, it should be noted that the above examples are only used to illustrate the method scheme of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above examples, ordinary method personnel in the field should understand that modifying the method schemes recorded in the above examples, or equivalently replacing some or all of the method features therein, does not cause the essence of the corresponding method scheme to deviate from the scope of the method schemes of the examples of the present invention.

Claims

1. A completely unsupervised Transformer network SAR marine aquaculture information extraction method, characterized in that: The SAR marine aquaculture information extraction method includes two stages: upstream self-supervised pre-training and downstream unsupervised pseudo-label training. First, in the upstream stage, massive unlabeled SAR image data is input into the self-supervised Transformer network for pre-training, the trained network weights are obtained, and the self-supervised transformer network is defined as a self-supervised Transformer network. Next, in the downstream stage, a global optimal pseudo-label generator and a fine-grained discrimination and completion module are designed. Specifically, first, the Teacher network is initialized using the network weights obtained from the upstream self-supervised pre-training, and the downstream training data is input into the Teacher network. The training image is generated by the global optimal pseudo-label generator. Initial pseudo-labels; secondly, the initial pseudo-labels are used as constraints to guide the training of the Student network; the image is segmented by the Student network, and its output is combined with the features learned by the Teacher network and input into the fine-grained discrimination and completion module; the initial pseudo-labels are fine-grainedly optimized by the fine-grained discrimination and completion module to refine the detailed features of the aquaculture targets and enhance their continuity information, thereby generating optimized pseudo-labels; finally, the optimized pseudo-labels are used to replace the initial pseudo-labels to constrain the segmentation network, and the output of the segmentation network in turn optimizes the pseudo-labels, and through the iterative optimization mechanism of the pseudo-labels, completely unsupervised SAR marine aquaculture information extraction is achieved; the following steps are included: The first step is to construct an unlabeled marine aquaculture dataset to obtain an upstream self-supervised pre-training dataset and a downstream unsupervised pseudo-label training dataset; The second step is the upstream pre-training stage, where the self-supervised Transformer network is used to train the upstream self-supervised pre-training dataset; Step 2.1, the self-supervised Transformer network consists of and Combination of The network parameters are , The network parameters are ,get and The two output probabilities are and ; Step 2.2, Update Parameters ; Step 2.3, use Network parameters renew Network parameters ; Step 2.4: After the upstream self-supervised Transformer network training is completed, freeze Parameters ; The third step is to conduct downstream unsupervised pseudo-label training. The downstream training network consists of a Teacher network and a Student network. First, freeze the second step. Parameters As the initial weight of the Teacher network, the weight of the Teacher network , , and then use the Teacher network to extract features from the images in the downstream unsupervised pseudo-label training data set. After that, the features output by the Teacher network are input into the global optimal pseudo-label generator to generate initial pseudo-labels, and the initial pseudo-labels are used as constraints to train the Student network; In the fourth step, the segmentation results of the Student network and the feature maps output by the Teacher network are input into the fine-grained discrimination and completion module to further optimize the pseudo labels. The Student network consists of an encoder and a decoder. The encoder is based on a multi-scale Transformer architecture to extract the features of the input data. The decoder adopts a UPerHead structure, including feature fusion and multi-scale information aggregation units. The initial parameters of the Student network are ; The fifth step is to establish the objective function of the downstream unsupervised pseudo-label training network and the network iterative optimization process.

2. The SAR marine aquaculture information extraction method of a fully unsupervised Transformer network according to claim 1 is characterized in that: The first step is as follows: Step 1.1, obtain SAR remote sensing image data from the satellite resource platform, and preprocess the SAR remote sensing image, which includes radiation calibration, image enhancement and terrain correction; Step 1.2, filtering the SAR remote sensing image processed in step 1.1; Step 1.3, crop the filtered SAR remote sensing image to a fixed size, generate a SAR marine aquaculture dataset, and divide it into an upstream self-supervised pre-training dataset and a downstream unsupervised pseudo-label training dataset; The downstream unsupervised pseudo-label training dataset is divided into a training set and a test set, and the test set is labeled.

3. The SAR marine aquaculture information extraction method of a fully unsupervised Transformer network according to claim 1 is characterized in that: The second step is as follows: Step 2.1, specifically: First, for each image in the upstream self-supervised pre-training dataset Perform random cropping to get a global view and partial views ; Set the global view Input to Train the local view Input to Train in and then get and The two output probabilities are and ; Step 2.2, use formula (1) to calculate and The cross entropy loss is calculated as The loss function is back-propagated to update Parameters , formula (1) and formula (2) are as follows: （1）； （2）； in, Indicates the length of the probability distribution vector output by the upstream pre-training model; express exist The predicted probability of each category; express In the The predicted probability of each category; express In the The predicted probability of the categories The logarithm of Represents the cross entropy loss function relative to parameter The gradient of Parameters; is the cross entropy loss pair Output Probability The gradient of Probability and Probability The error between yes Output Probability right parameter The partial derivative of Step 2.3, use the exponentially weighted moving average method to Network parameters renew Network parameters , as shown in formula (3): （3）； in, is the smoothing factor of EMA, used to control The degree of influence of historical parameters.

4. The SAR marine aquaculture information extraction method of a fully unsupervised Transformer network according to claim 3 is characterized in that: The third step is as follows: Step 3.1, using the frozen upstream pre-training Parameters Initialize the parameters of the Teacher network. The Teacher network consists of N multi-head self-attention modules. Each multi-head self-attention module consists of H self-attention heads. It performs feature extraction and global modeling on the input data. In the multi-head self-attention module, it contains query ,key Sum , in the last layer of multi-head self-attention module, there is a special classification vector , used to mark the overall classification information; Step 3.2, define the training images of the downstream unsupervised pseudo-label training dataset as ,Will Divide into P image blocks, input into the Teacher network, and extract features through N multi-head self-attention modules to obtain the final layer of multi-head self-attention feature representation , , image patch features Corresponding to the first The features of the image blocks, ,in and Respectively represent The image block is The query and key in each self-attention head; Step 3.3: The last layer of multi-head self-attention features obtained in step 3.2 Input into the global optimal pseudo-label generator and use formula (4) to calculate the classification vector With image patch features The affinity vector between , use formula (5) to calculate the image block features With image patch features The affinity matrix between , formula (4) and formula (5) are as follows: （4）； （5）； in, Indicates that in the self-attention head Down-classification vector With image patch features The affinity vector between ; is the dimension space of the calculation results, where is the number of image blocks, ; Represents the self-attention head Classification vector The query vector ; Indicates image patch in the self-attention head Keys in Take the transpose; Represents the currently calculated self-attention head; Indicates that in the self-attention head Below, the similarity matrix between image patch features; is the dimension space of the calculation results; Indicates image patch in the self-attention head Keys in And take the transpose of it; Step 3.4, insert affinity vector The image block with the highest affinity is defined as the key image block , ; Perform calculations on each self-attention head in turn to obtain a set of key image blocks , is the number of self-attention heads; then, for each key image block of the self-attention head Diffusion to the global set of patches In the global image patch set Indicates In the self-attention heads, all image blocks with high similarity to the key image blocks are as shown in formula (6): （6）； in, Indicates the image block index currently being calculated; represents a collection of image block indices, is the number of image blocks; Indicated in In the self-attention head, the key image block With image blocks The similarity of Extracted from; Step 3.5, using the method of calculating semantic similarity -Mean clustering algorithm, for each global image patch set of self-attention head obtained in step 3.4 The clusters are divided into two groups: aquaculture and seawater. The image block set of aquaculture is defined as , the set of image patches classified as seawater is defined as , use formula (7) to calculate the pseudo label of each self-attention head image block level ,in Indicates image blocks, After each block-level pseudo-label is generated, it can be spliced ​​into a pseudo-label at the whole image level according to the position of the image block in the whole image. , For the The pseudo labels of the whole image are obtained by the self-attention heads; then the initial pseudo label set is obtained , , is the number of self-attention heads, formula (7) is as follows: （7）； Step 3.6: For the initial pseudo-label set obtained in step 3.5 Filter and use formula (8) to calculate the entire image-level pseudo label corresponding to each attention head The density value , and then use formula (9) to calculate the image-level pseudo-label with the largest density value as the final initial pseudo-label , formulas (8) and (9) are as follows: （8）； （9）； in, is the number of image blocks; For image blocks The average distance of image patches within the class to which they belong; For image blocks The average distance to its different category image patches; Indicates taking and The maximum value in The value range is [-1, 1].

5. The SAR marine aquaculture information extraction method of a fully unsupervised Transformer network according to claim 4 is characterized in that: The fourth step is as follows: Step 4.1: Unsupervised pseudo-label training dataset images The enhanced image is obtained by data enhancement through rotation, scaling and linear stretching , the enhanced image Input into the encoder of the Student network to extract multi-scale features; The extracted multi-scale features are input to the decoder, and after step-by-step upsampling and feature fusion operations, feature maps are output. Then, the feature map is binarized after convolution to obtain the output segmentation result. At the same time, the enhanced image Input into the Teacher network, the Teacher network learns the correlation between different spatial positions based on the multi-head self-attention module to generate a salient feature map , the significant feature map is convolved and binarized, and the significant feature result is finally output ; Step 4.2: The segmentation result of the Student network And the significant feature results output by the Teacher network Input to the fine-grained discrimination and completion module; first, use the Canny edge detection operator to extract the segmentation result Boundary information of the target , boundary information Represented as a set, each element in the set Represents an independent target boundary in the segmentation result, ; According to each boundary The coordinates of the upper left corner, upper right corner, lower left corner, and lower right corner of the four points generate the minimum bounding box set , where each minimum bounding box As shown in formula (10): （10）； in, Indicates The minimum bounding box of the object boundary; bounding box Contains all pixels that meet the conditions ; Represents a pixel in the image, and its horizontal coordinate is , the vertical axis is ; Representing Boundaries The minimum value of the horizontal coordinates of all pixels in ; Representing Boundaries The maximum value of the horizontal coordinates of all pixels in ; Representing Boundaries The minimum value of the vertical coordinates of all pixels in ; Representing Boundaries The maximum value of the horizontal coordinates of all pixels in ; Indicates the number of breeding areas in the image, which is also the number of boundaries; Step 4.3: The number of boundaries and their corresponding bounding boxes in each image is determined by the number of objects in the image and is expressed in the form of a set; using formula (11), we get In the bounding box Features within , using formula (12) we get In the bounding box Features within , formulas (11) and (12) are as follows: （11）； （12）； in, Contains all the Pixel value ; Contains all satisfaction Pixel value ; Step 4.4: From step 4.3, we can get a set of feature extraction results, specifically: the feature set of the segmentation result output by the Student network in the bounding box The significant results of the Teacher network output are in the feature set of the bounding box ,in is the number of bounding boxes obtained in step 4.2; in the discrimination stage of the fine-grained discrimination and completion module, by inputting and For features within the same bounding box, perform IoU calculation; Step 4.4: The fine-grained discrimination and completion module operates independently on the features within each bounding box. and The IoU between them exceeds the IoU threshold When and Otherwise, keep until each box is completed, and finally the optimized pseudo label is obtained. , as shown in formulas (14) and (15): （14）； （15）； in, Represents a bounding box The characteristic results obtained in Indicates fusion and Features between; Indicates the IoU threshold.

6. The SAR marine aquaculture information extraction method of a fully unsupervised Transformer network according to claim 5 is characterized in that: In the fourth step: In step 4.3, the IoU calculation is shown in formula (13): （13）； in, Express and Perform IoU calculation; Representation calculation and The intersection between pixels; Representation calculation and The union between pixels; In step 4.3, the threshold The value range is [0, 1].

7. The SAR marine aquaculture information extraction method of a fully unsupervised Transformer network according to claim 5 is characterized in that: The fifth step is as follows: Step 5.1: Use formula (16) to calculate the initial pseudo-label generated by the global optimal pseudo-label Segmentation results output by Student network The cross entropy loss between ; Formula (17) is also used to calculate the optimized pseudo label Segmentation results output by Student network The cross entropy loss between , formulas (16) and (17) are as follows: （16）； （17）； in, Represents downstream unsupervised pseudo-label training images The total number of pixels; Step 5.2, the overall objective function of the Student network is , and is the coefficient; the objective function As the loss of the Student network, the weight parameters of the Student network are reversely updated using formula (18): ; At the same time, formula (19) is used to use the weight parameter of the Student network through the exponential weighted moving average method Update the weight parameters of the Teacher network , formula (18) and formula (19) are as follows: （18）； （19）； in, is the learning rate; is the parameter of the loss function relative to the network; is the smoothing factor of EMA, used to control The degree of influence of historical parameters; Step 5.3, when the third step to step 5.2 are completed, one round of training is completed; the training of the downstream unsupervised pseudo-label training network affects the Student network and the Teacher network through the pseudo-label, and the pseudo-label is optimized by the prediction results of the Student network and the Teacher network, so as to achieve the iterative update of the pseudo-label; when the number of training rounds is reached, the downstream unsupervised pseudo-label training stops training and takes the segmentation output by the Student network as the final output of the network.

8. The SAR marine aquaculture information extraction method of a fully unsupervised Transformer network according to claim 7 is characterized in that: In step 5.2, and Number of training rounds with downstream unsupervised pseudo-label training network About, when hour, ;when hour, .

Images