Classification method for marine raft culture area based on panoramic segmentation of optical remote sensing image

Abstract

The application provides a kind of based on panoramic segmentation optical remote sensing image marine raft type culture zone classification method, comprising: obtaining the image to be segmented, the image to be segmented is the optical remote sensing image of marine culture zone;The image to be segmented is input into the panoramic segmentation model trained in advance, and the multi-classification segmentation result is predicted;Wherein, the semantic segmentation branch network is used to carry out semantic segmentation to the image to be segmented, to obtain initial semantic segmentation result;The instance segmentation branch network is used to carry out instance segmentation to the image to be segmented, to obtain initial instance segmentation result;Initial semantic segmentation result and initial instance segmentation result are fused using panoramic fusion module, to obtain multi-classification segmentation result.The present application can effectively utilize the rich information in remote sensing image, realize the high-precision multi-classification identification task of marine raft type culture zone, and improve the segmentation accuracy of panoramic segmentation model.

Description

Technical Field

[0001] This invention relates to the fields of marine remote sensing and image processing technology, and in particular to a method for classifying marine raft aquaculture areas based on panoramic segmentation optical remote sensing images. Background Technology

[0002] Marine raft aquaculture is an important component of marine aquaculture. Compared to pond and tidal flat aquaculture near the coast, raft aquaculture covers a wide area and is geographically dispersed. Using traditional navigation and positioning systems for on-site measurements is not only time-consuming and labor-intensive but also struggles to obtain accurate results over large areas. The development of remote sensing technology has greatly compensated for the shortcomings of traditional ground-based measurements, such as limited coverage and insufficient data acquisition. Simultaneously, utilizing deep learning algorithms to quickly and accurately acquire information on the distribution and status of marine aquaculture areas is a reliable and advanced technical means for conducting dynamic monitoring of marine raft aquaculture. For the development and utilization of marine resources and to support macro-control, it is necessary to classify marine aquaculture areas more finely without reducing segmentation accuracy, for example, by subdividing them into fish (net cages), algae (longlines), shellfish (floating rafts), and others.

[0003] Synthetic Aperture Radar (SAR) is widely used in remote sensing due to its all-weather, all-day capability. However, SAR images suffer from drawbacks such as low resolution, susceptibility to noise interference, severe geometric distortion, and limited usable features. With the development of optical remote sensing technology, the resolution of optical remote sensing images has greatly improved, and the feature information they contain has become increasingly richer. However, some optical images are affected by cloud cover, fog, and lighting conditions. These interference factors limit the extraction of feature information from optical remote sensing images and increase the difficulty of target identification and segmentation.

[0004] Existing methods for extracting marine aquaculture areas based on convolutional neural networks mainly fall into semantic segmentation and instance segmentation, such as improved SOLO, D-ResUnet, and HCHNet segmentation algorithms. Both semantic segmentation and instance segmentation are pixel-level classifications. In semantic segmentation, the predicted value of each pixel is mapped to a probability value in the [0,1] range using the Softmax function. Then, the error between the predicted value and the true label value is determined using the cross-entropy loss function. The model is continuously trained using gradient descent to minimize this error. The more target categories (i.e., the more labels in the dataset) there are in semantic segmentation, the more interference occurs when identifying and segmenting each target. This results in a more dispersed probability distribution and a larger variance in the mathematical model, making it more difficult to focus the predicted probability distribution onto a single label value. This slows down the convergence speed of the loss function and reduces the accuracy of segmentation and recognition. From the above analysis, it can be concluded that multi-class recognition and high-precision segmentation are mutually restrictive in semantic segmentation tasks. Therefore, simple target detection, recognition, and segmentation cannot achieve refined classification of marine raft aquaculture areas.

[0005] The existing data labels for convolutional neural network models of marine raft aquaculture areas are only suitable for training single-task models for semantic or instance segmentation, and lack data labels for panoramic segmentation of marine aquaculture areas for multi-class classification.

[0006] In summary, there is an urgent need for a panoramic segmentation method for marine aquaculture areas that can achieve multi-task classification. Summary of the Invention

[0007] This invention provides a method for classifying marine raft aquaculture areas based on panoramic segmentation optical remote sensing images, in order to solve the above-mentioned problems.

[0008] This invention provides a method for classifying marine raft aquaculture areas based on panoramic segmentation optical remote sensing images, comprising:

[0009] Acquire an image to be segmented, wherein the image to be segmented is an optical remote sensing image of a marine aquaculture area;

[0010] The image to be segmented is input into a pre-trained panoramic segmentation model to predict multi-class segmentation results, which include raft aquaculture areas, non-raft aquaculture areas, and multiple aquaculture area categories.

[0011] The pre-trained panoramic segmentation model includes a semantic segmentation branch network, an instance segmentation branch network, and a panoramic fusion module.

[0012] The semantic segmentation branch network is used to perform semantic segmentation on the image to be segmented to obtain an initial semantic segmentation result, which includes an initial raft aquaculture area and an initial non-raft aquaculture area.

[0013] The instance segmentation branch network is used to perform instance segmentation on the image to be segmented to obtain an initial instance segmentation result, which includes multiple initial aquaculture area categories.

[0014] The panoramic fusion module is used to fuse the initial semantic segmentation result and the initial instance segmentation result to obtain a multi-class segmentation result.

[0015] According to the present invention, a classification method for marine raft aquaculture areas based on panoramic segmentation optical remote sensing images is provided, wherein the semantic segmentation branch network is an improved U... 2 -Net network, the improved U 2 The -Net network includes at least six U-shaped secondary encoders and five U-shaped secondary decoders. The six U-shaped secondary encoders consist of four first-level encoders and two second-level encoders, respectively. The five U-shaped secondary decoders consist of four first-level decoders and one second-level decoder, respectively. Each first-level encoder and each first-level decoder is composed of a first convolutional block, an LSFE module, multiple downsampling modules, a DPC module, a second convolutional block, a first convolutional block, and multiple upsampling modules, respectively.

[0016] The LSFE module is used to extract features of the aquaculture area within a large field of view, and it includes separable convolution and output filter.

[0017] The DPC module is used to capture remote context information, which includes separable convolutions and output channels.

[0018] According to the present invention, a method for classifying marine raft aquaculture areas based on panoramic segmentation optical remote sensing images is provided. The instance segmentation branch network includes an improved SOTR network, which includes at least a Transformer module. The Transformer module includes separable convolutions and iABN synchronization layers. The Transformer module is used to predict the class of each instance.

[0019] According to the present invention, a method for classifying marine raft aquaculture areas based on panoramic segmentation optical remote sensing images is provided. The instance segmentation branch network further includes a feature extraction module, which includes a moving inverse bottleneck unit and a bidirectional feature pyramid network.

[0020] The feature extraction module is used to extract features from the image to be segmented to obtain multi-scale features;

[0021] Based on the multi-scale features, the improved SOTR network is used to perform instance segmentation on the image to be segmented to obtain initial instance segmentation results.

[0022] According to the present invention, a method for classifying marine raft aquaculture areas based on panoramic segmentation optical remote sensing images is provided. The pre-trained panoramic segmentation model is trained in the following manner:

[0023] Obtain the training dataset and its corresponding labels, and construct a panoramic segmentation model; wherein, the labels include semantic labels for raft aquaculture areas and non-raft aquaculture areas, as well as instance labels for various aquaculture area categories;

[0024] The training dataset is input into the semantic segmentation branch network to predict the semantic segmentation result for training. The loss between the semantic segmentation result for training and the semantic label is calculated to obtain the first loss.

[0025] The training dataset is input into the instance segmentation branch network to predict the training instance segmentation result. The loss between the training instance segmentation result and the instance label is calculated to obtain the second loss.

[0026] The panoramic fusion module is used to adaptively fuse the semantic segmentation result and the instance segmentation result used for training to obtain the multi-classification result used for training.

[0027] The total loss is obtained based on the first loss and the second loss. The panoramic segmentation model is then trained based on the training multi-classification results and the total loss until the panoramic segmentation model converges, thereby obtaining a trained panoramic segmentation model.

[0028] According to the present invention, a method for classifying marine raft aquaculture areas based on panoramic segmentation optical remote sensing images, after obtaining the training dataset and its corresponding labels, the method further includes:

[0029] Normalized vegetation index features and normalized water index features are constructed for the training dataset respectively.

[0030] The normalized vegetation index features and normalized water index features are fused with the training dataset to obtain a shared synthetic dataset;

[0031] Accordingly, inputting the training dataset into the semantic segmentation branch network includes:

[0032] The shared synthetic dataset is input into the semantic segmentation branch network;

[0033] The step of inputting the training dataset into the instance segmentation branch network includes:

[0034] The shared synthetic dataset is input into the instance splitting branch network.

[0035] According to the present invention, a classification method for marine raft aquaculture areas based on panoramic segmentation optical remote sensing images is provided. The training dataset includes a labeled dataset and an adversarial example set. The labeled dataset is a dataset with corresponding labels after annotation. The adversarial example set is obtained by adversarial training on the segmentation results of the training instances.

[0036] According to the present invention, a classification method for marine raft aquaculture areas based on panoramic segmentation optical remote sensing images is provided, wherein the labeled dataset is obtained in the following manner:

[0037] Acquire optical remote sensing images of the marine aquaculture area for training, and perform at least the following processing on the optical remote sensing images: unified storage format, dehazing, normalization, and cropping.

[0038] According to the present invention, a method for classifying marine raft aquaculture areas based on panoramic segmentation optical remote sensing images is provided, wherein the various aquaculture area categories include at least fish, algae, and shellfish.

[0039] The present invention also provides a classification device for marine raft aquaculture areas based on panoramic segmentation optical remote sensing images, comprising:

[0040] The image acquisition module is used to acquire the image to be segmented, which is an optical remote sensing image of a marine aquaculture area;

[0041] The image segmentation module is used to input the image to be segmented into a pre-trained panoramic segmentation model to predict multi-class segmentation results, which include raft aquaculture areas, non-raft aquaculture areas, and multiple aquaculture area categories.

[0042] The pre-trained panoramic segmentation model includes a semantic segmentation branch network, an instance segmentation branch network, and a panoramic fusion module.

[0043] The semantic segmentation branch network is used to perform semantic segmentation on the image to be segmented to obtain an initial semantic segmentation result, which includes an initial raft aquaculture area and an initial non-raft aquaculture area.

[0044] The instance segmentation branch network is used to perform instance segmentation on the image to be segmented to obtain an initial instance segmentation result, which includes multiple initial aquaculture area categories.

[0045] The panoramic fusion module is used to fuse the initial semantic segmentation result and the initial instance segmentation result to obtain a multi-class segmentation result.

[0046] This invention provides a classification method for marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images. It segments the image to be segmented in parallel using a semantic segmentation branch network and an instance segmentation branch network, and then fuses the outputs of the two branch networks through a parameter-free panoramic fusion module to obtain multi-class segmentation results, achieving multi-task classification. Furthermore, this adaptive fusion method of the panoramic fusion module can more fully utilize the logical outputs of the semantic segmentation head and the instance segmentation head, improving the accuracy of multi-class classification tasks. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is one of the flowcharts of the classification method for marine raft aquaculture areas based on panoramic segmentation optical remote sensing images provided in the embodiments of the present invention;

[0049] Figure 2 This is the second flowchart of the method for classifying marine raft aquaculture areas based on panoramic segmentation optical remote sensing images provided in this embodiment of the invention.

[0050] Figure 3 It is an existing U 2 -Schematic diagram of the Net network structure;

[0051] Figure 4 It is an existing U 2 A structural comparison diagram of the En_1 secondary structure in the -Net network and the improved En_1 secondary structure of this invention;

[0052] Figure 5 This is a schematic diagram of the training process of the panoramic segmentation model provided in an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of the structure of the marine raft aquaculture area classification device based on panoramic segmentation optical remote sensing images provided in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0055] The existing convolutional neural network models mentioned above can only achieve single tasks of semantic or instance segmentation. Furthermore, given the complex and ever-changing marine environment, it is necessary to more fully and effectively utilize the rich information in optical remote sensing imagery. Therefore, this invention unifies the semantic segmentation and instance segmentation prediction sub-networks and fuses their outputs to form a holistic panoramic segmentation network model. It constructs a panoramic segmentation label dataset for multi-classification tasks in marine raft aquaculture areas, thereby achieving multi-classification and refining the classification of marine raft aquaculture areas, thus improving the model's segmentation accuracy. The following section, with reference to the accompanying figures, provides a detailed explanation of the proposed panoramic segmentation-based classification method for marine raft aquaculture areas using optical remote sensing imagery.

[0056] Figure 1 This is one of the flowcharts of the classification method for marine raft aquaculture areas based on panoramic segmentation optical remote sensing images provided in the embodiments of the present invention; Figure 2 This is the second flowchart illustrating the classification method for marine raft aquaculture areas based on panoramic segmentation optical remote sensing images provided in this embodiment of the invention.

[0057] like Figure 1 As shown in Figure 2, this classification method for marine raft aquaculture areas based on panoramic segmentation optical remote sensing images includes:

[0058] S101, Obtain the image to be segmented, which is an optical remote sensing image of a marine aquaculture area.

[0059] S102, the image to be segmented is input into a pre-trained panoramic segmentation model to predict multi-class segmentation results.

[0060] The multi-classification segmentation results include raft aquaculture areas, non-raft aquaculture areas, and multiple aquaculture area categories, which in turn include fish, algae, shellfish, and others.

[0061] The pre-trained panoptic segmentation model (HPPS) includes a semantic segmentation branch network, an instance segmentation branch network, and a panoptic fusion module.

[0062] The semantic segmentation branch network is used to perform semantic segmentation on the image to be segmented to obtain an initial semantic segmentation result, which includes an initial raft aquaculture area and an initial non-raft aquaculture area.

[0063] The instance segmentation branch network is used to perform instance segmentation on the image to be segmented to obtain initial instance segmentation results, which include multiple initial aquaculture area categories.

[0064] The panoramic fusion module is used to fuse the initial semantic segmentation result and the initial instance segmentation result to obtain a multi-class segmentation result.

[0065] The panoramic fusion module is a parameter-free panoramic fusion module. Based on pixel-level head prediction adaptability, it selectively attenuates or amplifies the logical output score of the fusion, derived from the initial semantic segmentation result and the initial instance segmentation result. By jointly optimizing the entire panoramic segmentation network in this end-to-end manner, a high-precision panoramic segmentation output result for multi-classification of marine raft aquaculture areas is obtained, realizing the multi-classification task of marine raft aquaculture areas from optical remote sensing images.

[0066] In addition, before inputting the image to be segmented into the pre-trained panoramic segmentation model, the image to be segmented is standardized and preprocessed, and then slidably cropped into an image of size 2048*2048. The cropped image is then input into the trained HPPS model (i.e., panoramic segmentation model) in sequence. The output of the HPPS model is the multi-classification result of the marine aquaculture area. All images are stitched together to obtain the overall panoramic segmentation result of the image to be segmented.

[0067] The present invention provides a classification method for marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images. This method segments the image to be segmented in parallel using a semantic segmentation branch network and an instance segmentation branch network. The outputs of the two branch networks are then fused using a parameter-free panoramic fusion module to obtain multi-class segmentation results, achieving multi-task classification. Furthermore, this adaptive fusion method of the panoramic fusion module can more fully utilize the logical outputs of the semantic segmentation head and the instance segmentation head, improving the accuracy of multi-class classification tasks.

[0068] Furthermore, the semantic segmentation branch network is an improved U... 2 -Net network, the semantic segmentation branch network is an improved U 2 -Net network, the improved U 2The -Net network includes at least 6 U-shaped secondary encoders and 5 U-shaped secondary decoders. The 6 U-shaped secondary encoders are, in order, 4 first-level encoders and 2 second-level encoders. The 5 U-shaped secondary decoders are, in order, 4 first-level decoders and 1 second-level decoder. The first-level encoders and the first-level decoders are each composed of, in order, a first convolutional block, an LSFE module, multiple downsampling modules, a DPC module, a second convolutional block, a first convolutional block, and multiple upsampling modules.

[0069] The LSFE module is used to extract features of the aquaculture area within a large field of view, and it includes separable convolutions and output filters. Specifically, it includes two 3×3 separable convolutions and 128 output filters.

[0070] The DPC module is used to capture remote context information and includes a separable convolution and output channels. Specifically, it includes a 3×3 separable convolution with 256 output channels, which is expanded to five parallel branches. The outputs of all parallel branches are then concatenated to produce a tensor with 1280 channels. This tensor is finally input to a 1×1 convolution with 256 output channels, and the output of this 1×1 convolution is the output of the DPC module.

[0071] In the existing technology, U 2 -Net network is a saliency detection model, and its specific network structure diagram is as follows: Figure 3 As shown, it is a two-level nested U-shaped structure. In this invention, the overall U-shaped structure is called the primary structure, and each small U-shaped structure contained within the primary structure is called the secondary structure. This invention does not make improvements to the primary structure, but makes specific improvements to the secondary structures.

[0072] The specific design concept for the improvement is as follows: Since remote sensing imagery involves observing a large area of ​​the ocean and coastline from a satellite platform, it offers a wide field of view and generates massive amounts of data, providing a comprehensive macroscopic reflection of the distribution of marine aquaculture areas. Therefore, the improved U proposed in this invention… 2 The -Net network model not only needs to achieve macroscopic detection of marine aquaculture areas, but also needs to identify, extract, and classify each small piece of aquaculture raft. To achieve this goal, this invention improves the secondary U-shaped structure by using a Large Scale Feature Extractor (LSFE) module and a Dense Prediction Cells (DPC) module in the secondary structure.

[0073] The following section discusses the improved U, starting from the overall network structure. 2 -The Net network will be described in detail.

[0074] First, the improved U provided by this invention 2 -Net networks, in their own structure, are also like... Figure 3 As shown, it specifically includes 6 U-shaped secondary encoders (i.e., En_1 to En_6) and 5 U-shaped secondary decoders (i.e., De_1 to De_5). Structurally, En_1 corresponds to De_1, En_2 corresponds to De_2, En_3 corresponds to De_3, En_4 corresponds to De_4, and En_5 corresponds to De_5.

[0075] This invention relates to U 2 The first four substructures of the -Net network are improved, namely the structures of En_1, De_1, En_2, De_2, En_3, De_3, En_4, and De_4. However, En_5, En_6, and De_5 are not improved and still use the existing structures. These will not be described in detail here.

[0076] Taking En_1 as an example, the existing En_1 network structure consists of two first convolutional blocks (i.e., Figure 4 The ①Conv+BN+RELU and 5 downsampling modules (i.e. Figure 4 The ③Downsample×1 / 2Conv+BN+RELU), and 1 second convolutional block (i.e. Figure 4 The five convolutional blocks are: ⑤ Conv+BN+RELU dilation = 4, 1 first convolutional block, and 5 upsampling modules (i.e., ...). Figure 4 (⑧Upsample×2Conv+BN+RELU).

[0077] The improved En_1 network structure consists of one first convolutional block and one LSFE module (i.e., ... Figure 4 (②), 4 downsampling modules, and 1 DPC module (i.e. Figure 4The improved En_2 network structure consists of 1 first convolutional block, 1 second convolutional block, 1 first convolutional block, and 5 upsampling modules. Specifically, the second first convolutional block in the prior art is replaced with an LSFE module, the last downsampling module is replaced with a DPC module, and the rest remain the same. Since the structural difference between En_2, En_3, and En_4 and the aforementioned En_1 is the decrease in the number of downsampling and upsampling modules, the improved En_2 network structure is similarly structured as follows: 1 first convolutional block, 1 LSFE module, 3 downsampling modules, 1 DPC module, 1 second convolutional block, 1 first convolutional block, and 4 upsampling modules. The improved En_3 network structure is similarly structured as follows: 1 first convolutional block, 1 LSFE module, 2 downsampling modules, 1 DPC module, 1 second convolutional block, 1 first convolutional block, and 3 upsampling modules. The improved En_4 network structure consists of one first convolutional block, one LSFE module, one downsampling module, one DPC module, one second convolutional block, one first convolutional block, and two upsampling modules.

[0078] In addition, the improved De_1, De_2, De_3, and De_4 correspond one-to-one with the aforementioned improved En_1, En_2, En_3, and En_4, which will not be described in detail here.

[0079] This invention employs binary classification semantic segmentation, with the corresponding dataset labels containing only label 0 (non-raft aquaculture area) and label 1 (raft aquaculture area). During training, the semantic segmentation branch network uses a binary cross-entropy loss function.

[0080] The classification method for marine raft aquaculture areas based on panoramic segmentation optical remote sensing images provided in this invention uses the LSFE module to extract features of the aquaculture area within a large field of view to achieve macroscopic detection of the marine aquaculture area; and uses the DPC module to capture remote context information to identify, extract and classify each small piece of aquaculture raft, thereby improving the segmentation accuracy of the semantic segmentation branch network.

[0081] Furthermore, the instance segmentation branch network includes an improved SOTR network, which at least includes a Transformer module; wherein the Transformer module includes separable convolutions and iABN synchronization layers; the Transformer module is used to predict each instance class.

[0082] In existing technologies, SOTR simplifies the segmentation process by utilizing the Transformer, employing two parallel subtasks: 1) predicting the class of each instance using the Transformer; and 2) dynamically generating segmentation masks using a multi-level upsampling module. The encoder-decoder Transformer model unifies the instance segmentation task through a series of learnable mask embeddings. This invention extends the Transformer by using separable convolutions and iABN (inplace activated batch normalization) synchronization layers, thereby improving segmentation accuracy and training convergence to a certain extent.

[0083] The classification method for marine raft aquaculture areas based on panoramic segmentation optical remote sensing images provided in this invention improves the segmentation accuracy and training convergence of the model through the extension of separable convolution and iABN synchronization layers.

[0084] Furthermore, the instance segmentation branch network also includes a feature extraction module, which includes a moving inverse bottleneck unit and a bidirectional feature pyramid network (FPN).

[0085] The feature extraction module is used to extract features from the image to be segmented to obtain multi-scale features.

[0086] Based on the multi-scale features, the improved SOTR network is used to perform instance segmentation on the image to be segmented to obtain initial instance segmentation results.

[0087] The classification method for marine raft aquaculture areas based on panoramic segmentation optical remote sensing images provided in this invention extracts multi-scale features through moving inverse bottleneck units and bidirectional feature pyramid networks. This allows for the acquisition of more large-scale features, small-scale features, shallow information, and deep information, thereby improving the accuracy of instance segmentation.

[0088] Figure 5 This is a schematic diagram illustrating the training process of the panoramic segmentation model provided in an embodiment of the present invention; as shown below. Figure 5 As shown, the pre-trained panoramic segmentation model is trained in the following manner:

[0089] Obtain the training dataset and its corresponding labels, and build a panoptic segmentation model.

[0090] The tags include semantic tags for raft aquaculture areas and non-raft aquaculture areas, as well as instance tags for various aquaculture area categories.

[0091] It should be noted that after obtaining the training dataset, it needs to be labeled. The labeling is divided into "stuff" and "thing" classes. The "stuff" class uses semantic masks, where 0 represents a non-raft aquaculture area and 1 represents a raft aquaculture area. The "thing" class uses instance masks, including four instances: fish, algae, shellfish, and others. Based on these labeling categories, semantic labels for the background (i.e., non-raft aquaculture area) and foreground (i.e., raft aquaculture area) are created for both the training and test sets. Based on these, instance labels for instance 1 (fish), instance 2 (algae), instance 3 (shellfish), and instance 4 (other) are then created.

[0092] The training dataset is input into the semantic segmentation branch network to predict the semantic segmentation result for training. The loss between the semantic segmentation result for training and the semantic label is calculated to obtain the first loss.

[0093] The training dataset is input into the instance segmentation branch network to predict the instance segmentation results for training. The loss between the instance segmentation results for training and the instance labels is calculated to obtain the second loss.

[0094] The panoramic fusion module is used to adaptively fuse the semantic segmentation results and the instance segmentation results used for training to obtain the multi-classification results used for training.

[0095] The total loss is obtained based on the first loss and the second loss. The panoramic segmentation model is then trained based on the training multi-classification results and the total loss until the panoramic segmentation model converges, thereby obtaining a trained panoramic segmentation model.

[0096] The total loss is obtained by adaptively weighting and summing the first loss and the second loss based on the logic output score of attenuation or amplification fusion.

[0097] It should be noted that this invention trains the panoptic segmentation model using a shared synthetic dataset. After the panoptic segmentation model reaches convergence accuracy, it is then tested using a test set to obtain a qualitative evaluation result.

[0098] If the test does not meet the accuracy requirements, the model is retrained by adjusting hyperparameters and supplementing training samples until the qualitative evaluation requirements are met. Then, the detection set is used to test the model and evaluate the panoramic segmentation accuracy PQ of the model. Finally, a well-trained panoramic segmentation model is obtained.

[0099] The classification method for marine raft aquaculture areas based on panoramic segmentation optical remote sensing images provided in this invention not only classifies and labels the raft aquaculture area and the background area as a whole, but also further classifies and labels various aquaculture categories in the raft aquaculture area in a more refined manner to achieve multi-classification tasks.

[0100] Furthermore, after obtaining the training dataset and its corresponding labels, the method further includes:

[0101] Normalized Difference Vegetation Index (NDVI) and Normalized Difference Water Index (NDWI) were constructed for the training dataset.

[0102] The normalized vegetation index features and normalized water index features are fused with the training dataset to obtain a shared synthetic dataset.

[0103] Accordingly, inputting the training dataset into the semantic segmentation branch network includes:

[0104] The shared synthetic dataset is input into the semantic segmentation branch network.

[0105] The step of inputting the training dataset into the instance segmentation branch network includes:

[0106] The shared synthetic dataset is input into the instance splitting branch network.

[0107] It should be noted that the normalized vegetation index (NDI) and normalized water index (NDI) features can also be fused with the image to be segmented after the image to be segmented is acquired, so as to use the panoramic segmentation model to segment the fused image.

[0108] The classification method for marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images provided in this invention obtains a shared synthetic dataset by fusing two custom features, NDVI and NDWI, into the training dataset. This allows for more full and effective utilization of the rich information in optical remote sensing images, thereby improving the segmentation accuracy of the panoramic segmentation model.

[0109] Furthermore, the training dataset includes a labeled dataset and an adversarial example set. The labeled dataset is a dataset with corresponding labels after annotation, and the adversarial example set is obtained by adversarial training on the segmentation results of the training instances.

[0110] Specifically, adversarial training is performed on the instance segmentation branch using a multi-class classification data label set from marine aquaculture areas to improve the anti-interference capability of multi-objective multi-class classification. Adversarial examples generated during the training process are added to the training dataset, forming the training dataset together with the label dataset.

[0111] It should be noted that adversarial training can be achieved by adding a discriminator or by generating new samples based on gradient backpropagation. Adversarial training methods are conventional and are not limited in this invention.

[0112] The classification method for marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images provided in this invention obtains an adversarial sample set through adversarial training, and trains the panoramic segmentation model based on the adversarial sample set and the label dataset, thereby improving the anti-interference ability of the panoramic segmentation model.

[0113] Furthermore, the labeled dataset is obtained in the following manner:

[0114] Acquire optical remote sensing images of the marine aquaculture area for training, and perform at least the following processing on the optical remote sensing images: unified storage format, dehazing, normalization, and cropping.

[0115] Specifically, we first acquired medium-resolution optical remote sensing images of Sentinel-2, GF-1 (PMS / WFV), and Landsat covering a 30km radius along my country's coastline. Then, we standardized the storage format, removed cloud cover, and normalized the optical remote sensing images. Finally, we cropped them into standard 2048*2048 images to form a standard labeled dataset.

[0116] The following describes the marine raft aquaculture area classification device based on panoramic segmentation optical remote sensing images provided by the present invention. The marine raft aquaculture area classification device based on panoramic segmentation optical remote sensing images described below can be referred to in correspondence with the marine raft aquaculture area classification method based on panoramic segmentation optical remote sensing images described above.

[0117] Figure 6 This is a schematic diagram of the structure of the marine raft aquaculture area classification device based on panoramic segmentation optical remote sensing images provided in an embodiment of the present invention; as shown below. Figure 6 As shown, the classification device for marine raft aquaculture areas based on panoramic segmentation optical remote sensing images includes an image acquisition module 601 and an image segmentation module 602.

[0118] The image acquisition module 601 is used to acquire an image to be segmented, which is an optical remote sensing image of a marine aquaculture area.

[0119] The image segmentation module 602 is used to input the image to be segmented into a pre-trained panoramic segmentation model to predict multi-class segmentation results.

[0120] The multi-classification segmentation results include raft aquaculture areas, non-raft aquaculture areas, and multiple aquaculture area categories, which in turn include fish, algae, shellfish, and others.

[0121] The pre-trained panoptic segmentation model (HPPS) includes a semantic segmentation branch network, an instance segmentation branch network, and a panoptic fusion module.

[0122] The semantic segmentation branch network is used to perform semantic segmentation on the image to be segmented to obtain an initial semantic segmentation result, which includes an initial raft aquaculture area and an initial non-raft aquaculture area.

[0123] The instance segmentation branch network is used to perform instance segmentation on the image to be segmented to obtain initial instance segmentation results, which include multiple initial aquaculture area categories.

[0124] The panoramic fusion module is used to fuse the initial semantic segmentation result and the initial instance segmentation result to obtain a multi-class segmentation result.

[0125] The panoramic fusion module is a parameter-free panoramic fusion module. Based on pixel-level head prediction adaptability, it selectively attenuates or amplifies the logical output score of the fusion, derived from the initial semantic segmentation result and the initial instance segmentation result. By jointly optimizing the entire panoramic segmentation network in this end-to-end manner, a high-precision panoramic segmentation output result for multi-classification of marine raft aquaculture areas is obtained, realizing the multi-classification task of marine raft aquaculture areas from optical remote sensing images.

[0126] In addition, before inputting the image to be segmented into the pre-trained panoramic segmentation model, the image to be segmented is standardized and preprocessed, and then slidably cropped into an image of size 2048*2048. The cropped image is then input into the trained HPPS model (i.e., panoramic segmentation model) in sequence. The output of the HPPS model is the multi-classification result of the marine aquaculture area. All images are stitched together to obtain the overall panoramic segmentation result of the image to be segmented.

[0127] The marine raft aquaculture area classification device based on panoramic segmentation of optical remote sensing images provided in this invention segmentes the image to be segmented in parallel using a semantic segmentation branch network and an instance segmentation branch network. The outputs of the two branch networks are then fused through a parameter-free panoramic fusion module to obtain multi-class segmentation results, achieving multi-task classification. Furthermore, this adaptive fusion method of the panoramic fusion module can more fully utilize the logical outputs of the semantic segmentation head and the instance segmentation head, improving the accuracy of multi-class classification tasks.

[0128] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0130] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for classifying marine raft culture areas in optical remote sensing images based on panoramic segmentation, characterized in that, The method comprises the following steps: acquiring an image to be segmented, which is an optical remote sensing image of a mariculture area; inputting the image to be segmented into a pre-trained panoramic segmentation model to obtain a multi-class segmentation result, which includes raft mariculture areas, non-raft mariculture areas and multiple mariculture area categories; wherein the pre-trained panoramic segmentation model comprises a semantic segmentation branch network, an instance segmentation branch network and a panoramic fusion module; performing semantic segmentation on the image to be segmented by using the semantic segmentation branch network to obtain an initial semantic segmentation result, which includes initial raft mariculture areas and initial non-raft mariculture areas; performing instance segmentation on the image to be segmented by using the instance segmentation branch network to obtain an initial instance segmentation result, which includes multiple initial mariculture area categories; fusing the initial semantic segmentation result and the initial instance segmentation result by using the panoramic fusion module to obtain a multi-class segmentation result; The semantic segmentation branch network is an improved U 2 -Net network, the improved U 2 The improved U-Net network at least includes 6 secondary encoders in U-shaped structure and 5 secondary decoders in U-shaped structure, the 6 secondary encoders in U-shaped structure are 4 first secondary encoders and 2 second secondary encoders in sequence, and the 5 secondary decoders in U-shaped structure are 4 first secondary decoders and 1 second secondary decoder in sequence; the first secondary encoder and the first secondary decoder are sequentially composed of a first convolution block, an LSFE module, a plurality of down-sampling modules, a DPC module, a second convolution block, a first convolution block and a plurality of up-sampling modules. wherein the LSFE module is used to extract features of mariculture areas in a large field of view, and comprises separable convolution and an output filter; the DPC module is used to capture remote context information, and comprises separable convolution and an output channel; the instance segmentation branch network comprises an improved SOTR network, which at least comprises a Transformer module; wherein the Transformer module comprises separable convolution and an iABN synchronization layer; the Transformer module is used to predict each instance category.

2. The panoramic segmentation-based method for classifying the marine cage area in optical remote sensing images according to claim 1, characterized in that, The instance segmentation branch network further comprises a feature extraction module, which comprises a mobile reverse bottleneck unit and a bidirectional feature pyramid network; performing feature extraction on the image to be segmented by using the feature extraction module to obtain multi-scale features; based on the multi-scale features, performing instance segmentation on the image to be segmented by using the improved SOTR network to obtain an initial instance segmentation result.

3. The method according to any one of claims 1-2, wherein the method is a method for classifying the marine raft farming area in an optical remote sensing image based on panoramic segmentation, characterized in that, The pre-trained panoramic segmentation model is trained in the following manner: acquiring a training data set and its corresponding label, and constructing a panoramic segmentation model; wherein the label comprises semantic labels of raft mariculture areas and non-raft mariculture areas, and instance labels of multiple mariculture area categories; inputting the training data set into the semantic segmentation branch network to obtain a training semantic segmentation result, calculating the loss between the training semantic segmentation result and the semantic label to obtain a first loss; inputting the training data set into the instance segmentation branch network to obtain a training instance segmentation result, calculating the loss between the training instance segmentation result and the instance label to obtain a second loss; performing adaptive fusion on the training semantic segmentation result and the training instance segmentation result by using the panoramic fusion module to obtain a training multi-class result; According to the first loss and the second loss, a total loss is obtained, and the panoramic segmentation model is trained based on the training multi-classification result and the total loss until the panoramic segmentation model converges to obtain a trained panoramic segmentation model.

4. The method according to claim 3, wherein the method is characterized by, After the training data set and the corresponding label thereof are obtained, the method further comprises: The normalized vegetation index features and the normalized water index features of the training data set are constructed respectively; The normalized vegetation index features and the normalized water index features are fused with the training data set to obtain a shared synthetic data set; Accordingly, the inputting of the training data set into the semantic segmentation branch network comprises: The shared synthetic data set is inputted into the semantic segmentation branch network; The inputting of the training data set into the instance segmentation branch network comprises: The shared synthetic data set is inputted into the instance segmentation branch network.

5. The method according to claim 3, wherein the method is characterized by, The training data set comprises a label data set and an adversarial sample set, the label data set is a data set with corresponding labels after labeling, and the adversarial sample set is obtained by adversarial training on the training instance segmentation result.

6. The panoramic segmentation-based optical remote sensing image classification method of the marine raft farming area according to claim 5, characterized in that, The label data set is obtained by: An optical remote sensing image of a marine aquaculture area is obtained, and at least storage format unification, cloud and fog removal processing, normalization processing and cropping processing are performed on the optical remote sensing image.

7. The panoramic segmentation-based optical remote sensing image classification method of the marine raft farming area according to claim 5, characterized in that, The plurality of aquaculture area categories at least includes fish, algae and shellfish.

8. A device for classifying marine raft culture areas in optical remote sensing images based on panoramic segmentation, characterized in that it comprises: Comprise: An image acquisition module is configured to acquire a to-be-segmented image, wherein the to-be-segmented image is an optical remote sensing image of a marine aquaculture area. An image segmentation module is configured to input the to-be-segmented image into a pre-trained panoramic segmentation model to predict a multi-classification segmentation result, wherein the multi-classification segmentation result comprises raft aquaculture areas, non-raft aquaculture areas and a plurality of aquaculture area categories. The pre-trained panoramic segmentation model comprises a semantic segmentation branch network, an instance segmentation branch network and a panoramic fusion module. The semantic segmentation branch network is used to perform semantic segmentation on the to-be-segmented image to obtain an initial semantic segmentation result, wherein the initial semantic segmentation result comprises initial raft aquaculture areas and initial non-raft aquaculture areas. The instance segmentation branch network is used to perform instance segmentation on the to-be-segmented image to obtain an initial instance segmentation result, wherein the initial instance segmentation result comprises a plurality of initial aquaculture area categories. The panoramic fusion module is used to fuse the initial semantic segmentation result and the initial instance segmentation result to obtain the multi-classification segmentation result. The semantic segmentation branch network is an improved U 2 -Net network, the improved U 2 The improved U-Net network at least includes 6 secondary encoders in U-shaped structure and 5 secondary decoders in U-shaped structure, the 6 secondary encoders in U-shaped structure are 4 first secondary encoders and 2 second secondary encoders in sequence, and the 5 secondary decoders in U-shaped structure are 4 first secondary decoders and 1 second secondary decoder in sequence; the first secondary encoder and the first secondary decoder are sequentially composed of a first convolution block, an LSFE module, a plurality of down-sampling modules, a DPC module, a second convolution block, a first convolution block and a plurality of up-sampling modules. The LSFE module is used to extract features of the aquaculture area in a large field of view, and comprises a separable convolution and an output filter. The DPC module is used to capture remote context information, and comprises a separable convolution and an output channel. The instance segmentation branch network comprises an improved SOTR network, and the improved SOTR network comprises at least a Transformer module; wherein the Transformer module comprises a separable convolution and an iABN synchronization layer; and the Transformer module is used to predict each instance category.

Images