Marine remote sensing image scene classification method based on multi-branch fusion

Abstract

The invention provides a marine remote sensing image scene classification method based on multi-branch fusion, and particularly relates to the technical field of remote sensing scenes and classification, and the method comprises the steps: constructing a network structure of a convolutional network and Transform multi-branch parallel computing, designing a kernel selection multi-scale aggregation module, and processing different scale features generated by a plurality of branches; by selecting different convolution kernels, a dynamic receptive field is established, features of different scales are processed adaptively, and semantic differences are reduced to realize effective aggregation of multi-scale features; in addition, shape information of a shallow layer is supplemented to deep features by setting learnable parameters, and the expression ability of final classification features is enhanced. According to the marine remote sensing image scene classification method provided by the invention, 2.74% of accuracy is improved on a baseline, multi-scale features are more effectively aggregated, and the feature representation capability is enhanced.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing scenes and classification, and in particular to a marine remote sensing image scene classification method based on multi-branch fusion. Background Art

[0002] In the field of remote sensing image interpretation, remote sensing scene classification is a basic and important task in the field of remote sensing. It is the process of annotating the semantic content of remote sensing images. Remote sensing scene images present highly complex geometric structures. Each scene contains objects of different sizes and appearances. These objects not only have varied shapes, but also have significant differences in spatial structures. Affected by these characteristics, the main objects in the remote sensing scene are difficult to capture clearly, and their complex scale information makes it particularly difficult to extract accurate and discriminative features from the image. Feature extraction is the most important step in remote sensing scene classification. Only by extracting features that can effectively express the intrinsic properties of the image and have a high degree of discriminability can the accurate classification of remote sensing scenes be achieved.

[0003] Therefore, when extracting features, how to optimize features to emphasize key object information; how to fuse features and effectively process multi-scale feature information, these series of problems need to be solved in remote sensing scene classification to adapt to the unique spatial structure of remote sensing images. In addition, in the field of marine remote sensing, there are also problems such as large image scale changes and loss of shape information that is beneficial to marine remote sensing scene classification. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a marine remote sensing image scene classification method based on multi-branch fusion, which can effectively aggregate multi-scale features and improve the feature representation capability by fusing remote sensing image scale information and hierarchical information.

[0005] In a first aspect of the present invention, the present invention discloses a method for classifying ocean remote sensing image scenes based on multi-branch fusion, comprising: Step S1, obtaining marine remote sensing image data and preprocessing it, the obtained marine remote sensing image data includes a training set, a validation set and a test set, which are used for subsequent model training, validation and evaluation respectively; Step S2, constructing an ocean remote sensing scene classification model based on multi-branch fusion, wherein the model adopts a pyramid network structure of CNN, including multiple multi-branch network blocks and a learnable multi-level fusion module, wherein the learnable multi-level aggregation module is used to fuse the multiple multi-branch network blocks; Step S3, training the ocean remote sensing scene classification model based on multi-branch fusion according to the data in the training set, the validation set and the test set, and updating the ocean remote sensing scene classification model based on multi-branch fusion; Step S4: input the ocean remote sensing image data to be classified into the trained ocean remote sensing scene classification model to obtain the classification result.

[0006] Furthermore, the marine remote sensing image data includes ArcGIS datasets, WDMI datasets and Google Earth datasets; wherein the ArcGIS dataset is mainly used for geographic information system analysis, and contains high-resolution marine environment images, and the capture time is limited to the past five years to ensure that the current marine scene changes are reflected; the WDMI dataset focuses on water body monitoring, covering a variety of marine scenes, including ports, coastlines and open seas; the Google Earth dataset provides high-resolution satellite images on a global scale, with reliable annotation information, to support supervised learning needs. The preprocessing of the marine remote sensing image data in step S1 includes: fixing the seed seed for each image in the marine remote sensing image dataset during the training process, changing the image size to 256×256, and cropping it to 224×224 using a center cropping method.

[0007] Furthermore, the model also includes a stem convolution module for processing the input ocean remote sensing image data and sending the processed image data to the multi-branch network block; Before the image is input into each processing module, it is first processed using a stem convolution module. An image consisting of two channels , , composed of 3×3 convolutional layers with a stride of 2, generating Features, The input dimension of the Multi-branch Network block.

[0008] Further, the multi-branch network block includes: The multi-scale embedding module is used to generate embedding features of different sizes and input them into the multi-branch module; A multi-branch module, comprising a CNN branch and a Transformer branch, wherein the CNN branch is used to obtain local features of the image data according to the embedded features output by the multi-scale embedding module, and the Transformer branch is used to obtain multi-scale features of the image data according to the embedded features output by the multi-scale embedding module; The kernel selection aggregation module is used to aggregate the local features obtained by the CNN branch and the multi-scale features obtained by the Transformer branch to obtain the output features; The multi-branch network block includes four stages: the first multi-branch network block, the second multi-branch network block, the third multi-branch network block, and the fourth multi-branch network block; the number of Transformer branches in each stage is set to [2, 3, 3, 3], and the corresponding number of encoders stacked for each Transformer branch is set to [1, 2, 4, 1].

[0009] Furthermore, the multi-scale embedding module is used to generate embedding features of different sizes and input them into the multi-branch module including: The multi-scale embedding module adjusts the sequence length of the token by changing the stride and padding, outputs features of the same shape with different patch sizes, and also achieves 2×2 downsampling of the current stage features by adjusting the stride. In order to reduce model parameters and computational overhead, a 3×3 depthwise separable convolution is used, which consists of a 3×3 depthwise convolution and a 1×1 pointwise convolution in the embedding layer. Batch normalization and Hardswish function are added after all convolutional layers. Finally, the token embedding features of different sizes are input into the multi-branch module respectively. The multi-scale embedding module uses a deep residual bottleneck layer consisting of 1×1 convolution, 3×3 depth convolution, 1×1 convolution and residual connection to process the embedded features, and at the same time inputs these features of different scales into the multi-branch module.

[0010] Furthermore, the Transformer branch is used to obtain multi-scale features of the image data according to the embedded features output by the multi-scale embedding module, including: The Transformer branch uses the classic Transformer encoder structure, using factorized self-attention and convolutional relative position encoding to form a new convolutional attention module; specifically, it includes: The input token is first convolutionally encoded and then input into the convolutional attention module, which includes the factorization attention module and the convolutional relative position encoding module. The factorization attention formula is as follows:

[0011]

[0012]

[0013] in, , , Three vectors representing self-attention, Represents the image feature data input to the Transformer encoder, Indicates the number of channels, superscript T indicates transposition, the first formula is the calculation formula of the original attention mechanism, and the second formula is the efficient factorized attention. It can be seen from the formula that on the basis of the original, through two functions and right and Decompose and calculate together and Multiply to approximate the original attention map; the third formula is the method used, function Use the identity mapping and add a scaling factor ,function Use the softmax function; In order to model the relationship between local tokens, deep convolution is used to encode relative positions and obtain the attention map of relative positions. , combining the factorized self-attention and position encoding to obtain the convolutional attention, the specific formula is as follows:

[0014]

[0015]

[0016] in, The attention map representing relative positions emphasizes how local features are extracted through convolution and combined with the query vector to capture local relationships; Represents a comprehensive attention map. By connecting multiple attention maps, the diversity of information is enhanced, which helps to improve the expressiveness and accuracy of the model. represents Hadamard Product, and W represents the weight value of the convolution kernel.

[0017] Furthermore, the kernel selection aggregation module, namely the kernel selection multi-scale aggregation module (KSMSA), adopts different large convolution kernels and uses dynamic receptive fields to perceive information of different scales, effectively models different contextual nuances of objects of different categories, further improves the efficiency of fusion, and enhances feature representation.

[0018] The core of the kernel selection multi-scale aggregation module is the large kernel selection module and the convolution layer for fusion and channel transformation. The large kernel selection module is embedded in two 1×1 convolution layers for enhancing the network representation capability. Its main steps are separation, fusion and selection. For the features of each branch of the input, the large kernel selection module first performs convolution calculations with convolution kernel sizes of 5 and 7 to obtain features processed by different convolution kernel sizes: and , these convolutions are implemented using efficient grouped convolutions, allowing channel mixing for each spatial feature vector. The purpose of aggregation is to be able to adaptively adjust the size of the receptive field according to the content. The basic idea is to use gates to control the information flow of the two convolution kernel branches. To achieve this, the gating mechanism needs to integrate this information and splice these features, as shown in the following formula, combining the features of the two branches: ; In order to enhance the network's ability to focus on relevant spatial context areas, a gating mechanism is used to select feature maps from large convolution kernel spaces of different scales. In order to effectively extract spatial relationships, a parallel operation of global average pooling and global maximum pooling is used, and then spliced ​​together again to generate spatial-wise statistics. The formula is as follows:

[0019]

[0020]

[0021] in, , Represents the height and width of the feature map, Indicates aisle, Represents the first Channel characteristics, Represents the first Channel characteristics, Represents the concatenation of two pooling features. Channel characteristics, Represents the splicing process. Then the convolution operation is performed to convert the pooled features into two spatial attention maps to achieve information interaction in different spaces. The sigmoid activation function is used to obtain the selection mask of each decomposed large core space to provide guidance for accurate and adaptive selection. The specific formula is as follows:

[0022] in, represents the sigmoid activation function, represents the convolution operation, Indicates all channels . Use the weight matrix to reweight and sum the features to get the final output feature of the current scale branch , and in order to prevent gradient disappearance and gradient explosion, a residual connection is added, as shown in the following formula:

[0023] These branch features processed by the large core selection module are spliced ​​together, and then channel transformed through 1×1 convolution to complete the multi-branch feature fusion of the current stage, and input into the multi-scale embedding module of the next stage. By using different convolution kernels to process features of different scales, the dynamic receptive field is obtained, the semantic differences between multi-scale features are effectively aligned, and multi-branch feature aggregation is better realized, the feature representation capability is enhanced, and the classification effect is improved.

[0024] Furthermore, the learnable multi-level aggregation module in step S2 is used to fuse the multiple multi-branch network blocks, including: The learnable multi-level aggregation module downsamples the features of the first multi-branch network block, the second multi-branch network block, and the third multi-branch network block to obtain feature maps of the same size, then concatenates these features together, and then calculates the fusion features of the shallow information through SKConv; then, the fusion features of the shallow information are obtained with the set learnable hyperparameters. Multiply them together to achieve dynamic summation in order to generate the final features that make up for the shallow information.

[0025] Furthermore, in step S3, label smoothing loss is introduced as a loss function in the training of the ocean remote sensing scene classification model based on multi-branch fusion; the label smoothing loss changes the one-hot hard coding of the label in the traditional cross entropy loss to soft coding, and adds a little noise in the one-hot to make the model more robust; the loss function formula is shown in the following formula:

[0026]

[0027] in, represents the predicted value of the classifier output, represents one-hot encoding, represents the true label, is a small constant, is the number of categories.

[0028] In a second aspect of the present invention, the present invention provides a computer-readable storage medium storing a computer program, which implements any method described in the first aspect when executed by a processor.

[0029] The present invention provides a marine remote sensing image scene classification method based on multi-branch fusion, and specifically relates to the field of remote sensing scenes and classification technology. The present invention constructs a network structure of convolutional network and Transformer multi-branch parallel calculation, designs a kernel selection multi-scale aggregation module, and processes different scale features generated by multiple branches; by selecting different convolution kernels, a dynamic receptive field is established, and features of different scales are adaptively processed, and semantic differences are reduced to achieve effective aggregation of multi-scale features; in addition, by setting learnable parameters, shallow shape information is supplemented to deep features, thereby enhancing the representation capability of final classification features; the marine remote sensing image scene classification method provided by the present invention achieves an accuracy improvement of 2.74% on the baseline, more effectively aggregates multi-scale features, and enhances feature representation capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a simplified flowchart of a method for scene classification of ocean remote sensing images based on multi-branch fusion; Figure 2 This is a schematic diagram of the network structure of the ocean remote sensing scene classification model based on multi-branch fusion; Figure 3 Schematic diagram of the network structure of the convolutional attention module; Figure 4 Diagram of the network structure for selecting the multi-scale aggregation module for the kernel. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to further clearly and completely describe the technical solutions in the embodiments of the present application. It should be noted that the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0032] In order to make the invention objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application are further described in detail in conjunction with the drawings in the specification: In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the advantages of the present invention will be further illustrated by comparing the embodiments in conjunction with the drawings and specific implementation methods.

[0033] like Figure 1 As shown, in embodiment 1, the present invention provides a method for classifying ocean remote sensing image scenes based on multi-branch fusion, comprising: Step S1, obtaining and preprocessing ocean remote sensing image data, wherein the obtained ocean remote sensing image data includes a training set, a validation set and a test set, which are used for subsequent model training, validation and evaluation respectively; Step S2, constructing an ocean remote sensing scene classification model based on multi-branch fusion, wherein the model adopts a pyramid network structure of CNN, including multiple multi-branch network blocks and a learnable multi-level fusion module, wherein the learnable multi-level aggregation module is used to fuse the multiple multi-branch network blocks; Step S3, training the ocean remote sensing scene classification model based on multi-branch fusion according to the data in the training set, the validation set and the test set, and updating the ocean remote sensing scene classification model based on multi-branch fusion; Step S4: input the ocean remote sensing image data to be classified into the trained ocean remote sensing scene classification model to obtain the classification result.

[0034] In this embodiment, the ocean remote sensing image data collected in step S1 includes ArcGIS dataset, WDMI dataset and Google Earth dataset. The acquired ocean remote sensing image data is divided into training set, validation set and test set according to the proportion, which are used for subsequent model training, validation and evaluation respectively.

[0035] Step S1 preprocesses the marine remote sensing image data set, enhances the data, and makes the network more focused on extracting semantic features in the image. The steps of preprocessing the marine remote sensing image data set, enhancing the data, and making the network more focused on extracting semantic features in the image include: fixing the seed seed for each image in the marine remote sensing image data set during the training process, changing the image size to 256×256, and cropping it to 224×224 using a center cropping method.

[0036] In this embodiment, the model in step S2 also includes a stem convolution module, which can better embed low-level feature representations, that is, it will not lose important information. Therefore, when acquiring embedded features in the first stage, a stem convolution module is first used to process An image consisting of two channels , , composed of 3×3 convolutional layers with a stride of 2, generating Features, The input dimension of the Multi-branch Network block.

[0037] In this embodiment, the classification algorithm network in the marine remote sensing scene classification model based on multi-branch fusion constructed in step S2 adopts the pyramid structure of CNN, including multiple multi-branch network blocks and a learnable multi-level fusion module, and the learnable multi-level aggregation module is used to fuse the multiple multi-branch network blocks.

[0038] The multiple multi-branch network blocks include four multi-branch network blocks, namely the first multi-branch network block, the second multi-branch network block, the third multi-branch network block, and the fourth multi-branch network block, which are divided into four stages. Each stage is designed to use multiple Transformer branches to process feature maps of different scales. The multi-branch network block of each stage includes a multi-scale embedding module, a multi-branch module, and a kernel selection aggregation module.

[0039] The multi-scale embedding module adjusts the sequence length of the token by changing the stride and padding, outputs features of the same shape with different patch sizes, and also achieves 2×2 downsampling of the current stage features by adjusting the stride. In order to reduce model parameters and computational overhead, 3×3 depthwise separable convolution is used, which consists of 3×3 depthwise convolution and 1× 1 pointwise convolution in the embedding layer. Batch normalization and Hardswish function are added after all convolutional layers. Finally, the token embedding features of different sizes are input into each encoder of the multi-branch module respectively. The multi-branch module includes a CNN convolution module and multiple Transformer modules.

[0040] In order to obtain local feature information, a deep residual bottleneck layer consisting of 1×1 convolution, 3×3 depth convolution, 1×1 convolution and residual connection is used to process the embedded features, and these features of different scales are input into the Transformer module of the multi-branch network block.

[0041] In this embodiment, the classic Transformer encoder structure is used in the Transformer branch. However, in order to reduce the computational burden brought by the use of multiple branches, the original attention module is modified, and the efficient factorized self-attention is used in combination with the convolutional relative position encoding to form a new convolutional attention module. The module is as follows: Figure 3 shown.

[0042] The input token is first convolutionally encoded and then input into the convolutional attention module, which includes the factorization attention module and the convolutional relative position encoding module. The efficient factorization attention is designed based on the original self-attention to reduce the amount of calculation. The formula is as follows:

[0043]

[0044]

[0045] in, , , Three vectors representing self-attention, Represents the image feature data input to the Transformer encoder, Indicates the number of channels, The first formula is the calculation formula of the original attention mechanism, and the second formula is the efficient factorized attention. It can be seen from the formula that on the basis of the original, through two functions and right and Decompose and calculate together and The third formula is the method used, the function P uses the identity mapping and adds the scaling factor ,function Use the softmax function.

[0046] According to the above factorized attention formula, , Represents the image feature data input to the Transformer encoder, Can be seen as a query mapping The linear transformation of the global data correlation of each eigenvector in , then if two The query vector , , ,That , then their self-attention outputs are the same, as shown in the following formula:

[0047] Without position encoding, the Transformer consists only of linear layers and self-attention modules, and cannot obtain the difference of local nearby features, so position encoding is needed. In order to model the relationship between local tokens, deep convolution is used for relative position encoding, as shown in the first formula below, to obtain the attention map of relative position , combining the two, we get the convolutional attention, as shown in the third formula below:

[0048]

[0049]

[0050] in, The attention map representing relative positions emphasizes how local features are extracted through convolution and combined with the query vector to capture local relationships; Represents a comprehensive attention map. By connecting multiple attention maps, the diversity of information is enhanced, which helps to improve the expressiveness and accuracy of the model. It represents Hadamard Product. Represents the weight value of the convolution kernel.

[0051] After multi-branch calculations, local features extracted by convolution and multi-scale features processed by Transformer are obtained. Then, the proposed kernel selection multi-scale aggregation module is used to effectively aggregate features of different scales obtained by CNN branches and Transformer branches. After feature extraction at different stages, in order to express features more richly, the features of the previous stage, that is, the shallow features of the CNN branch, are effectively added to the final feature representation using a learnable multi-level aggregation module to increase the feature representation capability.

[0052] The number of Transformer branches in each stage of the model's network structure is set to [2, 3, 3, 3]. In the first stage, due to the high image resolution, the features require more computation, so the number of branches in this layer is reduced; the corresponding number of encoders stacked in each Transformer branch is set to [1, 2, 4, 1].

[0053] In this implementation, the kernel selection multi-scale aggregation module uses a multi-scale Transformer branch in the network to process features of different scales. When aggregated, these features have large semantic differences. If they are directly fused by addition, concatenation or simple 1×1 convolution, the attention of unimportant objects will be increased, resulting in information redundancy and affecting classification accuracy.

[0054] The receptive fields of each layer of a standard CNN are designed to share the same size, but the scale of important features in each layer requires different receptive fields. When aggregating multi-scale features, a dynamic receptive field is required to adapt to the fusion of features of different scales, using a small receptive field for small-scale features and a large receptive field for large-scale features.

[0055] We designed a kernel selection multi-scale aggregation module (KSMSA), which uses different large convolution kernels and dynamic receptive fields to perceive information of different scales, effectively modeling the different contextual nuances of objects of different categories, further improving the efficiency of fusion and enhancing feature representation. The specific network structure diagram is shown in the figure below. Figure 4 shown.

[0056] The core of the kernel selection multi-scale aggregation module is the large kernel selection module and the convolution layer for fusion and channel transformation. The large kernel selection module is embedded in two 1×1 convolution layers for enhancing the network representation capability. Its main steps are separation, fusion and selection. For the features of each branch of the input, the large kernel selection module first performs convolution calculations with convolution kernel sizes of 5 and 7 to obtain features processed by different convolution kernel sizes. and These convolutions are implemented using efficient grouped convolutions, allowing channel mixing for each spatial feature vector. The purpose of aggregation is to be able to adaptively adjust the size of the receptive field based on the content. The basic idea is to use gates to control the information flow of the two convolution kernel branches. To achieve this, the gating mechanism needs to integrate this information and concatenate these features, as expressed in the following formula, combining the features of the two branches:

[0057] In order to enhance the network's ability to focus on relevant spatial context areas, a gating mechanism is used to select feature maps from large convolution kernel spaces of different scales. In order to effectively extract spatial relationships, a parallel operation of global average pooling and global maximum pooling is used, and then spliced ​​together again to generate spatial-wise statistics. The formula is as follows:

[0058]

[0059]

[0060] in, , Represents the height and width of the feature map, Indicates aisle, Represents the first Channel characteristics, Represents the first Channel characteristics, Represents the concatenation of two pooling features. Channel features. Then, a convolution operation is performed to convert the pooled features into two spatial attention maps to achieve information interaction in different spaces. The sigmoid activation function is used to obtain the selection mask of each decomposed large kernel space to provide guidance for accurate and adaptive selection, as shown in the following formula:

[0061] in, represents the sigmoid activation function, represents the convolution operation, Indicates all channels , use the weight matrix to reweight and sum the features to get the final output feature of the current scale branch , and in order to prevent gradient disappearance and gradient explosion, a residual connection is added, as shown in the following formula:

[0062] Finally, the branch features that have undergone the large core selection operation are spliced ​​together, and then the channels are transformed through 1×1 convolution to complete the multi-branch feature fusion of the current stage, so as to input the multi-scale embedding module of the next stage. By using different convolution kernels to process features of different scales, the dynamic receptive field is obtained, the semantic differences between multi-scale features are effectively aligned, and multi-branch feature aggregation is better realized, the feature representation capability is enhanced, and the classification effect is improved.

[0063] In this embodiment, the learnable multi-level aggregation module aggregates shallow information features, and controls the fusion of original deep features and shallow aggregate features by setting learnable parameters. Since the shallow layer of CNN obtains more shape features, while the deep layer extracts more semantic features, common classification networks only use deep semantic features for classification, but shallow low-level features also play an important role in the classification of marine remote sensing scenes. The texture advantage of objects in remote sensing images is weakened, and more shape information is needed to supplement. Although Transformer can capture some shape deviations, since the network of the present invention adopts a pyramid structure, downsampling operations are performed at each stage, and the resolution of the image will gradually decrease, resulting in the weakening of the shape information obtained by ViT, and as the network depth increases, the shape information will gradually be lost. In response to this problem, a learnable multi-level aggregation module (Learnable Multi-level Aggregation, LMLA) is proposed in this embodiment to aggregate shallow information features and control the fusion of original deep features and shallow aggregate features by setting learnable parameters.

[0064] The structure of the learnable multi-level aggregation module is as follows Figure 2 As shown in the figure, the features of the three shallow stages in multiple multi-branch network blocks (i.e., the first multi-branch network block, the second multi-branch network, and the third multi-branch network block) are downsampled to obtain feature maps of the same size, and then these features are spliced ​​together, and then the fusion features of the shallow information used are calculated by SKConv, and finally combined with the set learnable hyperparameters Multiply them together to achieve dynamic summation in order to generate the final features that make up for the shallow information. Considering that the shallow information is mainly to supplement the shape information missing from the deep semantic information, the setting of the hyperparameter cannot be too large, which will destroy the deep semantic information, so the initialization value is a smaller value.

[0065] In this embodiment, the model loss function value is calculated in step S3. ,The number of images in the remote sensing image dataset is small relative to ,general datasets, so in order to prevent overfitting problems during the training ,process, label smoothing loss is introduced as the loss function in the ,model training process.

[0066] Label smoothing loss is a regularization method used to prevent model overfitting. It changes the one-hot hard coding of labels in traditional cross entropy loss to soft coding, adds a little noise to the one-hot, and makes the model more robust. The loss function formula is shown in the formula below. The one-hot of the real label is modified according to the following formula:

[0067]

[0068] in, represents the predicted value of the classifier output, represents one-hot encoding, represents the true label, is a small constant, is the number of categories. After this modification, the probability value of the true label is reduced from 1 to a very small value, and the probability value of other labels is increased from 0 to a very small value, making The probability optimization target in the loss is no longer 1 and 0, and the optimal solution of the predicted value is no longer positive infinity, but a specific value, which improves the generalization ability of the model.

[0069] In a second embodiment of the present invention, the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program implements the method in the first embodiment provided above when executed by a processor.

[0070] The present invention provides a marine remote sensing image scene classification method based on multi-branch fusion, and specifically relates to the field of remote sensing scenes and classification technology. The present invention constructs a network structure of convolutional network and Transformer multi-branch parallel calculation, designs a kernel selection multi-scale aggregation module, and processes different scale features generated by multiple branches; by selecting different convolution kernels, a dynamic receptive field is established, and features of different scales are adaptively processed, and semantic differences are reduced to achieve effective aggregation of multi-scale features; in addition, by setting learnable parameters, shallow shape information is supplemented to deep features, thereby enhancing the representation capability of final classification features; the marine remote sensing image scene classification method provided by the present invention achieves an accuracy improvement of 2.74% on the baseline, more effectively aggregates multi-scale features, and enhances feature representation capability.

[0071] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A method for scene classification of marine remote sensing images based on multi-branch fusion, It is characterized in that include: Step S1, obtaining and preprocessing ocean remote sensing image data, wherein the obtained ocean remote sensing image data includes a training set, a validation set and a test set, which are respectively used for subsequent training, validation and evaluation of the model; the ocean remote sensing image data includes an ArcGIS data set, a WDMI data set and a Google Earth data set; The preprocessing of the ocean remote sensing image data in step S1 includes: fixing a seed for each image in the ocean remote sensing image data set during the training process, changing the image size to 256×256, and cropping it to 224×224 by center cropping; Step S2: constructing an ocean remote sensing scene classification model based on multi-branch fusion, wherein the model adopts a pyramid network structure of CNN, including multiple multi-branch network blocks and a learnable multi-level fusion module, wherein the learnable multi-level aggregation module is used to fuse the multiple multi-branch network blocks; the multi-branch network block includes: The multi-scale embedding module is used to generate embedding features of different sizes and input them into the multi-branch module; A multi-branch module, comprising a CNN branch and a Transformer branch, wherein the CNN branch is used to obtain local features of the image data according to the embedded features output by the multi-scale embedding module, and the Transformer branch is used to obtain multi-scale features of the image data according to the embedded features output by the multi-scale embedding module; The kernel selection aggregation module is used to aggregate the local features obtained by the CNN branch and the multi-scale features obtained by the Transformer branch to obtain the output features; The multi-branch network block includes four stages: a first multi-branch network block, a second multi-branch network block, a third multi-branch network block, and a fourth multi-branch network block; the number of Transformer branches in each stage is set to [2, 3, 3, 3], and the number of encoders stacked for each corresponding Transformer branch is set to [1, 2, 4, 1]; Step S3, training the ocean remote sensing scene classification model based on multi-branch fusion according to the data in the training set, the validation set and the test set, and updating the ocean remote sensing scene classification model based on multi-branch fusion; Step S4: input the ocean remote sensing image data to be classified into the trained ocean remote sensing scene classification model to obtain the classification result.

2. The method for classifying marine remote sensing images based on multi-branch fusion according to claim 1, It is characterized in that The model also includes a stem convolution module for processing the input ocean remote sensing image data and sending the processed image data to the multi-branch network block; Before the image is input into each processing module, it is first processed using a stem convolution module. An image consisting of two channels , , composed of 3×3 convolutional layers with a stride of 2, generating Features, The input dimension of the Multi-branch Network block.

3. The method for scene classification of marine remote sensing images based on multi-branch fusion according to claim 1, It is characterized in that The multi-scale embedding module is used to generate embedding features of different sizes and input them into the multi-branch module including: The multi-scale embedding module adjusts the sequence length of the token by changing the stride and padding, outputs features of the same shape with different patch sizes, and also achieves 2×2 downsampling of the current stage features by adjusting the stride; it is implemented using 3×3 depthwise separable convolution, which consists of 3×3 depthwise convolution and 1×1 pointwise convolution in the embedding layer; batch normalization and Hardswish function are added after all convolutional layers; finally, the token embedding features of different sizes are input into the multi-branch module respectively; The multi-scale embedding module uses a deep residual bottleneck layer consisting of 1×1 convolution, 3×3 depth convolution, 1×1 convolution and residual connection to process the embedded features, and at the same time inputs these features of different scales into the multi-branch module.

4. The method for classifying marine remote sensing images based on multi-branch fusion according to claim 1, It is characterized in that The Transformer branch is used to obtain multi-scale features of image data according to the embedded features output by the multi-scale embedding module, including: The Transformer branch uses the classic Transformer encoder structure, uses factorized self-attention and convolutional relative position encoding to form a new convolutional attention module; specifically, it includes: The input token is first convolutionally encoded and then input into the convolutional attention module, which includes the factorization attention module and the convolutional relative position encoding module. The factorization attention formula is as follows: ； ； ； in, , , Three vectors representing self-attention, Represents the image feature data input to the Transformer encoder, Indicates the number of channels, superscript T indicates matrix transposition, the first formula is the calculation formula of the original attention mechanism, and the second formula is the factorized attention formula. It can be seen from the formula that on the basis of the original, through two functions and right and Decompose and calculate together and Multiply to approximate the original attention map; the third formula is the method used, the function P uses the identity mapping and adds the scaling factor ,function Use the softmax function; In order to model the relationship between local tokens, deep convolution is used to encode relative positions and obtain the attention map of relative positions. , combining the factorized self-attention and position encoding to obtain the convolutional attention, the specific formula is as follows: ； ； ； in, The attention map representing relative positions emphasizes how local features are extracted through convolution and combined with the query vector to capture local relationships; Represents a comprehensive attention map. By connecting multiple attention maps, the diversity of information is enhanced, which helps to improve the expressiveness and accuracy of the model. represents Hadamard Product, and W represents the weight value of the convolution kernel.

5. The method for classifying marine remote sensing images based on multi-branch fusion according to claim 1, Features: The kernel selection aggregation module, namely the kernel selection multi-scale aggregation module (KSMSA), uses different large convolution kernels and dynamic receptive fields to perceive information of different scales, effectively models different contextual nuances of objects of different categories, further improves the efficiency of fusion, and enhances feature representation; The core of the kernel selection multi-scale aggregation module is the large kernel selection module and the convolution layer for fusion and channel transformation. The large kernel selection module is embedded in two 1×1 convolution layers for enhancing the network representation capability. Its main steps are separation, fusion and selection. For the features of each branch of the input, the large kernel selection module first performs convolution calculations with convolution kernel sizes of 5 and 7 to obtain features processed with different convolution kernel sizes: and , these convolutions are implemented using efficient grouped convolutions, allowing channel mixing for each spatial feature vector; the purpose of aggregation is to be able to adaptively adjust the size of the receptive field according to the content. The basic idea is to use gates to control the information flow of the two convolution kernel branches; to achieve this, the gating mechanism needs to integrate this information and splice these features, as shown in the following formula, combining the features of the two branches: ; In order to enhance the network's ability to focus on relevant spatial context areas, a gating mechanism is used to select feature maps from large convolution kernel spaces of different scales; in order to effectively extract spatial relationships, a parallel operation of global average pooling and global maximum pooling is used, and then spliced ​​together again to generate spatial-wise statistical information. The formula is as follows: ； ； ； in, , Represents the height and width of the feature map, Indicates aisle, Represents the first Channel characteristics, Represents the first Channel characteristics, Represents the concatenation of two pooling features. Channel characteristics, Represents the splicing process; then the convolution operation is performed to convert the pooled features into two spatial attention maps to achieve information interaction in different spaces; the sigmoid activation function is used to obtain the selection mask of each decomposed large core space to provide guidance for accurate and adaptive selection. The specific formula is as follows: ； in, represents the sigmoid activation function, represents the convolution operation, Indicates all channels ; Use the weight matrix to reweight and sum the features to obtain the final output features of the current scale branch , and in order to prevent gradient disappearance and gradient explosion, a residual connection is added, as shown in the following formula: ； The branch features processed by the large kernel selection module are spliced ​​together, and then channel transformed through 1×1 convolution to complete the multi-branch feature fusion of the current stage to input the multi-scale embedding module of the next stage; by using different convolution kernels to process features of different scales, the dynamic receptive field is obtained, the semantic differences between multi-scale features are effectively aligned, and multi-branch feature aggregation is better realized, the feature representation capability is enhanced, and the classification effect is improved.

6. The method for classifying marine remote sensing images based on multi-branch fusion according to claim 1, It is characterized in that The learnable multi-level aggregation module in step S2 is used to fuse the multiple multi-branch network blocks, including: The learnable multi-level aggregation module downsamples the features of the first multi-branch network block, the second multi-branch network block and the third multi-branch network block to obtain feature maps of the same size, then concatenates these features together, and then calculates the fusion features of the shallow information through SKConv; then, the fusion features of the shallow information are obtained with the set learnable hyperparameters Multiply them together to achieve dynamic summation in order to generate the final features that make up for the shallow information.

7. The method for classifying marine remote sensing images based on multi-branch fusion according to claim 1, It is characterized in that In step S3, label smoothing loss is introduced as a loss function in the training of the ocean remote sensing scene classification model based on multi-branch fusion; the label smoothing loss changes the one-hot hard coding method of the label in the traditional cross entropy loss to soft coding, and adds a little noise in the one-hot to make the model more robust; the loss function formula is shown in the following formula: ； ； in, represents the predicted value of the classifier output, represents one-hot encoding, represents the true label, is a small constant, is the number of categories.

8. A computer-readable storage medium storing a computer program, It is characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images