Port area remote sensing image segmentation method and device, electronic equipment and medium

Abstract

The present application relates to the technical field of image processing, and provides a port area remote sensing image segmentation method and device, electronic equipment and medium, the method comprises the following steps: collecting a port area remote sensing image; inputting the port area remote sensing image into an image segmentation model to obtain a contour segmentation image of the port area remote sensing image output by the image segmentation model; wherein the image segmentation model comprises a reparameterization network, a dense connection structure and a dilated convolution layer. The present application uses a pre-constructed image segmentation model to perform port area contour segmentation on a port area remote sensing image, the image segmentation model uses a structural reparameterization network to extract shallow feature maps, middle feature maps and first deep feature maps of the port area remote sensing image, and performs feature extraction and feature fusion on the first deep feature maps through a dense connection and a dilated convolution layer, thereby improving the segmentation accuracy of the port area remote sensing image.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, electronic device, and medium for segmenting remote sensing images of a port area. Background Technology

[0002] With the development of remote sensing technology, port scenes are becoming increasingly complex. Traditional remote sensing image processing methods for port areas are not easy to effectively analyze remote sensing images, cannot effectively extract information from the features of complex scenes, and are difficult to accurately segment the coastline of remote sensing images.

[0003] While traditional convolutional neural networks (CNNs) have a certain degree of universality in semantic segmentation, their performance is less than ideal when segmenting remote sensing images of port areas. With the development of remote sensing technology, the resolution and information content of remote sensing images are increasing daily, resulting in a large number of structural parameters for traditional CNNs. This makes them unsuitable for efficiently processing large-scale remote sensing port images. Furthermore, feature differences at different scales and the blurring of some boundaries in nearshore areas of remote sensing images significantly impact the segmentation accuracy of traditional CNNs. Additionally, the imbalance of marine and land sample categories in remote sensing images can also affect segmentation accuracy. Summary of the Invention

[0004] This invention provides a method, apparatus, electronic device, and medium for segmenting remote sensing images of port areas, addressing the problem of low segmentation accuracy in remote sensing images of port areas. It employs a pre-constructed image segmentation model to segment the port area contour of the remote sensing image. This model uses a structured reparameterization network to extract shallow, mid-level, and first deep feature maps from the port area remote sensing image. Furthermore, it utilizes dense connections and dilated convolutional layers to extract and fuse features from the first deep feature map, thereby improving the segmentation accuracy of the port area remote sensing image.

[0005] This invention provides a method for segmenting remote sensing images of a port area, comprising:

[0006] Collect remote sensing images of the port area;

[0007] The remote sensing image of the port area is input into the image segmentation model to obtain the contour segmentation image of the remote sensing image of the port area output by the image segmentation model.

[0008] The image segmentation model's encoder includes a feature extraction module and a scale feature fusion module. The feature extraction module extracts feature information from the remote sensing image of the port area based on a structure reparameterization network. The feature information includes a shallow feature map, a mid-level feature map, and a first deep feature map. The structure reparameterization network is trained using a multi-branch network and inferred using a single-branch network. During inference, the parameters of the multi-branch network and the single-branch network are fused. The scale feature fusion module extracts and fuses features from the first deep feature map through dense connections and dilated convolutional layers.

[0009] In one embodiment, the image segmentation model is obtained through the following steps:

[0010] Collect remote sensing images of the sample port area;

[0011] Extract sample feature information from the remote sensing image of the sample port area, the sample feature information including shallow sample feature map, middle sample feature map and first deep sample feature map;

[0012] The deep feature map of the first sample is obtained by extracting and fusing features through dense connections and dilated convolutional layers.

[0013] Upsampling is performed on the deep feature map of the second sample to obtain the deep feature map of the third sample;

[0014] A sample feature map is obtained by performing skip connections on the deep feature map of the third sample, the shallow feature map of the sample, and the middle feature map of the sample.

[0015] Convolution is performed on the sample feature map to obtain the predicted contour segmentation map of the remote sensing image of the sample port area;

[0016] The image segmentation model is obtained by training a preset loss function using the predicted contour segmentation map and the real contour segmentation map.

[0017] In one embodiment, the multi-branch network includes convolutional layers and batch normalization (BN) layers;

[0018] The operation formula for the convolutional layer is:

[0019] y conv =ω·x+b;

[0020] Among them, y conv For convolution, x is the input, ω is the weight, and b is the bias;

[0021] The operation formula for the BN layer is:

[0022]

[0023] Among them, BN γ，β (y conv ) represents the BN layer before fusion, γ represents the translation parameter updated during network backpropagation, β represents the scaling parameter updated during network backpropagation, and μ represents the scaling parameter. B Let σ be the mean of all training data. B Let be the variance of all training data, and ε be a constant.

[0024] In one embodiment, the fusion of parameters of the multi-branch network and the single-branch network includes: fusion between the BN layer and the convolutional layer, and fusion between the convolutional layers.

[0025] The fusion formula between the BN layer and the convolutional layer is as follows:

[0026]

[0027] Among them, BN γ，β (x) represents the fused BN layer, ω is the weight, b is the bias, γ is the translation parameter updated during backpropagation, β is the scaling parameter updated during backpropagation, and μ B Let σ be the mean of all training data. B Let ε be the variance of all training data, ε be a constant, and x be the input.

[0028] The fusion formula between the convolutional layers is:

[0029]

[0030] Where O represents the fused features, I represents the input features, K1, K2, and K3 are the convolution parameters, and B1, B2, and B3 are the offsets.

[0031] In one embodiment, the dilated convolutional layer includes multiple dilated convolutions with different sampling rates, used to extract multi-scale feature information from the first deep feature map and generate a second deep feature map.

[0032] In one embodiment, the Nth dilated convolutional layer in the dense connection is connected to the second deep feature map of the N-1 dilated convolutional layers, wherein the second deep feature map of the Nth dilated convolutional layer is generated by combining the feature information of the second deep feature maps of the N-1 dilated convolutional layers, and all the second deep feature maps are fused through dense connection to generate a third deep feature map.

[0033] In one embodiment, the formula for calculating the preset loss function is:

[0034] loss(x,class)=weight[class](x[class]+log∑ j exp(x[j]));

[0035] Where x is the predicted value, j is the number of iterations, loss is the loss, weight is the weight, and class is the pixel category of the segmentation.

[0036] The present invention also provides a device for contour segmentation of remote sensing images of a port area, comprising:

[0037] The image acquisition module is used to acquire remote sensing images of the port area;

[0038] The image segmentation module is used to input the remote sensing image of the port area into the image segmentation model to obtain the contour segmentation image of the remote sensing image of the port area output by the image segmentation model.

[0039] The image segmentation model's encoder includes a feature extraction module and a scale feature fusion module. The feature extraction module extracts feature information from the remote sensing image of the port area based on a structure reparameterization network. The feature information includes a shallow feature map, a mid-level feature map, and a first deep feature map. The structure reparameterization network is trained using a multi-branch network and inferred using a single-branch network. During inference, the parameters of the multi-branch network and the single-branch network are fused. The scale feature fusion module extracts and fuses features from the first deep feature map through dense connections and dilated convolutional layers.

[0040] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the segmentation method for remote sensing images of port areas as described above.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the segmentation method for remote sensing images of port areas as described above.

[0042] The present invention provides a method, apparatus, electronic device, and medium for segmenting remote sensing images of port areas. The method involves acquiring remote sensing images of a port area; inputting the remote sensing images of the port area into an image segmentation model to obtain a contour segmentation image of the remote sensing images of the port area output by the image segmentation model; wherein the encoder of the image segmentation model includes a feature extraction module and a scale feature fusion module; the feature extraction module extracts feature information of the remote sensing images of the port area based on a structure reparameterization network, the feature information including a shallow feature map, a mid-level feature map, and a first deep feature map; the structure reparameterization network is trained using a multi-branch network and inferred using a single-branch network, and the parameters of the multi-branch network and the single-branch network are fused during the inference process; the scale feature fusion module extracts and fuses features from the first deep feature map through dense connections and dilated convolutional layers. This invention employs a pre-constructed image segmentation model to segment the port area contour of remote sensing images. This image segmentation model uses a structured reparameterization network to extract shallow feature maps, mid-level feature maps, and a first deep feature map from the remote sensing images of the port area. It then uses dense connections and dilated convolutional layers to extract and fuse features from the first deep feature map, thereby improving the segmentation accuracy of the remote sensing images of the port area. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0044] Figure 1 This is one of the flowcharts illustrating the segmentation method for remote sensing images of port areas provided by the present invention;

[0045] Figure 2 This is the second flowchart illustrating the method for segmenting remote sensing images of port areas provided by the present invention.

[0046] Figure 3 This invention provides a multi-branch network structure for a structurally reparameterized network.

[0047] Figure 4 This is a comparative diagram of dilated convolution and ordinary convolution provided by the present invention;

[0048] Figure 5 This is a schematic diagram of the structure of the hollow convolutional layer with a dense structure provided by the present invention;

[0049] Figure 6 This is a schematic diagram of the structure of the port area remote sensing image contour segmentation device provided by the present invention;

[0050] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] The following is combined Figures 1-7 The present invention describes a method, apparatus, electronic device, and medium for segmenting remote sensing images of port areas.

[0053] Specifically, the present invention provides a method for segmenting remote sensing images of a port area, referring to... Figure 1 , 2 .

[0054] The method for segmenting remote sensing images of port areas provided in this embodiment of the invention includes:

[0055] S100, acquires remote sensing images of the port area;

[0056] It should be noted that remote sensing images refer to films or photographs that record the magnitude of electromagnetic waves emitted by various ground objects. Remote sensing images have characteristics such as spatial resolution, spectral resolution, time series, and multi-source data fusion, enabling comprehensive monitoring and observation of the Earth's surface.

[0057] A port is located along the coast of the sea, river, lake, or reservoir.

[0058] Remote sensing images of the port area are collected through platforms such as satellites, aircraft, and drones, or through ground-based measuring instruments.

[0059] Furthermore, in order to improve the accuracy of contour segmentation in remote sensing images of the port area, it is necessary to preprocess the remote sensing images of the port area after acquisition, such as image fusion, image segmentation, linear stretching, and image filtering.

[0060] S200: Input the remote sensing image of the port area into the image segmentation model and obtain the contour segmentation image of the remote sensing image of the port area output by the image segmentation model.

[0061] The image segmentation model's encoder includes a feature extraction module and a scale feature fusion module. The feature extraction module extracts feature information from remote sensing images of the port area based on a structure reparameterization network. The feature information includes shallow feature maps, mid-level feature maps, and a first deep feature map. The structure reparameterization network is trained using a multi-branch network and inferred using a single-branch network. During inference, the parameters of the multi-branch network and the single-branch network are fused. The scale feature fusion module extracts and fuses features from the first deep feature map through dense connections and dilated convolutional layers.

[0062] The preprocessed remote sensing image of the port area is input into the image segmentation model. The feature extraction module in the image segmentation model extracts shallow feature maps, mid-level feature maps, and a first deep feature map from the remote sensing image of the port area based on structured reparameters. The scale feature fusion module extracts and fuses features from the first deep feature map through dense connections and dilated convolutional layers to obtain a third deep feature map; the receptive field is expanded through dilated convolutional layers, and the feature context information in the first deep feature map is enhanced through dense connections.

[0063] Skip connections are made to the third deep feature map, shallow feature map, and middle feature map to fuse them into a feature map, thereby improving the robustness of the image segmentation model. Convolution and upsampling are then performed on the feature map to obtain the contour segmentation image of the remote sensing image of the port area.

[0064] The port area remote sensing image segmentation method provided in this embodiment of the invention involves acquiring remote sensing images of the port area; inputting the remote sensing images of the port area into an image segmentation model to obtain a contour segmentation image of the port area remote sensing image output by the image segmentation model; wherein, the encoder of the image segmentation model includes a feature extraction module and a scale feature fusion module; the feature extraction module extracts feature information of the port area remote sensing image based on a structure reparameterization network, the feature information including shallow feature map, mid-level feature map, and first deep feature map; the structure reparameterization network is trained using a multi-branch network and inferred using a single-branch network, and the parameters of the multi-branch network and the single-branch network are fused during the inference process; the scale feature fusion module extracts and fuses features from the first deep feature map through dense connections and dilated convolutional layers. This invention employs a pre-constructed image segmentation model to segment the port area contour of remote sensing images. This image segmentation model uses a structured reparameterization network to extract shallow feature maps, mid-level feature maps, and a first deep feature map from the remote sensing images of the port area. It then uses dense connections and dilated convolutional layers to extract and fuse features from the first deep feature map, thereby improving the segmentation accuracy of the remote sensing images of the port area.

[0065] Based on the above embodiments, the image segmentation model is obtained through the following steps:

[0066] S201, Collect remote sensing images of the sample port area;

[0067] S202, Extract sample feature information from remote sensing images of the port area. The sample feature information includes shallow feature map, middle feature map and deep feature map of the first sample.

[0068] S203, through dense connections and dilated convolutional layers, feature extraction and feature fusion are performed on the deep feature map of the first sample to obtain the deep feature map of the second sample;

[0069] S204, Upsample the deep feature map of the second sample to obtain the deep feature map of the third sample;

[0070] S205, perform a skip connection on the deep feature map of the third sample, the shallow feature map of the sample, and the middle feature map of the sample to obtain the sample feature map;

[0071] S206, Convolve the sample feature map to obtain the predicted contour segmentation map of the remote sensing image of the sample port area;

[0072] S207, The image segmentation model is obtained by training a preset loss function using the predicted contour segmentation map and the real contour segmentation map.

[0073] It should be noted that the image segmentation model is obtained by training a preset model using remote sensing images of the sample port area. The preset model includes an encoder and a decoder. The encoder includes a feature extraction module and a scale feature fusion module.

[0074] After acquiring remote sensing images of the sample port area, the images are preprocessed and then input into the encoder. The encoder's feature extraction module extracts features from the preprocessed remote sensing images of the sample port area through a structure reparameterization network, obtaining the sample shallow feature map, sample middle feature map, and first sample deep feature map of the sample port area remote sensing images.

[0075] The encoder's scale feature fusion module fuses contextual information from different scales of the first sample deep feature map through dilated convolutions with varying dilation rates. It also captures global information from the first sample deep feature map using a global pooling layer to obtain the second deep feature map. Before inputting the first sample deep feature map into the multi-scale feature pyramid, a pre-convolutional layer is introduced. This pre-convolutional layer downsamples the first sample deep feature map, resulting in a reduced first sample deep feature map. This reduced first sample deep feature map and the original first sample deep feature map are then input into the multi-scale feature pyramid for feature extraction to obtain the second sample deep feature map. Understandably, the pre-convolutional layer helps the network better handle small targets in the first sample deep feature map.

[0076] The decoder includes an upsampling module, skip connections, and a 1x1 convolution module. Because the encoder uses downsampling, the decoder uses upsampling to restore the second sample deep feature map to the original size of the first sample deep feature map, thus obtaining the third sample deep feature map.

[0077] Optionally, upsampling can be performed using the bilinear interpolation method, which can rapidly amplify the deep feature map of the second sample.

[0078] A sample feature map is obtained by skip-connecting the shallow and medium-level sample feature maps of the encoder with the third deep sample feature map of the decoder. This preserves more detailed image information. For example, skip connections include: concatenating the shallow and medium-level sample feature maps of the encoder with the third deep sample feature map of the decoder to obtain the sample feature map; or concatenating the shallow and medium-level sample feature maps of the encoder with the third deep sample feature map of the decoder after 1*1 convolution compression to obtain the sample feature map.

[0079] Finally, the decoder performs multiple 1x1 convolution operations on the sample feature maps to extract features. Multiple 1x1 convolutions reduce computation and parameter count. Specifically, the multiple 1x1 convolution operations are implemented using two 1x1 convolutional layers, which adjust the number of channels and integrate features from the skip-connected sample feature maps, respectively. The sample feature maps after multiple convolutions are mapped to the semantic segmentation result. This classifier includes a 1x1 convolutional layer and an interpolation layer. The convolutional layer with a kernel size of 1 reduces the number of channels in the feature map to achieve channel compression. Then, the selected interpolation method is used to upsample the sample feature maps, resulting in a predicted contour segmentation map of the remote sensing image of the sample port area.

[0080] The port outline in the remote sensing image of the sample port area is labeled to obtain the true outline segmentation map.

[0081] The image segmentation model is obtained by training a preset loss function using predicted contour segmentation maps and real contour segmentation maps.

[0082] This invention provides an image segmentation model trained on a preset model using remote sensing images of a sample port area. The training process employs a structured reparameterization network to extract shallow, mid-level, and deep feature maps from the remote sensing images of the sample port area. These deep feature maps are then fused using dense connections and dilated convolutional layers to obtain the image segmentation model, thereby improving its segmentation accuracy.

[0083] Based on the above embodiments, the multi-branch network includes convolutional layers and batch normalization (BN) layers; the operation formula for the convolutional layer is:

[0084] y conv =ω·x+b;

[0085] Among them, y conv For convolution, x is the input, ω is the weight, and b is the bias;

[0086] The operation formula for the BN layer is:

[0087]

[0088] Among them, BN γ，β (y conv ) represents the BN layer before fusion, γ represents the translation parameter updated during network backpropagation, β represents the scaling parameter updated during network backpropagation, and μ represents the scaling parameter. B Let σ be the mean of all training data. B Let be the variance of all training data, and ε be a constant.

[0089] Through structural reparameterization, a multi-branch network is used for training, while a single-branch network is used for inference. The training process employs a multi-branch structure, which includes convolutional layers and batch normalization (BN) layers. The bottommost branch includes a BN layer, as shown below. Figure 3 As shown.

[0090] This invention improves the feature extraction capability of the structure reparameterization network by setting convolutional layers and batch normalization (BN) layers in the structure reparameterization network, and also speeds up the inference process.

[0091] Based on the above embodiments, the fusion of parameters of multi-branch networks and single-branch networks includes: fusion between BN layers and convolutional layers, and fusion between convolutional layers.

[0092] The fusion formula between the BN layer and the convolutional layer is as follows:

[0093]

[0094] Among them, BN γ，β (x) represents the fused BN layer, ω is the weight, b is the bias, γ is the translation parameter updated during backpropagation, β is the scaling parameter updated during backpropagation, and μ B Let σ be the mean of all training data. B Let ε be the variance of all training data, ε be a constant, and x be the input.

[0095] The formula for merging convolutional layers is:

[0096]

[0097] Where O represents the fused features, I represents the input features, K1, K2, and K3 are the convolution parameters, and B1, B2, and B3 are the offsets.

[0098] During inference, all branch parameters are fused, including the fusion between BN layers and convolutional layers, and the fusion between convolutional layers.

[0099] During the fusion of BN layers and convolutional layers, if Setting ε to 0 will cause calculation errors. To avoid this, we can set ε to a very small constant, let:

[0100]

[0101]

[0102] Therefore, we can obtain:

[0103]

[0104] in, The weights after setting This is the bias after setting.

[0105] By fusing the parameters of the convolutional layer and the BN layer, inference can be performed using a single-branch network.

[0106] When merging convolutional layers, the 1*1 convolutional branches and the branches with only BN layers are first expanded into 3*3 convolutional branches. The expansion method is as follows: the 1*1 convolution is padded with 0 to make it 3*3, and the branches with only BN layers are also expanded with 0 to make them 3*3 convolutions. Then, the additivity of convolution is used to merge these convolutional branches together.

[0107] This invention achieves inference using a single-branch network by fusing BN layers with convolutional layers and convolutional layers with each other, thereby accelerating network inference speed and reducing computational load during inference, thus improving the network's inference speed.

[0108] Based on the above embodiments, the dilated convolutional layer includes multiple dilated convolutions with different sampling rates, used to extract multi-scale feature information from the first deep feature map and generate a second deep feature map.

[0109] like Figure 4As shown, dilated convolution creates holes in the convolutional layer through equally spaced sampling. By changing the dilation rate, dilated convolution adjusts the size of features calculated by the feature extraction network, thus obtaining different receptive field sizes. Compared to ordinary convolution operations, dilated convolution can increase the receptive field of a module without changing the number of parameters, thereby extracting complex features from the input image more efficiently. Furthermore, dilated convolution can acquire more multi-scale context by adjusting the dilation rate, and utilize this multi-scale context information to extract features at different scales. By using multiple dilated convolutional layers with different sampling rates to collect first-deep feature maps, multiple second-deep feature maps of the same size are obtained, solving the problem of complex port area features and effectively improving the network's feature extraction capability without increasing computational load.

[0110] Dilated convolutional layers acquire multi-scale feature information from the first deep feature map through dilated convolution with different sampling rates, and generate a second deep feature map. By using dilated convolution with corresponding dilation factors on feature maps of different scales, the dilated convolutional layers ensure that the outputs are all second deep feature maps of the same size, allowing the network to overcome scale limitations during both training and inference processes.

[0111] In this embodiment of the invention, dilated convolutional layers are used to extract features from the first deep feature map. For example, four different dilated convolutional layers with varying dilation rates are used to perform convolution operations on the output of the previous layer, expanding the receptive field in one operation to obtain multi-scale information, and then upsampling is performed to restore the correct dimensionality. By constructing dilated convolutions with different receptive fields, the feature map is reduced without increasing computational cost, thereby obtaining multi-scale feature information, such as... Figure 4 As shown.

[0112] In this embodiment of the invention, four types of dilated convolutional layers are convolved to obtain four second deep feature maps of the same size. Then, the four second deep feature maps of the same size are fused through dense connections to obtain a third deep feature map.

[0113] In this embodiment of the invention, by employing densely connected and dilated convolutional layers to extract and fuse features from the first deep feature map, the utilization rate of the first deep feature map is improved, which is beneficial to improving the accuracy of the image segmentation model.

[0114] Based on the above embodiments, the Nth dilated convolutional layer in the dense connection is connected to the second deep feature maps of N-1 dilated convolutional layers. The second deep feature map of the Nth dilated convolutional layer is generated by combining the feature information of the second deep feature maps of N-1 dilated convolutional layers. All the second deep feature maps are fused through dense connection to generate the third deep feature map.

[0115] like Figure 5As shown, dense connections achieve high feature sharing and high information flow in the network by connecting the outputs of all convolutional layers, resulting in better feature reuse capabilities and fewer parameters.

[0116] It should be further noted that N is an integer greater than or equal to 2.

[0117] The Nth dilated convolutional layer is connected to the second deep feature maps of the preceding N-1 dilated convolutional layers, and the deep feature information from the N-1 second deep feature maps of these N-1 dilated convolutional layers is obtained. The Nth dilated convolutional layer combines the deep feature information to generate the second deep feature map of the Nth dilated convolutional layer.

[0118] In this embodiment of the invention, the number of dilated convolutional layers is 4. The 4 dilated convolutional layers generate a total of 4 second deep feature maps. The 4 second deep feature maps are fused to obtain a third deep feature map.

[0119] Using dense connections in dilated convolutions allows the network to better utilize feature information. Each dilated convolutional layer connects to the outputs of all preceding dilated convolutional layers, allowing each dilated convolutional layer to receive feature information from the outputs of all preceding dilated convolutional layers.

[0120] In this embodiment of the invention, by connecting the outputs of all preceding dilated convolutional layers to each dilated convolutional layer, the reusability and expressive power of the second deep feature map are improved, allowing the second deep feature map to be utilized more comprehensively. At the same time, the problems of information loss and feature duplication are avoided. Furthermore, by utilizing dense connection modules to achieve shared features, the number of network parameters is reduced.

[0121] Based on the above embodiments, the formula for calculating the preset loss function is as follows:

[0122] loss(x,class)=weight[class](x[class]+log∑ j exp(x[j]));

[0123] Where x is the predicted value, j is the number of iterations, loss is the loss, weight is the weight, and class is the pixel category of the segmentation.

[0124] Because the imbalance of samples in the remote sensing images of port areas can easily affect segmentation accuracy during segmentation, and because remote sensing images of port areas often face the problem of imbalanced marine and land samples, this invention uses a weighted loss function to optimize the image segmentation model. In semantic segmentation tasks, the cross-entropy loss function is typically used to train the model, and its calculation formula is as follows:

[0125]

[0126] Where x is the predicted value, j is the number of iterations, loss is the loss, and class is the pixel category of the segmentation.

[0127] However, since this loss is calculated by summing the pixels, it cannot effectively solve the segmentation problem caused by class imbalance in remote sensing images of port areas. To address the sample imbalance between marine and land port areas in remote sensing images of port areas, this embodiment of the invention adds a weighting factor weight[class] to the cross-entropy loss function, making it a weighted loss function.

[0128] This invention, by adding a weighted loss function to the cross-entropy loss function, better overcomes the problem of land-sea sample imbalance in remote sensing images of port areas, and further improves the accuracy of the image segmentation model.

[0129] like Figure 6 As shown, this embodiment of the invention also provides a port area remote sensing image contour segmentation device, comprising:

[0130] Image acquisition module 601 is used to acquire remote sensing images of the port area;

[0131] Image segmentation module 602 is used to input the remote sensing image of the port area into the image segmentation model to obtain the contour segmentation image of the remote sensing image of the port area output by the image segmentation module;

[0132] The image segmentation model's encoder includes a feature extraction component and a scale feature fusion component. The feature extraction component extracts feature information from remote sensing images of the port area based on a structure reparameterization network. The feature information includes shallow feature maps, mid-level feature maps, and a first deep feature map. The structure reparameterization network is trained using a multi-branch network and inferred using a single-branch network. During inference, the parameters of the multi-branch network and the single-branch network are fused. The scale feature fusion component extracts and fuses features from the first deep feature map through dense connections and dilated convolutional layers.

[0133] The port area remote sensing image contour segmentation device of this invention acquires remote sensing images of the port area; inputs the remote sensing images of the port area into an image segmentation model, and obtains the contour segmentation image of the port area remote sensing image output by the image segmentation model; wherein, the encoder of the image segmentation model includes a feature extraction module and a scale feature fusion module; the feature extraction module extracts feature information of the port area remote sensing image based on a structure reparameterization network, the feature information including shallow feature map, mid-level feature map, and first deep feature map; the structure reparameterization network is trained using a multi-branch network and inferred using a single-branch network, and the parameters of the multi-branch network and the single-branch network are fused during the inference process; the scale feature fusion module extracts and fuses features from the first deep feature map through dense connections and dilated convolutional layers. This invention employs a pre-constructed image segmentation model to segment the port area contour of remote sensing images. This image segmentation model uses a structured reparameterization network to extract shallow feature maps, mid-level feature maps, and a first deep feature map from the remote sensing images of the port area. It then uses dense connections and dilated convolutional layers to extract and fuse features from the first deep feature map, thereby improving the segmentation accuracy of the remote sensing images of the port area.

[0134] In one embodiment, the port area remote sensing image contour segmentation device further includes a model training module;

[0135] The model training module is used to acquire remote sensing images of the sample port area; extract sample feature information from the remote sensing images of the sample port area, including shallow feature maps, mid-level feature maps, and a first deep feature map; extract and fuse features from the first deep feature map through dense connections and dilated convolutional layers to obtain a second deep feature map; upsample the second deep feature map to obtain a third deep feature map; perform skip connections on the third deep feature map, shallow feature map, and mid-level feature map to obtain a sample feature map; convolve the sample feature maps to obtain a predicted contour segmentation map of the remote sensing image of the sample port area; and train a preset loss function using the predicted contour segmentation map and the true contour segmentation map to obtain an image segmentation model.

[0136] In one embodiment, the multi-branch network includes convolutional layers and batch normalization (BN) layers; the operation formula for the convolutional layer is:

[0137] y conv =ω·x+b;

[0138] Among them, y conv For convolution, x is the input, ω is the weight, and b is the bias;

[0139] The operation formula for the BN layer is:

[0140]

[0141] Among them, BN γ，β (y conv ) represents the BN layer before fusion, γ represents the translation parameter updated during network backpropagation, β represents the scaling parameter updated during network backpropagation, and μ represents the scaling parameter. B Let σ be the mean of all training data. B Let be the variance of all training data, and ε be a constant.

[0142] In one embodiment, the fusion of parameters of the multi-branch network and the single-branch network includes: fusion between BN layers and convolutional layers and fusion between convolutional layers.

[0143] The fusion formula between the BN layer and the convolutional layer is as follows:

[0144]

[0145] Among them, BN γ，β (x) represents the fused BN layer, ω is the weight, b is the bias, γ is the translation parameter updated during backpropagation, β is the scaling parameter updated during backpropagation, and μ B Let σ be the mean of all training data. B Let ε be the variance of all training data, ε be a constant, and x be the input.

[0146] The formula for merging convolutional layers is:

[0147]

[0148] Where O represents the fused features, I represents the input features, K1, K2, and K3 are the convolution parameters, and B1, B2, and B3 are the offsets.

[0149] In one embodiment, the dilated convolutional layer includes multiple dilated convolutions with different sampling rates to extract multi-scale feature information from the first deep feature map and generate a second deep feature map.

[0150] In one embodiment, the Nth dilated convolutional layer in the dense connection is connected to the second deep feature maps of N-1 dilated convolutional layers. The second deep feature map of the Nth dilated convolutional layer is generated by combining the feature information of the second deep feature maps of N-1 dilated convolutional layers. All the second deep feature maps are fused through dense connection to generate the third deep feature map.

[0151] In one embodiment, the formula for calculating the preset loss function is:

[0152] loss(x,class)=weight[class](x[class]+log∑j exp(x[j]));

[0153] Where x is the predicted value, j is the number of iterations, loss is the loss, weight is the weight, and class is the pixel category of the segmentation.

[0154] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a segmentation method for a remote sensing image of a port area, the method including:

[0155] Collect remote sensing images of the port area;

[0156] Input the remote sensing image of the port area into the image segmentation model to obtain the contour segmentation image of the remote sensing image of the port area output by the image segmentation model;

[0157] The image segmentation model's encoder includes a feature extraction module and a scale feature fusion module. The feature extraction module extracts feature information from remote sensing images of the port area based on a structure reparameterization network. The feature information includes shallow feature maps, mid-level feature maps, and a first deep feature map. The structure reparameterization network is trained using a multi-branch network and inferred using a single-branch network. During inference, the parameters of the multi-branch network and the single-branch network are fused. The scale feature fusion module extracts and fuses features from the first deep feature map through dense connections and dilated convolutional layers.

[0158] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0159] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform segmentation of remote sensing images of port areas provided by the methods described above, the method comprising:

[0160] Collect remote sensing images of the port area;

[0161] Input the remote sensing image of the port area into the image segmentation model to obtain the contour segmentation image of the remote sensing image of the port area output by the image segmentation model;

[0162] The image segmentation model's encoder includes a feature extraction module and a scale feature fusion module. The feature extraction module extracts feature information from remote sensing images of the port area based on a structure reparameterization network. The feature information includes shallow feature maps, mid-level feature maps, and a first deep feature map. The structure reparameterization network is trained using a multi-branch network and inferred using a single-branch network. During inference, the parameters of the multi-branch network and the single-branch network are fused. The scale feature fusion module extracts and fuses features from the first deep feature map through dense connections and dilated convolutional layers.

[0163] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0164] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for segmenting a remote sensing image of a port area, characterized in that, The method comprises the following steps: collecting a port area remote sensing image; inputting the port area remote sensing image into an image segmentation model to obtain a contour segmentation image of the port area remote sensing image output by the image segmentation model; wherein the encoder of the image segmentation model comprises a feature extraction module and a scale feature fusion module; the feature extraction module extracts feature information of the port area remote sensing image based on a structure reparameterization network, and the feature information comprises a shallow feature map, a middle feature map and a first deep feature map; the structure reparameterization network is trained by using a multi-branch network and inferred by using a single-branch network, and parameters of the multi-branch network and the single-branch network are fused in the inference process; the scale feature fusion module performs feature extraction and feature fusion on the first deep feature map through dense connection and a dilated convolution layer; the image segmentation model is obtained by the following steps: collecting a sample port area remote sensing image; extracting sample feature information of the sample port area remote sensing image, wherein the sample feature information comprises a sample shallow feature map, a sample middle feature map and a first sample deep feature map; performing feature extraction and feature fusion on the first sample deep feature map through dense connection and a dilated convolution layer to obtain a second sample deep feature map; up-sampling the second sample deep feature map to obtain a third sample deep feature map; performing jump connection on the third sample deep feature map, the sample shallow feature map and the sample middle feature map to obtain a sample feature map; performing convolution on the sample feature map to obtain a predicted contour segmentation image of the sample port area remote sensing image; training a preset loss function by using the predicted contour segmentation image and a real contour segmentation image to obtain the image segmentation model.

2. The segmentation method of a port area remote sensing image according to claim 1, characterized in that, the multi-branch network comprises a convolution layer and a BN layer; the operation formula of the convolution layer is: y conv = ω · x + b; where y conv is a convolution, x is an input, ω is a weight, and b is a bias. the operation formula of the BN layer is: where BN γ，β (y conv ) is the BN layer before fusion, γ is the translation parameter updated during the network back propagation process, β is the scaling parameter updated during the network back propagation process, μ B is the mean of all training data, σ B is the variance of all training data, and ε is a constant.

3. The segmentation method of a port area remote sensing image according to claim 2, characterized in that, the fusion of the parameters of the multi-branch network and the single-branch network comprises fusion between the BN layer and the convolution layer, and fusion between the convolution layers; wherein the fusion formula between the BN layer and the convolution layer is: where BN γ，β (x) is the BN layer after fusion, ω is the weight, b is the bias, γ is the translation parameter updated during the network back propagation process, β is the scaling parameter updated during the network back propagation process, μ B is the mean of all training data, σ B is the variance of all training data, ε is a constant, and x is the input. the fusion formula between the convolution layers is: wherein O is the fused feature, I is the input feature, K1, K2 and K3 are convolution parameters, and B1, B2 and B3 are offsets.

4. The segmentation method of a harbor area remote sensing image according to claim 1, characterized in that, the dilated convolution layer comprises a plurality of dilated convolutions with different sampling rates, which are used to extract multi-scale feature information of the first deep feature map and generate a second deep feature map.

5. The segmentation method of a port area remote sensing image according to claim 4, characterized in that, the Nth dilated convolution layer in the dense connection connects the second deep feature maps of N-1 dilated convolution layers, wherein the second deep feature map of the Nth dilated convolution layer is generated by combining the feature information in the second deep feature maps of N-1 dilated convolution layers, and all the second deep feature maps are fused to generate a third deep feature map through dense connection.

6. The segmentation method of a port area remote sensing image according to claim 1, characterized in that, the calculation formula of the preset loss function is: loss(x, class) = weight[class](x[class] + log∑ j exp(x[j])) wherein x is a predicted value, j is an iteration, loss is a loss, weight is a weight, and class is a pixel class for segmentation.

7. A harbor area remote sensing image contour segmenting device, characterized by, The method comprises the following steps: An image acquisition module is configured to acquire a port area remote sensing image. An image segmentation module is configured to input the port area remote sensing image into an image segmentation model to obtain a contour segmentation image of the port area remote sensing image output by the image segmentation model. The encoder of the image segmentation model comprises a feature extraction module and a scale feature fusion module. The feature extraction module extracts feature information of the port area remote sensing image based on a structure reparameterization network, wherein the feature information comprises a shallow feature map, a middle feature map and a first deep feature map; the structure reparameterization network is trained by using a multi-branch network and is inferred by using a single-branch network, and in the inference process, parameters of the multi-branch network and the single-branch network are fused; the scale feature fusion module performs feature extraction and feature fusion on the first deep feature map by using dense connection and a dilated convolution layer. The image segmentation module is further configured to acquire a sample port area remote sensing image; extract sample feature information of the sample port area remote sensing image, wherein the sample feature information comprises a sample shallow feature map, a sample middle feature map and a first sample deep feature map; perform feature extraction and feature fusion on the first sample deep feature map by using dense connection and a dilated convolution layer to obtain a second sample deep feature map; perform up-sampling on the second sample deep feature map to obtain a third sample deep feature map; perform skip connection on the third sample deep feature map, the sample shallow feature map and the sample middle feature map to obtain a sample feature map; perform convolution on the sample feature map to obtain a predicted contour segmentation image of the sample port area remote sensing image; and train a preset loss function by using the predicted contour segmentation image and a real contour segmentation image to obtain the image segmentation model.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the port area remote sensing image segmentation method according to any one of claims 1 to 6 when executing the program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the port area remote sensing image segmentation method according to any one of claims 1 to 6 when executed by the processor.

Images