A method for extracting information of floating raft culture of SAR image by semi-supervised cyclic consistency generative adversarial network

Abstract

The application discloses a semi-supervised cyclic consistency generative adversarial network SAR image floating raft culture information extraction method, first, the SAR image is pretreated, and the significant difference characteristics of the culture information are enhanced; secondly, a cyclic consistency generative adversarial network with a forward training network and a reverse training network is constructed, the label is reused, and a semi-supervised discriminator is added, the forward training network and the reverse training network are combined, the training is strengthened through generated pseudo labels, and the purpose that the output of the reverse training network is used to strengthen the forward training network is realized; finally, a bias convolution block is used to strengthen the segmentation effect of the fuzzy boundary of the culture area, and the precision of the culture information extraction is improved. The application can solve the problem that the label cannot be reused, the semi-supervised discriminator can solve the problem that the culture prediction picture precision is not high due to the confused label in the training process, and the SAR image floating raft culture information extraction work can be carried out by using the method.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of marine remote sensing and artificial intelligence, and relates to a method for extracting floating raft aquaculture information from SAR images using a semi-supervised cyclic consistency generative adversarial network. Background Technology

[0002] China possesses abundant marine resources, and in recent years, with rapid economic development and increasing demand for seafood, the scale of mariculture has expanded rapidly. Floating raft aquaculture is one of the important aquaculture methods in China's coastal waters, providing significant economic benefits. However, pollutants generated by mariculture have adversely affected the ecological environment of coastal areas. Therefore, it is of great significance to efficiently and accurately extract aquaculture information, and subsequently implement aquaculture investigation and rectification, strengthen monitoring and supervision, and create a sound and sustainable marine ecological environment.

[0003] In recent years, satellite remote sensing technology has been widely used in aquaculture information extraction. Synthetic Aperture Radar (SAR) can obtain high-resolution remote sensing images similar to optical photography under weather conditions with extremely low visibility. It has the advantages of effectively penetrating cover, being unaffected by adverse weather conditions, working in all weather conditions, and covering a wide area.

[0004] Floating raft aquaculture area, such as Figure 1 As shown, when using SAR imagery to extract aquaculture information, the surface roughness of the ocean varies randomly with factors such as weather, topography, and ocean currents. Crops within the aquaculture area are affected by these factors, leading to changes in backscattering characteristics. Therefore, traditional machine learning methods cannot guarantee accurate extraction of aquaculture information. In recent years, deep learning has been widely used in aquaculture information extraction. However, most deep learning-based aquaculture information extraction methods are fully supervised learning methods, requiring a large number of labels. This undoubtedly increases the human and material resources required for extraction, and the labels are time-sensitive and cannot be reused.

[0005] In view of the above problems, the semi-supervised generative adversarial network for aquaculture information extraction has not been studied yet. Therefore, a semi-supervised cycle-consistent generative adversarial network (SCycleGAN) for SAR image floating raft aquaculture information extraction is designed. The semi-supervised cycle-consistent generative adversarial network can enhance the spatial continuity of pixels in prediction, introduce the interference generated by the spatio-temporal transfer of elements into the generative adversarial network, and reduce the differences in SAR aquaculture data caused by spatio-temporal transfer through the mutual antagonistic iterative learning of the generator and the discriminator. Moreover, the cycle-consistent generative adversarial network proposes a semi-supervised method to address the problem of large workload and high cost of label making for full-supervised semantic segmentation, uses labels that do not need to be strictly paired with images and can be reused, realizes the aquaculture extraction task, and improves the segmentation accuracy while effectively reducing the labor cost. SUMMARY

[0006] The present application mainly solves the problems of differences in SAR aquaculture data caused by spatio-temporal transfer and large workload and high cost of label making in floating raft aquaculture information extraction. In traditional full-supervised learning, labels need to be one-to-one corresponding to images. If the pairing information of labels and images is missing, the machine will not be able to learn in the correct direction. Therefore, a semi-supervised cycle-consistent generative adversarial network for SAR image floating raft aquaculture information extraction is proposed, which introduces cycle-consistent loss, uses two groups of networks for forward training network and reverse training network to train and iterate remote sensing images and labels respectively, and labels can learn in the correct direction without pairing with images, realizing the reuse of label data and greatly reducing manpower and material resources. On this basis, a semi-supervised discriminator is added, the label reconstruction image is used as a pseudo label to strengthen the training, the extraction accuracy is further improved, the problem of missing small target aquaculture area is improved, and finally the generator network structure is improved, a bias convolution block is added to strengthen the segmentation effect of the fuzzy boundary of the aquaculture area, realizing high-precision semi-supervised SAR image floating raft aquaculture information extraction.

[0007] To achieve the above purpose, the technical scheme adopted by the present application is:

[0008] The application discloses a semi-supervised cyclic consistency generative adversarial network SAR image floating raft culture information extraction method, which comprises the following steps:

[0009] Firstly, the SAR image is preprocessed to enhance the significant difference characteristics of the culture information.

[0010] 1.1) radiation calibration, eliminating the radiation deviation of the SAR image caused by sensor error.

[0011] 1.2) coherent spot filtering, using a refined Lee filter to reduce the coherent spot noise in the SAR image.

[0012] 1.3) terrain correction, eliminating the perspective shrinkage, overlap, shadow and other phenomena caused by the terrain undulation in the SAR image.

[0013] 1.4) data set making, selecting the SAR image with good imaging effect in the floating raft culture area as the original data, cutting the image into 1024*1024 size, 1000 as the training set, 100 as the test set, and finally using the labelme software to make a label for each image through the artificial visual interpretation method.

[0014] Secondly, a cyclic consistency generative adversarial network model is built.

[0015] The cyclic consistency generative adversarial network is an improved generative adversarial network (GAN) model, which is used for learning image to image conversion without paired information. The model can extract the semantic features of the remote sensing image without paired information, and convert the data set into another type of data set while retaining these semantic features, that is, learning the conversion between the source domain and the target domain without paired information, such as Figure 2As shown: (a), (c) are the pre-processed remote sensing images, (b), (d) are labels, the paired training data is composed of (a), (b), and the corresponding accurate location of the culture area in the label can be clearly found through the pairing information, and the unpaired training data is composed of (c) and (d), and there is no pairing information between them, so the accurate corresponding position of the culture area in the label cannot be found. Therefore, when the cycle consistency generative adversarial network is used for floating raft culture information extraction, the same label can be used to train different SAR remote sensing images, and the label reuse is realized.

[0016] The pre-processed floating raft culture SAR image is input into the cycle consistency generative adversarial network for semantic segmentation task. The cycle consistency generative adversarial network includes two groups of networks of forward training network and reverse training network, and the cycle consistency generative adversarial network is as shown in the figure. Figure 3 The model uses two generators G and F, three discriminators D Y , D X and D semi . Specifically as follows:

[0017] The forward training network and the reverse training network share the two generators G and F, but do not share the same weights in the training process, and the input of the forward training network first enters the generator G, and the input of the reverse training network first enters the generator F. The framework of the forward training network and the reverse training network is as shown in the figure. Figure 4 The generator G converts the input image from the X domain to the Y domain, and the generator F converts the input image from the Y domain to the X domain. The purpose of the discriminator D Y is to distinguish the remote sensing image x and the generated image , and the purpose of the discriminator D X is to distinguish the label image y and the generated image . In order to reduce the difference of semantic information, the cycle consistency loss is introduced, so that the generator F in the forward training network generates closer to the remote sensing image x, and the generator G in the reverse training network generates closer to the label image y.

[0018] The generator G has the same structure as F. The first layer is an image enhancement layer, used to symmetrically enhance the image along the edges in all directions, increasing the image resolution. Next, three convolutional blocks are connected for downsampling; each convolutional block consists of a convolutional layer, an IN regularization layer, and a Leaky ReLU activation function. Then, a residual block consisting of nine residual modules is connected for data restoration and enhancement. Next, two deconvolutional blocks are connected for upsampling to restore the image size; each deconvolutional block consists of a deconvolutional layer, an IN regularization layer, and a ReLU activation function. Then, a bias convolutional block is connected to learn the features of specific pixels from four directions, transforming blurred boundary features into latent channel features. The bias convolutional block includes four bias convolutional layers, a ReLU layer, a Batch Normalization (BN) layer, and a convolutional layer. Finally, the image enhancement layer further improves the image resolution. Discriminator D Y D X With D semi The structure is also the same, D Y Used for positive training of the network, D X Used for reverse training of the network, D semi Between the forward and inverse training of the network, the discriminator's structure consists of five convolutional blocks. The second and fourth convolutional blocks each consist of a convolutional layer, an IN regularization layer, and a LeakyReLU activation function. The first convolutional block consists of a convolutional layer and a LeakyReLU activation function. The last convolutional block consists of only one convolutional layer.

[0019] 2.1) Improve the two generators G and F.

[0020] A biased convolutional block is added to the generator network. This biased convolutional block consists of four biased convolutional layers, one ReLU layer, one batch normalization (BN) layer, and one convolutional layer. The biased convolutional layer adds reflection padding to the input feature map in a specific direction. For a given m x n input matrix X, the reflection padding operation is as shown in equation (6):

[0021]

[0022] Among them, l,r,u, These represent the fill lengths in the left, right, top, and bottom directions of X, respectively. Furthermore, to overcome the influence of speckle noise on SAR images, dilated convolution is used in the bias convolutional layer. Dilated convolution expands the receptive field of the convolution kernel; the dilated convolution operator is defined as...

[0023]

[0024] in,* dcis the operator of dilated convolution, F is the image matrix, k is the convolution kernel, p is the superposition limit of position coordinates s and t, and γ is the expansion rate. Since γ and k are independent parameters, γ can be increased without changing k. Therefore, dilated convolution can expand the sampling range without increasing the parameters. Based on the above theory, the basic convolution block used in the bias convolution block can be written as

[0025] DCB(X, k, γ) ReLU(BN(Conv(X, k, γ))) (8)

[0026] where X is the input, ReLU is the ReLU activation function, BN is the batch norm function, and Conv is the convolution function. Therefore, the bias convolution can be defined as

[0027] OC(X, l, r, u, d, k, γ) = DCB((RP(X, l, r, u, d)), k, γ) (9)

[0028] where l, r, u, and d are bias control parameters, which are the padding lengths of the northeast, northwest, southwest, and southeast directions of X, respectively, as shown in Figure 5 By padding the northeast, northwest, southwest, and southeast directions of X respectively, four feature maps are obtained. Assuming that there is an element A that will not be consistent with the corresponding elements in the four new feature maps obtained after the bias convolution block. The offset convolution can learn the blurred boundary features of the specific elements in the A direction and convert them into channel features. Finally, the four feature maps are spliced by channel, and finally the features in all channels are extracted through a convolution layer to restore the number of input feature map channels.

[0029] By improving the generator network structure, the problem of poor extraction of fuzzy boundaries in the breeding area is improved, and the segmentation accuracy is improved.

[0030] 2.2) Build a forward training network model, which uses a generator G, a generator F and a discriminator D Y , as shown in the forward training network framework Figure 3 . First, the input of the generator G is the preprocessed and cropped remote sensing image, and the output is the segmentation map. Then the segmentation map is input into the generator F, and finally the generator F outputs the breeding reconstruction map. The input of the discriminator D Y is the segmentation map and the label, and the output of the discriminator D Y is a score between 0 and 1 for discriminating true and false. The discriminator D Y and the generator G are mutually optimized, and the generator G is guided to generate more accurate breeding prediction maps through backpropagation.

[0031] The objective function of the generator G and the discriminator D Y is shown in the following formula (10):

[0032]

[0033] Where x is the remote sensing image, y is the label, G generates the segmentation map G(x), and D... Y It will distinguish between G(x) and the label y. D Y The goal of generator D is to maximize the expectation of the objective function, while the goal of training G is to minimize the expectation of the objective function. These two learn against each other, enabling generator G to produce better segmentation maps, ultimately leading to the successful improvement of discriminator D. Y It is impossible to distinguish between the segmentation map and the label generated by generator G.

[0034] Then, the segmentation map generated by generator G is input into generator F. Generator F outputs the aquaculture reconstruction map. Cyclic consistency loss is used to reduce the semantic information difference between the generated aquaculture reconstruction map and the remote sensing image, and to maintain the consistency of semantic information. The cyclic consistency loss function is shown in the following equation (11):

[0035]

[0036] Where F(G(x)) is the aquaculture reconstruction map, and L1 loss is used to reduce the pixel-by-pixel distance with the remote sensing image x, that is, to minimize the cycle consistency loss function.

[0037] By building a forward-trained network model, D Y The generator G is iteratively optimized against each other, enabling it to generate better segmentation maps. Furthermore, the introduction of cycle consistency loss makes the aquaculture reconstruction map more realistic, reduces the loss of semantic information in the aquaculture reconstruction map, and maintains cycle consistency during the training process.

[0038] 2.3) Construct the reverse-trained network model. The reverse-trained network uses generator F, generator G, and discriminator D. X Although the inverse training network also uses generators F and G, the order in which they are used is different, and they do not share weights with the forward training network. The inverse training network is as follows: Figure 3 As shown. First, the labels are input into the generator F, which outputs the remote sensing prediction map F(y), and the discriminator D... X The input is F(y) and the remote sensing image, and the discriminator D Y The output is a score between 0 and 1 used to determine whether something is true or false. Generator F and discriminator D X The objective function is shown in equation (12) below:

[0039]

[0040] Similar to a forward-trained network, where F predicts the remote-sensed map F(y), and D... XThe remote sensing prediction map and the remote sensing image will be distinguished. D X The goal of training F is to minimize the expectation of the objective function, and the goal of training G is to maximize the expectation of the objective function. The two are optimized against each other, and ultimately make the discriminator D X cannot distinguish between real remote sensing images and remote sensing prediction maps. Then input the remote sensing prediction map into the generator G, output the label reconstruction map G(F(y)), minimize the cycle consistency loss to reduce the pixel distance with the label y, and the loss function is shown in the following formula (13):

[0041]

[0042] By building a reverse training network model, the semantic information loss between the label reconstruction map and the label is reduced by using the cycle consistency loss, and the semantic information consistency in the training process is maintained.

[0043] 2.4) Build a semi-supervised discriminator D semi , the label reconstruction map G(F(y)) generated by the reverse training network is used as a pseudo label and the segmentation map G(x) generated by the forward training network are input into the semi-supervised discriminator D semi , and the output is a score between 0 and 1 for distinguishing true from false, and the objective function is shown in the following formula (14):

[0044]

[0045] The generator G and the semi-supervised discriminator D semi upgrade each other, train D semi to maximize the expectation of the objective function, and train G to minimize the expectation of the objective function, ultimately making D semi cannot distinguish between the label reconstruction map as a pseudo label and the segmentation map, and the pseudo label is used to strengthen the training, thereby guiding G to generate more realistic images.

[0046] 2.5) The total objective function of the network is shown in formula (15):

[0047] L(G,D)=L(G,D Y )+L(F,D X )+λ semi L(G,F,D semi )+λ cyc L cyc (G,F) (15)

[0048] Where λ semi is a semi-supervised loss control parameter, λ cyc is a cycle consistency loss control parameter, and a semi-supervised discriminator D semiThe forward training network is combined with the reverse training network, and the accuracy of the model for generating the segmentation graph is further improved. The generator and the discriminator are alternately optimized and trained until the model converges, and when the balance state is reached, only the generator is retained, the discriminator is removed, and the generator is used to generate the segmentation graph.

[0049] The present application has the following advantages:

[0050] The present application uses the cycle consistency generative adversarial network to realize the floating raft culture segmentation task, can solve the problem that the label cannot be reused, saves a lot of manpower and material resources, and uses the built semi-supervised discriminator to solve the problem that the culture prediction graph accuracy is not high due to the confusion label in the training process. The method proposed in the present application achieves high accuracy in the floating raft culture semantic segmentation task, and can be used for floating raft culture information extraction work on SAR images. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 It is a schematic diagram of a floating raft culture area;

[0052] Figure 2 It is a schematic diagram of paired training data and unpaired training data, (a) and (c) are preprocessed remote sensing images, (b) and (d) are labels, the paired training data is composed of (a) and (b), and the unpaired training data is composed of (c) and (d);

[0053] Figure 3 It is a whole block diagram of the SAR image floating raft culture information extraction method of the semi-supervised cycle consistency generative adversarial network;

[0054] Figure 4 It is a schematic diagram of the forward training network and the reverse training network framework;

[0055] Figure 5 It is a bias convolution block construction diagram;

[0056] Figure 6 It is training process data, (a) is a preprocessed culture remote sensing image, (b) is an unpaired label used during training of the remote sensing image, and (c) is a training result graph;

[0057] Figure 7 It is a GF-3 satellite floating raft culture extraction result comparison diagram, (a) is a preprocessed culture remote sensing image, (b) is a culture area segmentation label, (c) is an extraction result without adding a semi-supervised discriminator, and (d) is an extraction result after adding a semi-supervised discriminator.

[0058] Figure 8For the patent method GF-3 satellite floating raft culture extraction result details, (a), (d), (g) are the pre-processed culture remote sensing images, (b), (e), (h) are the results of the method of the patent, (c), (f), (i) are the culture area segmentation labels, the average accuracy maccuracy=0.92, the average intersection over union mIOU=0.76, the average kappa coefficient mkappa=0.71, the average precision mprecision=0.93, and the average recall mrecall=0.96. DETAILED DESCRIPTION

[0059] In order to make the method problems solved by the present application, the method schemes adopted and the method effects achieved more clear, the present application will be further described in detail below with reference to the drawings and examples. It can be understood that the specific examples described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that only the parts related to the present application are shown in the drawings for the convenience of description, but not all the contents.

[0060] As Figure 1 shown, the SAR image floating raft culture information extraction method of the semi-supervised cyclic consistency generative adversarial network provided by the embodiment of the present application comprises:

[0061] The compilation is carried out under the windows 10 system of python3.8, pytorch1.7.1 and cuda10.1, the GPU of RTX3080TI is used for running, and the input data size is 1024x1024 SAR image.

[0062] Firstly, SAR remote sensing image data is acquired, and the remote sensing image is pre-processed. The SAR data processing operation adopts radiation calibration, coherent spot filtering and terrain correction, and finally a data set and a label are made.

[0063] 1.1) Radiation calibration, eliminating the radiation deviation of SAR image caused by sensor error.

[0064] 1.2) Coherent spot filtering, using a Refined Lee filter to reduce the coherent spot noise in the SAR image.

[0065] 1.3) Terrain correction, eliminating the phenomena such as perspective shrinkage, overlap, shadow and the like in the SAR image caused by terrain undulation.

[0066] 1.4) Making a data set, selecting SAR images with good imaging effect in the floating raft culture area as original data, cutting them into 1024x1024 size images, 1000 as a training set, 100 as a test set, and finally using the labelme software to make a label for each image by manual visual interpretation method.

[0067] Secondly, a cycle-consistent generative adversarial network model is built.

[0068] The cycle-consistent generative adversarial network is an improved generative adversarial network (GAN) model for learning image-to-image conversion without paired information. The model can extract semantic features of remote sensing images without paired information and convert the data set into another type of data set while preserving these semantic features, i.e. learning the conversion between the source domain and the target domain without paired information, such as Figure 2 As shown in the figure. (a) and (c) are preprocessed remote sensing images, (b) and (d) are labels, and the paired training data is composed of (a) and (b), and the corresponding accurate location of the culture area in the label can be clearly found through the paired information. The unpaired training data is composed of (c) and (d), and there is no paired information between them, so the accurate corresponding position of the culture area in the label cannot be found. Therefore, when the cycle-consistent generative adversarial network is used for floating raft culture information extraction, the same label can be used to train different SAR remote sensing images, realizing the reuse of labels.

[0069] The preprocessed floating raft culture SAR image is input into the cycle-consistent generative adversarial network for semantic segmentation task. The cycle-consistent generative adversarial network includes two groups of networks, forward training network and reverse training network, as shown in the figure. Figure 3 The model uses two generators G and F, three discriminators D Y , D X and D semi .

[0070] Although the forward training network and the reverse training network share the two generators G and F, they do not share the same weights in the training process, and the input of the forward training network first enters the generator G, and the input of the reverse training network first enters the generator F. The framework of the forward training network and the reverse training network is as shown in the figure. Figure 4 The generator G converts the input image from the X domain to the Y domain, and the generator F converts the input image from the Y domain to the X domain. The purpose of the discriminator D Y is to distinguish the remote sensing image x and the generated image , and the purpose of the discriminator D X is to distinguish the label image y and the generated image . In order to reduce the difference in semantic information, cycle consistency loss is introduced to make the generated by the generator F in the forward training network closer to the remote sensing image x, and the generated by the generator G in the reverse training network closer to the label image y.

[0071] The generator G has the same structure as F, the first layer is an image enhancement layer, which is used to symmetrize the image up and down and left and right along the edge, and increase the resolution of the image; then connect three convolutional blocks for down-sampling, each convolutional block is composed of a convolutional layer, an IN regularization layer and a LeakyReLU activation function; then connect a residual block composed of nine residual modules, which is used to recover and enhance the data; then connect two deconvolutional blocks for up-sampling, which is used to restore the image size, each deconvolutional block is composed of a deconvolutional layer, an IN regularization layer and a ReLU activation function; then connect a bias convolutional block, which is used to learn the features of specific pixels from four directions, and convert the blurred boundary features into potential channel features, the bias convolutional block includes four bias convolutional layers, a ReLU layer, a BN batch normalization layer and a convolutional layer; finally, the image resolution is improved through the image enhancement layer. The discriminator D Y , D X has the same structure as D semi , D Y is used for forward training network, D X is used for reverse training network, D semi is used for forward and reverse training network, the structure of the discriminator is composed of five convolutional blocks, among them, the second and fourth convolutional blocks are composed of a convolutional layer, an IN regularization layer and a LeakyReLU activation function, the first convolutional block is composed of a convolutional layer and a LeakyReLU activation function, and the last convolutional block is only composed of a convolutional layer.

[0072] 2.1) Improve the generator, add a bias convolutional block in the generator network, the bias convolutional block is composed of four bias convolutional layers, a ReLU layer, a BN batch normalization layer and a convolutional layer, the bias convolutional layer adds reflection padding to the input feature map in a specific direction. For a given m row n column input matrix X, the reflection padding operation is as shown in formula (6):

[0073]

[0074] Wherein, l, r, u, are the padding lengths of the left, right, top and bottom directions of X respectively. In addition, in order to overcome the influence of speckle noise on SAR image, dilated convolution is used in the bias convolutional layer, which expands the receptive field of the convolution kernel, and the dilated convolution operator is defined as

[0075]

[0076] Wherein, * dcis the operator of dilated convolution, F is the image matrix, k is the convolution kernel, p is the superposition limit of position coordinates s and t, and γ is the expansion rate. Since γ and k are independent parameters, γ can be increased without changing k. Therefore, dilated convolution can expand the sampling range without increasing the parameters. Based on the above theory, the basic convolution block used in the bias convolution block can be written as

[0077] DCB(X, k, γ) ReLU(BN(Conv(X, k, γ))) (8)

[0078] where X is the input, ReLU is the ReLU activation function, BN is the batch norm function, and Conv is the convolution function. Therefore, the bias convolution can be defined as

[0079] OC(X, l, r, u, d, k, γ) = DCB((RP(X, l, r, u, d)), k, γ) (9)

[0080] where l, r, u, and d are bias control parameters, which are the padding lengths of the northeast, northwest, southwest, and southeast directions of X, respectively, as shown in Figure 5 By padding the northeast, northwest, southwest, and southeast directions of X respectively, four feature maps are obtained. Assuming that there is an element A, it will not be consistent with the corresponding elements in the four new feature maps obtained after the bias convolution block. The offset convolution can learn the blurred boundary features of the specific elements in the A direction and convert them into channel features. Finally, the four feature map channels are spliced, and finally the features in all channels are extracted through a convolution layer to restore the number of input feature map channels.

[0081] By improving the generator network structure, the problem of poor extraction of fuzzy boundaries in the breeding area is improved, and the segmentation accuracy is improved.

[0082] 2.2) Build a forward training network model, which uses a generator G, a generator F and a discriminator D Y , as shown in the forward training network framework Figure 3 . First, the input of the generator G is the preprocessed and cropped remote sensing image, and the output is the segmentation map. Then the segmentation map is input into the generator F, and finally the generator F outputs the breeding reconstruction map. The input of the discriminator D Y is the segmentation map and the label, and the output of the discriminator D Y is a score between 0 and 1 for discriminating true and false. The discriminator D Y and the generator G are mutually optimized, and the generator G is guided to generate more accurate breeding prediction maps through backpropagation.

[0083] The objective function of the generator and the discriminator D Y is shown in the following formula (10):

[0084]

[0085] wherein x is a remote sensing image, y is a label, G generates a segmentation map G(x), and D Y discriminates G(x) from the label y. D Y The objective of G is to maximize the expectation of the objective function, and the objective of training G is to minimize the expectation of the objective function. Both of them learn each other to make the generator G generate better segmentation maps, and finally make the discriminator D Y unable to distinguish the segmentation maps generated by the generator G from the labels.

[0086] Then the segmentation map generated by the generator G is input into the generator F, and the output of the generator F is a farming reconstruction map. The semantic information difference between the generated farming reconstruction map and the remote sensing image is reduced using the cycle consistency loss, and the consistency of the semantic information is maintained. The cycle consistency loss function is shown in the following formula (11):

[0087]

[0088] wherein F(G(x)) is a farming reconstruction map, and the pixel-by-pixel distance from the remote sensing image x is reduced using L1loss, i.e. the cycle consistency loss function is minimized, so that the farming reconstruction map is more realistic, the semantic information loss of the farming reconstruction map is reduced, and the cycle consistency in the training process is maintained.

[0089] By building a forward training network model, D Y and G are iteratively optimized against each other, so that the generator G can generate better segmentation maps, and the cycle consistency loss is introduced to make the farming reconstruction map more realistic, reduce the semantic information loss of the farming reconstruction map, and maintain the cycle consistency in the training process.

[0090] 2.3) Build a reverse training network model. The reverse training network uses the generator F, the generator G and the discriminator D X Although the reverse training network also uses the generator F and the generator G, the order of use is different, and the weights are not shared with the forward training network. The reverse training network is shown in the following formula (12): Figure 3 First, the label is input into the generator F, and the output is a remote sensing prediction map F(y). The input of the discriminator D X is F(y) and the remote sensing image, and the output of the discriminator D Y is a score between 0 and 1 for discriminating true or false. The objective function of the generator F and the discriminator D X is shown in the following formula (12):

[0091]

[0092] Similar to the forward training network, where F will predict the remote sensing prediction map F(y), while D X will distinguish the remote sensing prediction map from the remote sensing image. X The goal of training D X is to maximize the objective function expectation, while the goal of training F is to minimize the objective function expectation. They optimize each other, and finally make D semi unable to distinguish between real remote sensing images and remote sensing prediction maps. Then input the remote sensing prediction map into the generator G, output the label reconstruction map G(F(y)), minimize the cycle consistency loss to reduce the pixel distance with the label y, and the loss function is shown in the following formula (13):

[0093]

[0094] By building a reverse training network model, the semantic information loss between the label reconstruction map and the label is reduced by using the cycle consistency loss, and the semantic information consistency in the training process is maintained.

[0095] 2.4) Build a semi-supervised discriminator D semi , and input the label reconstruction map G(F(y)) generated by the reverse training network as a pseudo label together with the segmentation map G(x) generated by the forward training network into the semi-supervised discriminator D semi , and output a score between 0 and 1 for distinguishing true from false, and the objective function is shown in the following formula (14):

[0096]

[0097] The generator G and the semi-supervised discriminator D semi upgrade each other, train D semi to maximize the objective function expectation, and train G to minimize the objective function expectation, so that D semi cannot distinguish between the label reconstruction map as a pseudo label and the segmentation map, and the pseudo label is used to strengthen the training, thereby guiding G to generate more realistic images.

[0098] 2.5) The total objective function of the network is shown in formula (15):

[0099] L(G,D)=L(G,D Y )+L(F,D X )+λ semi L(G,F,D semi )+λ cyc L cyc (G,F) (15)

[0100] Where λ semi is the control semi-supervised loss parameter, λ cyc is the control cycle consistency loss parameter, and λ semiSet to 3, l cyc Set to 10, add semi-supervised discriminator D semi The forward training network is combined with the reverse training network to further improve the accuracy of the model in generating the segmentation graph. The generator and the discriminator are alternately optimized and trained until the model converges. When reaching the balance state, only the generator is retained, and the discriminator is removed. The generator is used to generate the segmentation graph. The results of the method of the present patent are shown in Figure 8 The average accuracy maccuracy=0.92, the average intersection over union mIOU=0.76, the average kappa coefficient mkappa=0.71, the average precision mprecision=0.93, and the average recall mrecall=0.96. The method has good effects on small target extraction and extraction of fuzzy boundaries in the breeding area.

[0101] Finally, it should be noted that: the above embodiments are only used to illustrate the method scheme of the present application, but not to limit it; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: the modification of the method scheme recorded in the foregoing embodiments, or the equivalent replacement of part or all of the method features, does not make the essence of the corresponding method scheme deviate from the scope of the method scheme of the embodiments of the present application.

Claims

1. A method for extracting information from SAR imagery for floating raft aquaculture using a semi-supervised, cyclic consistency generative adversarial network, characterized in that, The method for extracting information from floating raft aquaculture using SAR images firstly preprocesses the SAR images to enhance the significant differences in aquaculture information. Secondly, it constructs a cyclic consistency generative adversarial network with both forward and inverse training networks to achieve label reuse and adds a semi-supervised discriminator. This combines the forward and inverse training networks, generating pseudo-labels to strengthen training, improve the accuracy of aquaculture information extraction, and mitigate the problem of missed detections in small-target aquaculture areas, achieving the goal of using the output of the inverse training network to strengthen the forward training network. Finally, it uses biased convolutional blocks to enhance the segmentation effect of blurred boundaries in aquaculture areas, further improving the accuracy of aquaculture information extraction. The method includes the following steps: The first step is to acquire SAR remote sensing image data and preprocess the remote sensing images. The second step is to build a cycle-consistent generative adversarial network model. Cyclic Consistency Generative Adversarial Networks (CGRANs) are an improved Generative Adversarial Network (GAN) model used to learn image-to-image transformations without paired information. This model extracts semantic features from remote sensing images without paired information and transforms the dataset into another type of dataset while preserving these semantic features. When extracting information about floating raft aquaculture, the CGRAN uses the same labels to train different SAR remote sensing images, achieving label reuse. Preprocessed floating raft aquaculture SAR images are input into the CGRAN for semantic segmentation. The CGRAN consists of two networks: a forward training network and a backward training network. This model uses... and Two generators, , and Three discriminators; 2.1) Build a forward-training network model using a generator. Generator and discriminator First, the generator The input is a preprocessed and cropped remote sensing image, and the output is a segmentation map, which is then input into the generator. In the middle, the final generator Output the aquaculture reconstruction graph; discriminator The input is a segmentation image and labels, and the discriminator... The output is a score between 0 and 1 used to determine whether something is true or false; the discriminator With generator Mutual optimization, guiding the generator through backpropagation. Generate more accurate aquaculture prediction maps; Generator and discriminator The objective function is shown in equation (10) below: (10) ； in, For remote sensing images, For tags, Generate segmentation map ,and Will to With tags To distinguish; The goal is to maximize the expectation of the objective function, while training The goal is to minimize the expectation of the objective function; the two learn against each other, making the generator... It was able to generate better segmentation maps, ultimately enabling the discriminator to achieve better results. Unable to distinguish generators The generated segmentation map and labels; Then the generator The generated segmentation map is input into the generator. In the middle, generator The output is a reconstructed aquaculture map. Cyclic consistency loss is used to reduce the semantic information difference between the generated reconstructed aquaculture map and the remote sensing image, and to maintain the consistency of semantic information. The cyclic consistency loss function is shown in equation (11) below: (11) ； in, Reconstructing diagrams for aquaculture; By building a forward-trained network model, and The algorithm employs iterative optimization with mutual resistance and introduces a cycle consistency loss to reduce the loss of semantic information in the aquaculture reconstruction graph and maintain cycle consistency during training. 2.2) Build a reverse-training network model. The reverse-training network uses a generator. Generator and discriminator Although the reverse training of the network also uses a generator With generator However, the order in which they are used is different, and they do not share weights with the forward-trained network; first, the labels are input into the generator. In the middle, output remote sensing prediction map Discriminator The input is With remote sensing images, discriminator The output is a score between 0 and 1 used to determine whether it is true or false; the generator and discriminator The objective function is shown in equation (12) below: (12) ； Among them, through the generator Output remote sensing prediction map ,and It can distinguish between remote sensing prediction maps and remote sensing images; The goal is to maximize the expected value of the objective function, while training... The goal is to minimize the expectation of the objective function; the two work in opposition to each other, ultimately improving the discriminator. Unable to distinguish between real remote sensing images and remote sensing prediction maps; the remote sensing prediction map is then input into the generator. In the middle, output the label reconstruction graph Minimize cycle consistency loss and reduce label The pixel-wise distance is calculated, and the loss function is shown in equation (13) below: (13) ； By building a reverse-trained network model, the loss of semantic information between the label reconstruction graph and the label is reduced by using cycle consistency loss, thus maintaining the consistency of semantic information during the training process. The third step is to build a semi-supervised discriminator. Reconstruct the label graph generated by the inverse-trained network. As a pseudo-label Segmentation map generated by the forward-trained network The inputs are fed into the semi-supervised discriminator. In the equation, the output is a score between 0 and 1 used to distinguish between true and false, and the objective function is shown in equation (14) below: (14) ； Generator Semi-supervised discriminator The confrontation escalates, training Maximize the expected value of the objective function, training To minimize the expectation of the objective function, ultimately making Unable to distinguish between the label reconstruction map and the segmentation map used as pseudo-labels, pseudo-labels are used to enhance training, thereby guiding... Generate more realistic images; Fourth step, the overall objective function of the network is shown in formula (15): (15) ； in, To control the hyperparameter of the semi-supervised loss, To control the hyperparameter of cycle consistency loss, a semi-supervised discriminator is added. Combining forward and inverse training networks improves the accuracy of the model's generated segmentation maps; The generator and discriminator are continuously optimized and trained alternately until the model converges. Once an equilibrium is reached, only the generator is retained and the discriminator is removed. The generator is then used to generate the segmentation map.

2. The method for extracting information from SAR images of floating raft aquaculture using a semi-supervised cyclic consistency generative adversarial network according to claim 1, characterized in that, The first preprocessing step includes: 1.1) Radiometric calibration to eliminate radiometric bias in SAR images caused by sensor errors; 1.2) Speckle filtering: A Refined Lee filter is used to reduce speckle noise in SAR images; 1.3) Terrain correction to eliminate perspective shrinkage, overlay, shadows or other phenomena in SAR images caused by terrain undulations; 1.4) Create a dataset by selecting SAR images with good imaging effects of the floating raft aquaculture area as the raw data, dividing them into training and testing sets, and finally creating labels for each image through manual visual interpretation.

3. The method for extracting information from SAR images of floating raft aquaculture using a semi-supervised cyclic consistency generative adversarial network according to claim 1, characterized in that, The generator and The structure is the same. The first layer is an image enhancement layer, used to symmetrically enhance the image along the edges in all directions, increasing the image resolution. Next, three convolutional blocks are connected for downsampling, each consisting of a convolutional layer, an IN regularization layer, and a Leaky ReLU activation function. Then, a residual block consisting of nine residual modules is connected for data restoration and enhancement. Next, two deconvolutional blocks are connected for upsampling to restore the image size, each consisting of a deconvolutional layer, an IN regularization layer, and a ReLU activation function. Then, a bias convolutional block is connected to learn the features of specific pixels from four directions, transforming blurred boundary features into latent channel features. Finally, the image enhancement layer improves the image resolution. The discriminator described above… , and The structure is also the same. Used for positive training of networks, Used for reverse training of networks. Between the forward and inverse training of the network, the discriminator's structure consists of five convolutional blocks. The second and fourth convolutional blocks each consist of a convolutional layer, an IN regularization layer, and a LeakyReLU activation function. The first convolutional block consists of a convolutional layer and a LeakyReLU activation function. The last convolutional block consists of only one convolutional layer.

4. The method for extracting information from SAR images of floating raft aquaculture using a semi-supervised cyclic consistency generative adversarial network according to claim 2, characterized in that, Adding biased convolutional blocks to the generator networks G and F is a way to... and Two generators are improved; the biased convolutional block consists of four biased convolutional layers, one ReLU layer, one BN batch normalization layer, and one convolutional layer. The biased convolutional layer adds reflection padding to the input feature map in a specific direction; for a given m x n input matrix X, the reflection padding operation is as shown in formula (6): (6) ； in, These represent the fill lengths in the left, right, top, and bottom directions of X, respectively. Furthermore, to overcome the influence of speckle noise on SAR images, dilated convolution is used in the bias convolutional layer. Dilated convolution expands the receptive field of the convolution kernel, and the dilated convolution operator is defined as: (7) ； in, It is the operator for dilated convolution. It is an image matrix. It is a convolution kernel. Position coordinates and The superposition limit, It is the dilation rate; the basic convolutional block used in the biased convolutional block is written as: (8) ； in, Given the input, ReLU is the ReLU activation function, BN is the batch norm function, and Conv is the convolution function; therefore, biased convolution is defined as: (9) ； in, , , , The bias control parameters are as follows: The filling length in the four directions of northeast, northwest, southwest, and southeast is determined by respectively... The four directions of northeast, northwest, southwest, and southeast are filled to obtain four feature maps; Offset convolution concatenates the four feature maps obtained by channel, and finally extracts the features from all channels through a convolutional layer to restore the number of channels of the input feature map.

Images