Semi-supervised remote sensing image road extraction method and device based on generative adversarial network

Abstract

The present application relates to a semi-supervised remote sensing image road extraction method and device based on a generative adversarial network. The method involves obtaining a training set, constructing a semi-supervised generative adversarial network model, and inputting labeled images and unlabeled images into the generator for road feature extraction. The labeled images are spliced ​​with labeled road prediction images, and the unlabeled images are spliced ​​with pseudo-labeled road prediction images, and then input into a first discriminator to obtain a first discrimination result. The labeled images are spliced ​​with labeled entropy maps, and the unlabeled images are spliced ​​with pseudo-labeled entropy maps, and then input into a second discriminator to obtain a second discrimination result. The semi-supervised generative adversarial network model is trained according to a pre-constructed loss function, the first discrimination result, and the second discrimination result to obtain a trained semi-supervised generative adversarial network model. Road extraction from remote sensing images is performed using the trained semi-supervised generative adversarial network model. This method can greatly improve the road extraction effect.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a semi-supervised remote sensing image road extraction method and device based on a generative adversarial network. Background Art

[0002] Road extraction can be viewed as a semantic segmentation task. The goal of semantic segmentation is to assign semantic labels to image pixels. Road extraction is a semantic segmentation task that assigns road and non-road labels to all pixels.

[0003] Although road extraction has received a lot of attention, the task remains challenging. Due to the presence of shadows and roadside objects (such as trees, buildings, and cars), occluded road areas are easily assigned as non-road labels. The characteristics of other non-road areas (such as parking lots) are similar to road areas in remote sensing images, so these non-road areas are easily assigned road labels. In addition, the shape of roads is thin and long, which means that the number of road labels and non-road labels is unbalanced. In addition, convolutional neural networks mainly focus on local information. For roads, due to their thin and long characteristics, more attention needs to be paid to global information. Therefore, the current defects faced by road extraction are mainly: roadside obstacles and road-like areas interfere with the road extraction effect, and traditional semantic segmentation convolutional neural networks have difficulty in acquiring global information.

[0004] At the same time, existing semi-supervised semantic segmentation methods do not fully utilize unlabeled data. Pseudo-labels do not utilize pixel information with confidence levels below a threshold, and the overall confidence of road predictions in unlabeled images is often lower than that of labeled images. In general, confidence levels near the boundaries between different regions in the semantic segmentation results are often below the threshold. For road extraction tasks, due to the narrowness of roads, they are often close to the boundaries. Therefore, much of the road information in the unlabeled data is not utilized. Therefore, the main drawback of current semi-supervised semantic segmentation is that it does not fully utilize information in low-confidence areas. Summary of the Invention

[0005] Based on this, it is necessary to provide a semi-supervised remote sensing image road extraction method and device based on generative adversarial network to eliminate obstacle interference and improve road extraction effect in order to solve the above technical problems.

[0006] A semi-supervised remote sensing image road extraction method based on a generative adversarial network, the method comprising:

[0007] Obtaining a training set, wherein the training set includes labeled images and unlabeled images;

[0008] Constructing a semi-supervised generative adversarial network model; the semi-supervised generative adversarial network model includes a generator, a first discriminator, and a second discriminator;

[0009] Inputting the labeled image and the unlabeled image into a generator respectively to extract road features to obtain a road prediction image; the road prediction image includes a labeled road prediction image and a pseudo-label road prediction image;

[0010] splicing the labeled image with the labeled road prediction image, and splicing the unlabeled image with the pseudo-labeled road prediction image, and inputting the resulting spliced ​​images into the first discriminator for judgment to obtain a first discrimination result;

[0011] Converting the labeled road prediction image into a labeled entropy map, and converting the pseudo-label road prediction image into a pseudo-label entropy map; splicing the labeled image with the labeled entropy map, and splicing the unlabeled image with the pseudo-label entropy map, and then inputting the spliced ​​images into the second discriminator for judgment to obtain a second discrimination result;

[0012] Training the semi-supervised generative adversarial network model according to a pre-constructed loss function, the first discrimination result, and the second discrimination result to obtain a trained semi-supervised generative adversarial network model;

[0013] Road extraction from remote sensing images is performed using a trained semi-supervised generative adversarial network model.

[0014] In one embodiment, the generator includes: a convolutional layer, a residual module, an attention module, a transposed convolutional layer, and a dilated convolutional bridge;

[0015] The residual module processes high-dimensional information layer by layer;

[0016] The attention module includes a channel attention module and a spatial attention module. The channel attention module adopts a lightweight channel attention module, and the spatial attention module adopts a non-local spatial attention module.

[0017] The transposed convolution layer gradually restores the size of the image features to the input image size;

[0018] The dilated convolution bridge increases the receptive field through dilated convolution;

[0019] The generator structures use jump connections to obtain low-dimensional and high-dimensional information.

[0020] In one embodiment, the spatial attention module adopts a non-local spatial attention module, including: the spatial attention module includes a longitudinal module and a transverse module, and the non-local spatial attention module is obtained by fusing the longitudinal module and the transverse module.

[0021] In one embodiment, the pseudo-label road prediction image includes: assigning a pseudo-label to an unlabeled image to obtain a pseudo-label road prediction image;

[0022] The pseudo labels are:

[0023]

[0024] Among them, x u is an unlabeled image, f is a model, f(x u ) is the model’s prediction vector for the unlabeled data category, max(f(x u )) is the confidence level, θ is the confidence threshold, y u Pseudo-labeling of unlabeled data.

[0025] In one embodiment, the rotational consistency loss is:

[0026]

[0027] Among them, H and W are the height and width of the image respectively. is the road prediction image, i.e. G(x)), is the rotation road prediction image, that is, G(x r ), x r To rotate the image, is the binary cross entropy loss, which is defined as:

[0028]

[0029] where y p and y t are the predicted categories and true labels, respectively.

[0030] In one embodiment, the label loss is:

[0031]

[0032] Among them, the cross entropy loss Defined as:

[0033]

[0034] Dice loss Defined as:

[0035]

[0036] Where x is the predicted value, y is the label value, n is the number of pixels involved in the calculation. For labeled images, n is equal to H × W, and for unlabeled images, n is equal to the number of pseudo-label pixels. |·| represents the sum of the pixel values ​​of the calculated image, and x∩y is the intersection of x and y.

[0037] In one embodiment, the first loss is:

[0038]

[0039] Among them, the generator loss Defined as:

[0040]

[0041] First discriminator loss Defined as:

[0042]

[0043] Among them, the dataset Can be divided into labeled parts and the unlabeled part x u is an unlabeled image, x l is a labeled image, y l is x l Corresponding label, D1 is the first discriminator.

[0044] In one embodiment, the second loss is:

[0045]

[0046] Among them, the generator loss Defined as:

[0047]

[0048] Second discriminator loss Defined as:

[0049]

[0050] Among them, the dataset Divided into labeled sections and the unlabeled part x u is an unlabeled image, x l is a labeled image, y l is x l Corresponding label, e u is the unlabeled entropy graph, e l is a labeled entropy map, and D2 is the second discriminator.

[0051] A semi-supervised remote sensing image road extraction device based on a generative adversarial network, the device comprising:

[0052] A sample acquisition module is used to acquire a training set, wherein the training set includes labeled images and unlabeled images;

[0053] A model construction module, configured to construct a semi-supervised generative adversarial network model; the semi-supervised generative adversarial network model comprises a generator, a first discriminator, and a second discriminator;

[0054] a feature extraction module, configured to input the labeled image and the unlabeled image into a generator to extract road features, thereby obtaining a road prediction image; the road prediction image includes a labeled road prediction image and a pseudo-labeled road prediction image;

[0055] a first discriminator network discrimination module, configured to splice the labeled image with the labeled road prediction image, and splice the unlabeled image with the pseudo-labeled road prediction image, and then input the spliced ​​images into the first discriminator for judgment to obtain a first discrimination result;

[0056] a second discriminator network discrimination module, configured to convert the labeled road prediction image into a labeled entropy map, and convert the pseudo-label road prediction image into a pseudo-label entropy map; concatenate the labeled image with the labeled entropy map, and concatenate the unlabeled image with the pseudo-label entropy map, and then input the concatenated images into the second discriminator for judgment, to obtain a second discrimination result;

[0057] A model training module, configured to train the semi-supervised generative adversarial network model according to a pre-constructed loss function, the first discrimination result, and the second discrimination result to obtain a trained semi-supervised generative adversarial network model;

[0058] The road extraction module is used to extract roads from remote sensing images using a trained semi-supervised generative adversarial network model.

[0059] The semi-supervised remote sensing image road extraction method and device based on generative adversarial network obtains a training set, which includes labeled images and unlabeled images; constructs a semi-supervised generative adversarial network model; the semi-supervised generative adversarial network model includes a generator, a first discriminator and a second discriminator; the labeled image and the unlabeled image are respectively input into the generator for road feature extraction to obtain a road prediction image; the road prediction image includes a labeled road prediction image and a pseudo-labeled road prediction image; the labeled image and the labeled road prediction image are spliced, and the unlabeled image and the pseudo-labeled road prediction image are spliced ​​and then input into the generator. Enter the first discriminator for judgment to obtain a first judgment result; convert the labeled road prediction image into a labeled entropy map, and convert the pseudo-label road prediction image into a pseudo-label entropy map; splice the labeled image with the labeled entropy map, and splice the unlabeled image with the pseudo-label entropy map, and then input them into the second discriminator for judgment to obtain a second judgment result; train the semi-supervised generative adversarial network model according to the pre-constructed loss function, the first judgment result, and the second judgment result to obtain a trained semi-supervised generative adversarial network model; perform road extraction on remote sensing images using the trained semi-supervised generative adversarial network model. The generator designed by the present invention for the complex spatial structure of roads and the nearly linear characteristics of roads can better adapt to road characteristics when extracting road features, effectively eliminate obstacle interference, and obtain more accurate road features; and by additionally using a discriminator to judge the entropy map of the road prediction image, the entropy information of the image is used to improve the effect of semi-supervised learning; at the same time, the semi-supervised generative adversarial network model is trained in combination with a preset loss function, which greatly improves the road extraction effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 1. A flow chart of a semi-supervised road extraction method for remote sensing images based on a generative adversarial network in one embodiment;

[0061] Figure 2 This is a structural diagram of RENet in one embodiment;

[0062] Figure 3 1 is an overall framework diagram of a semi-supervised remote sensing image road extraction method based on a generative adversarial network in one embodiment;

[0063] Figure 4 is a structural diagram of an attention module in one embodiment;

[0064] Figure 5 A longitudinal module structure diagram in one embodiment;

[0065] Figure 6 This is a horizontal module structure diagram in one embodiment;

[0066] Figure 71 is a projection diagram of a road in the horizontal and vertical directions in one embodiment;

[0067] Figure 8 A comparison chart of the results of training the DeepGlobe Road Extraction dataset in one embodiment, including: (a) remote sensing image; (b) road label; (c) D-LinkNet; (d) SII-Net; (e) AdvNet; (f) s4GAN; (g) ST++; (h) RENet; (i) SemiRENet;

[0068] Figure 9 A comparison chart of the results of training the Massachusetts Roads dataset in one embodiment, including: (a) remote sensing image; (b) road label; (c) D-LinkNet; (d) SII-Net; (e) AdvNet; (f) s4GAN; (g) ST++; (h) RENet; (i) SemiRENet;

[0069] Figure 10 A comparison chart of the results of training the CHN6-CUG dataset in one embodiment, including: (a) remote sensing image; (b) road label; (c) D-LinkNet; (d) SII-Net; (e) AdvNet; (f) s4GAN; (g) ST++; (h) RENet; (i) SemiRENet. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0071] It is worth noting that the descriptions such as “first” and “second” involved in the present invention only represent naming methods, without distinguishing between size and order, and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features.

[0072] In one embodiment, Figure 1 As shown, a semi-supervised remote sensing image road extraction method based on a generative adversarial network is provided, comprising the following steps:

[0073] Step 102: Obtain a training set, which includes labeled images and unlabeled images.

[0074] It's worth noting that for semi-supervised training, the amount of unlabeled image data is typically much greater than the amount of labeled image data. The two are mutually exclusive. Semi-supervised learning training data consists of labeled data and their corresponding labels, as well as unlabeled data. This means that labeled and unlabeled images are distinct, and these two, along with the corresponding labels for labeled images, form the training set.

[0075] Step 104: construct a semi-supervised generative adversarial network model; the semi-supervised generative adversarial network model includes a generator, a first discriminator, and a second discriminator.

[0076] Specifically, the first discriminator and the second discriminator both use UNet; the generator uses a new road extraction network proposed in this invention, named RENet, such as Figure 2 As shown in the figure, it is the structure diagram of RENet, including convolutional layer, residual module, attention module, transposed convolutional layer and void convolution bridge.

[0077] It is worth noting that the semi-supervised generative adversarial network model proposed in this invention is named SemiRENet.

[0078] Step 106: Input the labeled image and the unlabeled image into the generator respectively to extract road features to obtain a road prediction image; the road prediction image includes a labeled road prediction image and a pseudo-label road prediction image.

[0079] Specifically, road feature extraction refers to assigning road labels and non-road labels to pixels in remote sensing images, that is, performing binary classification on each pixel in the unlabeled image. Figure 2 In the RENet structure shown in the figure, labeled and unlabeled images are input into the generator respectively, and high-dimensional features are obtained through the convolutional layer, attention module, and residual module in sequence. The high-dimensional features pass through the void convolution bridge and then through the transposed convolution layer and convolution layer to obtain the road prediction results. In order to more effectively utilize low-dimensional features, some low-dimensional features are added to the features of the transposed convolution through skip connections.

[0080] Step 108 : splicing the labeled image with the labeled road prediction image, and splicing the unlabeled image with the pseudo-labeled road prediction image, and then inputting them into a first discriminator for judgment to obtain a first judgment result.

[0081] It is worth noting that if Figure 3 As shown, the road prediction image The first discriminator distinguishes whether the input image is labeled or unlabeled, and outputs an H×W judgment image p1. The judgment image p1 and the road prediction image The size is the same, determine the pixel value (p1) of the image in row i and column j i,j, represents the pixel position corresponding to the road prediction image The probability of coming from a labeled image. For a labeled image, its image type label is an image whose H×W values ​​are all 1; for an unlabeled image, its image type label is an image whose H×W values ​​are all 0.

[0082] Step 110: Convert the labeled road prediction image into a labeled entropy map, and convert the pseudo-label road prediction image into a pseudo-label entropy map; splice the labeled image with the labeled entropy map, and splice the unlabeled image with the pseudo-label entropy map, and then input them into the second discriminator for judgment to obtain a second discrimination result.

[0083] Specifically, the road prediction image will be obtained through the generator (ie, G(x)) is converted into an entropy graph e, where e = -G(x)·log(G(x)).

[0084] like Figure 3 As shown in Figure 1, the entropy map e is concatenated with the original training set x and input into the second discriminator. Similar to the first discriminator’s discriminant process, the second discriminator distinguishes whether the input image is labeled or unlabeled, and outputs an H×W judgment image.

[0085] Step 112: Train the semi-supervised generative adversarial network model according to the pre-constructed loss function, the first discrimination result, and the second discrimination result to obtain a trained semi-supervised generative adversarial network model.

[0086] Step 114: perform road extraction from the remote sensing image using the trained semi-supervised generative adversarial network model.

[0087] The semi-supervised remote sensing image road extraction method based on generative adversarial network obtains a training set, which includes labeled images and unlabeled images; constructs a semi-supervised generative adversarial network model; the semi-supervised generative adversarial network model includes a generator, a first discriminator and a second discriminator; the labeled image and the unlabeled image are respectively input into the generator for road feature extraction to obtain a road prediction image; the road prediction image includes a labeled road prediction image and a pseudo-labeled road prediction image; the labeled image and the labeled road prediction image are spliced, and the unlabeled image and the pseudo-labeled road prediction image are spliced ​​and then input into the The first discriminator is used to make a judgment to obtain a first judgment result; the labeled road prediction image is converted into a labeled entropy map, and the pseudo-label road prediction image is converted into a pseudo-label entropy map; the labeled image and the labeled entropy map are spliced, and the unlabeled image and the pseudo-label entropy map are spliced ​​and input into the second discriminator for judgment to obtain a second judgment result; the semi-supervised generative adversarial network model is trained according to the pre-constructed loss function, the first judgment result, and the second judgment result to obtain a trained semi-supervised generative adversarial network model; the road extraction of the remote sensing image is performed using the trained semi-supervised generative adversarial network model. The generator designed by the present invention is based on the characteristics of complex road spatial structure and near linearity of roads. When extracting road features, it can better adapt to road characteristics, effectively eliminate obstacle interference, and obtain more accurate road features; and by additionally using a discriminator to judge the entropy map of the road prediction image, the entropy information of the image is used to improve the effect of semi-supervised learning; at the same time, the semi-supervised generative adversarial network model is trained in combination with a preset loss function, which greatly improves the road extraction effect.

[0088] In one embodiment, the generator includes: a convolution layer, a residual module, an attention module, a transposed convolution layer and a dilated convolution bridge; the residual module processes high-dimensional information layer by layer; the attention module includes a channel attention module and a spatial attention module, the channel attention module adopts a lightweight channel attention module, and the spatial attention module adopts a non-local spatial attention module; the transposed convolution layer gradually restores the size of the image features to the input image size; the dilated convolution bridge increases the receptive field through dilated convolution; and the various structures of the generator use jump connections to obtain low-dimensional and high-dimensional information.

[0089] It is worth noting that skip connections are used to connect to an interval layer to obtain low-dimensional or high-dimensional information, rather than connecting layer by layer. In this way, when facing a very deep network model, the computational cost will not be too high, so that the network can always maintain the best state.

[0090] The dilated convolution bridge is used to increase the receptive field through dilated convolution. This can maximize the range of the receptive field and output more information without causing the loss of feature map resolution and retaining spatial detail information as much as possible.

[0091] It is worth noting that the road-to-space distribution is much more complex than the feature distribution. Therefore, in order to rationally utilize computing and storage resources, the channel attention module adopts a lightweight channel attention module, and the spatial attention module adopts a non-local spatial attention module.

[0092] like Figure 4 As shown in the figure, the channel attention block sequentially obtains attention weights for each channel through pooling, convolution, and normalization. Multiplying the channel attention weights by each channel yields the features that passed the channel attention. The spatial attention module consists of a longitudinal module and a transverse module. After passing through the longitudinal and transverse modules, the features are summed and multiplied by the learnable parameter γ. Finally, the features are added to the features before passing through the spatial attention module to yield the features that passed through the spatial attention module. The attention module combines the advantages of the channel attention module and the spatial attention module to highlight important road features.

[0093] In one embodiment, the spatial attention module includes a longitudinal module and a transverse module, and the non-local spatial attention module is obtained by fusing the longitudinal module and the transverse module.

[0094] Specifically, such as Figure 5 、 6 As shown in the figure, there are longitudinal module structure diagram and transverse module structure diagram respectively. Such structure is adopted because:

[0095] 1. Traditional non-local spatial attention needs to assign weights to each pixel in the image, which often requires large computing resources and video memory resources, while roads are usually close to linear (such as Figure 7 This approach can save resources.

[0096] 2. Since roads are usually close to linear, the global attention obtained by fusing horizontal attention and vertical attention is close to linear, which better adapts to road characteristics and better resists interference from roadside obstacles and similar road areas.

[0097] The calculation process of the vertical module is the same as that of the horizontal module. They first obtain three feature vectors Q, K, and V through convolution, then multiply Q and K to obtain the intermediate feature vector. The intermediate feature vector is passed through the Softmax function and then multiplied by V. The vector after the dot product is then dimensionalized to obtain the weighted features.

[0098] In one embodiment, the pseudo-labeling road prediction image includes: assigning a pseudo-label to an unlabeled image to obtain a pseudo-labeled road prediction image;

[0099] The pseudo labels are:

[0100]

[0101] Among them, x u is an unlabeled image, f is a model, f(x u ) is the model’s prediction vector for the unlabeled data category, max(f(x u )) is the confidence level, θ is the confidence threshold, y u Pseudo-labeling of unlabeled data.

[0102] In one embodiment, the rotational consistency loss is:

[0103]

[0104] Among them, H and W are the height and width of the image respectively. is the road prediction image, i.e. G(x)), is the rotation road prediction image, that is, G(x r ), x r To rotate the image, is the binary cross entropy loss, which is defined as:

[0105]

[0106] where y p and y t are the predicted categories and true labels, respectively.

[0107] It is worth noting that if Figure 3 As shown, the remote sensing images in the training set x are rotated to obtain the rotated training set x r , the training set and the rotated training set are input into the generator to obtain the road prediction images (ie G(x)), rotated road prediction image (i.e. G(x r )). Rotational consistency requires that the road prediction image obtained by the rotated image input model is also rotated by the corresponding angle.

[0108] The purpose of rotating the remote sensing image is to increase the amount of data and obtain more information-rich variants based on existing training samples to train the generator, so that the trained semi-supervised generative adversarial network model can extract road images more accurately.

[0109] In one embodiment, the label loss is:

[0110]

[0111] Among them, the cross entropy loss Defined as:

[0112]

[0113] Dice loss Defined as:

[0114]

[0115] Where x is the predicted value, y is the label value, n is the number of pixels involved in the calculation. For labeled images, n is equal to H × W, and for unlabeled images, n is equal to the number of pseudo-label pixels. |·| represents the sum of the pixel values ​​of the calculated image, and x∩y is the intersection of x and y.

[0116] It is worth noting that the label loss is used to utilize the label information of labeled images and the pseudo label information of unlabeled images. Since the number of road labels is much less than the number of non-road labels, the label loss is composed of the cross entropy loss. and Dice loss Pseudo-labels are generated by assigning corresponding labels to the predicted images of the unlabeled images with confidence levels greater than a certain threshold. In this application, the threshold is 0.95, and the value range of each pixel in the road prediction image is 0-1 (i.e., the probability that the pixel is a road). For values ​​between 0 and 0.05, the label 0 is assigned, for values ​​between 0.95 and 1, the label 1 is assigned, and for values ​​between 0.05 and 0.95, there is no pseudo-label. Other thresholds can also be set according to actual conditions.

[0117] In one embodiment, the first loss is:

[0118]

[0119] Among them, the generator loss Defined as:

[0120]

[0121] First discriminator loss Defined as:

[0122]

[0123] Among them, the dataset Can be divided into labeled parts and the unlabeled part x u is an unlabeled image, x l is a labeled image, y l is x l Corresponding label, D1 is the first discriminator.

[0124] It is worth noting that the generator loss in the first loss Used to update the generator parameters, the first discriminator loss in the first loss Used to update the parameters of the first discriminator so that the unlabeled road prediction image is close to the labeled road prediction image.

[0125] In one embodiment, the second loss is:

[0126]

[0127] Among them, the generator loss Defined as:

[0128]

[0129] Second discriminator loss Defined as:

[0130]

[0131] Among them, the dataset Can be divided into labeled parts and the unlabeled part x u is an unlabeled image, x l is a labeled image, y l is x l Corresponding label, e u is the unlabeled entropy graph, e l is a labeled entropy map, and D2 is the second discriminator.

[0132] It is worth noting that the generator loss in the second loss Used to update the generator parameters, the second discriminator loss in the second loss Used to update the parameters of the second discriminator so that the entropy map of the unlabeled road prediction image is close to the entropy map of the labeled road prediction image

[0133] In one embodiment, three public remote sensing image road datasets are used for experimental comparison, namely: DeepGlobe Road Extraction dataset, Massachusetts Roads dataset, and CHN6-CUG dataset.

[0134] Table 1 Number of images in each dataset

[0135]

[0136] To demonstrate the effectiveness of the proposed method, several advanced road extraction methods (D-LinkNet, SII-Net) and semi-supervised semantic segmentation methods (AdvNet, S4GAN, ST++) are used to evaluate the performance of the proposed model, and quantitative and qualitative comparisons are performed.

[0137] In the comparative experiments, we used three image quality evaluation metrics to compare SemiRENet with other models. Tables 2, 3, and 4 list the quantitative results of using each method to train the DeepGlobe Road Extraction dataset, the Massachusetts Roads dataset, and the CHN6-CUG dataset, respectively. Figure 8 、 Figure 9 、 Figure 10 Qualitative results for each method on the DeepGlobe Road Extraction dataset, the Massachusetts Roads dataset, and the CHN6-CUG dataset are listed. The highest score is in bold, and the second-best is underlined. Overall, the proposed SemiRENet performs best, achieving the highest scores on the evaluation metrics and ranking last in terms of image continuity. SemiRENet's generator, RENet, is generally inferior. Therefore, these three experiments verify that:

[0138] 1. RENet adopts a structure and attention module that adapts to road characteristics. It can efficiently utilize high-dimensional and low-dimensional, local and global information, and is effective in resisting interference from roadside obstacles and similar road areas.

[0139] 2. SemiRENet makes full use of the entropy information of the image, which makes full use of the information of the low-confidence area, thereby improving the effect of semi-supervised learning.

[0140] Table 2 Experimental results of DeepGlobe Road Extraction dataset

[0141]

[0142] Table 3 Experimental results of Massachusetts Roads dataset

[0143]

[0144] Table 4 Experimental results of CHN6-CUG dataset

[0145]

[0146] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0147] In one embodiment, a semi-supervised remote sensing image road extraction device based on a generative adversarial network is provided, comprising:

[0148] The sample acquisition module is used to acquire a training set, where the training set includes labeled images and unlabeled images.

[0149] The model construction module is used to construct a semi-supervised generative adversarial network model; the semi-supervised generative adversarial network model includes a generator, a first discriminator and a second discriminator.

[0150] The feature extraction module is used to input the labeled image and the unlabeled image into the generator respectively to extract road features and obtain a road prediction image; the road prediction image includes a labeled road prediction image and a pseudo-label road prediction image.

[0151] The first discriminator network discrimination module is used to splice the labeled image with the labeled road prediction image, and splice the unlabeled image with the pseudo-label road prediction image, and then input them into the first discriminator for judgment to obtain a first discrimination result.

[0152] The second discriminator network discrimination module is used to convert the labeled road prediction image into a labeled entropy map, and convert the pseudo-label road prediction image into a pseudo-label entropy map; the labeled image and the labeled entropy map are spliced, and the unlabeled image and the pseudo-label entropy map are spliced, and then input into the second discriminator for judgment to obtain a second discrimination result.

[0153] The model training module is used to train the semi-supervised generative adversarial network model according to the pre-constructed loss function, the first discrimination result, and the second discrimination result to obtain a trained semi-supervised generative adversarial network model.

[0154] The road extraction module is used to extract roads from remote sensing images using a trained semi-supervised generative adversarial network model.

[0155] The specific limitations of the semi-supervised road extraction device for remote sensing images based on a generative adversarial network can be found in the limitations of the semi-supervised road extraction method for remote sensing images based on a generative adversarial network described above and will not be repeated here. Each module in the aforementioned semi-supervised road extraction device for remote sensing images based on a generative adversarial network can be implemented in whole or in part through software, hardware, or a combination thereof. Each of these modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in the computer device in software form, so that the processor can call and execute the corresponding operations of each of these modules.

[0156] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0157] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0158] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A semi-supervised road extraction method for remote sensing images based on generative adversarial networks, characterized in that: The method comprises: Obtaining a training set, wherein the training set includes labeled images and unlabeled images; Constructing a semi-supervised generative adversarial network model; the semi-supervised generative adversarial network model includes a generator, a first discriminator, and a second discriminator; Inputting the labeled image and the unlabeled image into a generator respectively to extract road features to obtain a road prediction image; the road prediction image includes a labeled road prediction image and a pseudo-label road prediction image; splicing the labeled image with the labeled road prediction image, and splicing the unlabeled image with the pseudo-labeled road prediction image, and inputting the resulting spliced ​​images into the first discriminator for judgment to obtain a first discrimination result; Converting the labeled road prediction image into a labeled entropy map, and converting the pseudo-label road prediction image into a pseudo-label entropy map; splicing the labeled image with the labeled entropy map, and splicing the unlabeled image with the pseudo-label entropy map, and then inputting the spliced ​​images into the second discriminator for judgment to obtain a second discrimination result; Training the semi-supervised generative adversarial network model according to a pre-constructed loss function, the first discrimination result, and the second discrimination result to obtain a trained semi-supervised generative adversarial network model; Road extraction from remote sensing images is performed using a trained semi-supervised generative adversarial network model; The pre-built loss function includes rotation consistency loss, label loss, first loss and second loss; wherein the first loss includes generator loss With the first discriminator loss , the second loss includes the generator loss With the second discriminator loss .

2. The method according to claim 1, characterized in that The generator includes: a convolutional layer, a residual module, an attention module, a transposed convolutional layer, and a dilated convolutional bridge; The residual module processes high-dimensional information layer by layer; The attention module includes a channel attention module and a spatial attention module. The channel attention module adopts a lightweight channel attention module, and the spatial attention module adopts a non-local spatial attention module. The transposed convolution layer gradually restores the size of the image features to the input image size; The dilated convolution bridge increases the receptive field through dilated convolution; The generator structures use jump connections to obtain low-dimensional and high-dimensional information.

3. The method according to claim 2, characterized in that The spatial attention module adopts a non-local spatial attention module, including: The spatial attention module includes a longitudinal module and a transverse module, and a non-local spatial attention module is obtained by fusing the longitudinal module and the transverse module.

4. The method according to claim 3, characterized in that The pseudo-label road prediction image comprises: assigning a pseudo-label to an unlabeled image to obtain a pseudo-label road prediction image; The pseudo labels are: in, is an unlabeled image, For the model, is the model’s prediction vector for the unlabeled data category, is the confidence level, is the confidence threshold, Pseudo-labeling of unlabeled data.

5. The method according to any one of claims 1 to 4, characterized in that The rotational consistency loss is: in, and are the height and width of the image, is the road prediction image, i.e. ), is the rotation road prediction image, i.e. , To rotate the image, is the binary cross entropy loss, which is defined as: in and are the predicted categories and true labels, respectively.

6. The method according to any one of claims 1 to 4, characterized in that The label loss is: Among them, the cross entropy loss Defined as: Dice loss Defined as: in, is the predicted value, is the label value, is the number of pixels involved in the calculation. For labeled images, equal , for unlabeled images, Equal to the number of pseudo-label pixels, Represents the sum of the pixel values ​​of the calculated image, yes and The intersection of .

7. The method according to any one of claims 1 to 4, characterized in that The first loss is: Among them, the generator loss Defined as: First discriminator loss Defined as: Among them, the dataset Divided into labeled sections and the unlabeled part , is an unlabeled image, For labeled images, for Corresponding label, is the first discriminator.

8. The method according to any one of claims 1 to 4, characterized in that The second loss is: Among them, the generator loss Defined as: Second discriminator loss Defined as: Among them, the dataset Divided into labeled sections and the unlabeled part , is an unlabeled image, For labeled images, for Corresponding label, is the unlabeled entropy graph, is a labeled entropy graph, is the second discriminator.

9. A semi-supervised remote sensing image road extraction device based on generative adversarial network, characterized in that: The device comprises: A sample acquisition module is used to acquire a training set, wherein the training set includes labeled images and unlabeled images; A model construction module is used to construct a semi-supervised generative adversarial network model; the semi-supervised generative adversarial network model includes a generator, a first discriminator and a second discriminator; a feature extraction module, configured to input the labeled image and the unlabeled image into a generator to extract road features, thereby obtaining a road prediction image; the road prediction image includes a labeled road prediction image and a pseudo-labeled road prediction image; a first discriminator network discrimination module, configured to splice the labeled image with the labeled road prediction image, and splice the unlabeled image with the pseudo-labeled road prediction image, and then input the spliced ​​images into the first discriminator for judgment to obtain a first discrimination result; a second discriminator network discrimination module, configured to convert the labeled road prediction image into a labeled entropy map, and convert the pseudo-label road prediction image into a pseudo-label entropy map; concatenate the labeled image with the labeled entropy map, and concatenate the unlabeled image with the pseudo-label entropy map, and then input the concatenated images into the second discriminator for judgment, to obtain a second discrimination result; A model training module, configured to train the semi-supervised generative adversarial network model according to a pre-constructed loss function, the first discrimination result, and the second discrimination result to obtain a trained semi-supervised generative adversarial network model; The road extraction module is used to extract roads from remote sensing images using a trained semi-supervised generative adversarial network model; In the model training module, the pre-built loss function includes rotation consistency loss, label loss, first loss and second loss; wherein the first loss includes generator loss With the first discriminator loss , the second loss includes the generator loss With the second discriminator loss .

Images