A method and apparatus for target extraction from remote sensing images based on deep learning

Abstract

This invention provides a method and apparatus for target extraction from remote sensing images based on deep learning, applicable to the field of remote sensing. The method includes: randomly cropping a remote sensing image to obtain multiple images to be classified; normalizing the multiple images to obtain multiple input images of the same size; inputting the input images into a deep learning model for classification to obtain a classification label for each input image; labeling the corresponding input image according to the classification label; restoring the size of the labeled input image to the size of the corresponding image to be classified, obtaining multiple classified images; and stitching the multiple classified images together according to the positions of the corresponding images to be classified to obtain a stitched remote sensing image. This invention has a simple implementation principle, simple operation method, good extraction effect on man-made structures from remote sensing images, and low implementation cost.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing technology, and particularly relates to a method and apparatus for target extraction from remote sensing images based on deep learning. Background Technology

[0002] In recent years, deep learning image processing has made tremendous progress in both theory and practice, and has gradually permeated all areas of human activity, yielding significant economic benefits. There are numerous successful examples of using deep learning to process remote sensing images; for instance, deep learning has been used to segment crops and farmland in remote sensing images.

[0003] The goal of deep learning-based remote sensing image extraction of man-made structures is to extract the regions of man-made structures from remote sensing images. Viewed from above, man-made structures exhibit diverse shapes, sizes, and scattered distribution. Compared to their surroundings, man-made structures, such as houses, appear relatively small. When the resolution of remote sensing images is insufficient, the number of pixels for a single man-made structure may not exceed a double digit. Unlike common image recognition methods, such as human identification, which often involve millions of pixels, the limited number of pixels not only increases the difficulty of semantic segmentation of man-made structures but also leads to the loss of features, making deep learning-based remote sensing image processing extremely challenging.

[0004] Deep learning image processing falls into two categories: image classification and semantic segmentation. Deep learning is suitable for processing objects with distinct features, such as human recognition. A person consists of a head, torso, hands, feet, etc., and the head is further composed of eyes, eyebrows, nose, mouth, etc., making the features extremely distinctive. During semantic segmentation, it can be easily distinguished from the surrounding environment. However, remote sensing images, which view the earth from high altitudes, often present challenges because man-made structures are difficult to distinguish from surrounding pixels using semantic segmentation. Consequently, deep learning may not be effective when processing man-made structures. Summary of the Invention

[0005] To address the problems existing in current technologies, this invention provides a method and apparatus for target extraction from remote sensing images based on deep learning. When deep learning semantic segmentation is difficult to implement, the problem is transformed into a classification problem. By classifying images of small regions within a remote sensing image and then stitching the classification results together, man-made building regions can be extracted from large-area remote sensing images.

[0006] In a first aspect, the present invention provides a method for target extraction from remote sensing images based on deep learning, comprising: randomly cropping a remote sensing image to obtain multiple images to be classified, each image to be classified having a different size, wherein the remote sensing image contains a target to be extracted; normalizing the multiple images to be classified to obtain multiple input images of the same size, inputting the input images into a deep learning model for classification to obtain a classification label for each input image; labeling the corresponding input image according to the classification label, restoring the size of the labeled input image to the size of the corresponding image to be classified, thereby obtaining multiple classified images, each classified image having the same label as the corresponding input image, and stitching the multiple classified images together according to the positions of the corresponding images to be classified to obtain a stitched remote sensing image, wherein the stitched remote sensing image displays the distribution of the target.

[0007] Furthermore, the deep learning model includes: convolutional layers, max pooling layers, upper convolutional layers, concatenation layers, fully connected layers, and activation function layers.

[0008] Furthermore, the network structure of the deep learning model is as follows:

[0009] Layer 0 is the input layer, which receives the normalized image.

[0010] The first layer is a convolutional layer with a kernel size of 3*3 and a number of 64 kernels.

[0011] The second layer is a convolutional layer with a kernel size of 3*3 and a number of 64 kernels.

[0012] The third layer contains three max pooling operations with pooling window sizes of 2, 3, and 6 respectively.

[0013] The 4th and 5th layers are both convolutional layers, with a kernel size of 3*3 and a kernel count of 64.

[0014] The 6th layer is an upper convolutional layer, containing three types of upper convolution operations with kernel sizes of 2*2, 3*3 and 6*6, and the number of upper convolution kernels is 32.

[0015] The 7th layer is the splicing layer;

[0016] The 8th layer is the max pooling layer, and the pooling window for the max pooling operation is 2.

[0017] The 9th layer is a convolutional layer with a kernel size of 3*3 and 128 kernels.

[0018] The 10th layer is the max pooling layer, and the pooling window for the max pooling operation is 2.

[0019] The 11th layer is a convolutional layer with a kernel size of 3*3 and a number of 256 kernels.

[0020] The 12th layer is the max pooling layer, and the pooling window for the max pooling operation is 2.

[0021] The 13th layer is a convolutional layer with a kernel size of 3*3 and a number of 512 kernels.

[0022] Layer 14 consists of a fully connected layer and a ReLU layer;

[0023] Layer 15 consists of a fully connected layer and a ReLU layer;

[0024] The 16th layer is the Softmax layer;

[0025] The 17th layer is the classification output layer.

[0026] Furthermore, the actual spatial distance corresponding to the width and / or height of the image to be classified is a random number between L1 and L2.

[0027] Furthermore, the classification label of each input image includes 0, 1, and 2, where 0 indicates that there is no target in the input image corresponding to the classification label, 1 indicates that there is a sparsely distributed target in the input image corresponding to the classification label, and 2 indicates that there is a densely distributed target in the input image corresponding to the classification label.

[0028] Further, the step of marking the corresponding input image according to the classification label and restoring the size of the marked input image to the size of the corresponding image to be classified, thereby obtaining multiple classified images, includes: setting each pixel in the corresponding input image to a different color according to the classification label, with each color corresponding to a classification label, to obtain a marked input image; restoring the size of the marked input image to the size of the corresponding image to be classified, thereby obtaining multiple classified images, wherein the color of each pixel in the classified images is the same as the color of each pixel in the corresponding input image.

[0029] Secondly, the present invention also provides a deep learning-based remote sensing image annotation method. The dataset generated by the annotation method is used to train the deep learning model in the aforementioned deep learning-based remote sensing image target extraction method. The annotation method includes: acquiring a dataset to be annotated, wherein the images to be annotated in the dataset are remote sensing images including targets; for each image to be annotated, determining a first average value w of the pixel distance between the bounding rectangles of each target region in the width direction of the image to be annotated. ave And the number of pixels w in the width direction of the image to be labeled, based on the first average value w aveThe first ratio α is determined by the ratio of the number of pixels w in the width direction of the image to be labeled; for each image to be labeled, the second average value h of the pixel distances between the bounding rectangles of each target region in the height direction of the image to be labeled is determined. ave And the number of pixels h in the height direction of the image to be labeled, based on the second average value h. ave The second ratio β is determined by the ratio of the first ratio α to the number of pixels h in the height direction of the image to be labeled; the distribution of targets in the image to be labeled is then determined based on the first ratio α and the second ratio β.

[0030] Thirdly, the present invention also provides a remote sensing image target extraction device based on deep learning, comprising: a first processing module, used to randomly crop a remote sensing image to obtain multiple images to be classified, each image to be classified having a different size, wherein the remote sensing image contains a target to be extracted; a second processing module, used to normalize the multiple images to be classified to obtain multiple input images of the same size, input the input images into a deep learning model for classification, and obtain a classification label for each input image; a third processing module, used to mark the corresponding input images according to the classification labels, restore the size of the marked input images to the size of the corresponding images to be classified, to obtain multiple classified images, each classified image having the same label as the corresponding input image, and stitching the multiple classified images according to the positions of the corresponding images to be classified to obtain a stitched remote sensing image, wherein the distribution of the target is displayed in the stitched remote sensing image.

[0031] Fourthly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the deep learning-based remote sensing image target extraction method described above.

[0032] Fifthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the deep learning-based remote sensing image target extraction method described above.

[0033] In a sixth aspect, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the deep learning-based remote sensing image target extraction method described above.

[0034] The remote sensing image target extraction method and apparatus based on deep learning provided by this invention have the following advantages:

[0035] 1. Randomly crop the original large-size remote sensing image, resulting in an image of random size. Because the distribution of man-made structures in remote sensing images is random, and the size of the distribution areas is also random, it is difficult to obtain a suitable cropping size. Random cropping avoids the problem of selecting the cropping size that occurs with regular cropping. 2. This invention classifies and labels images based on the density of man-made structures in smaller images, classifying the regions corresponding to the smaller images. This transforms the semantic segmentation problem into a classification problem, avoiding the difficulty of segmenting and labeling image pixels during semantic segmentation. The classification operation is simple, greatly reducing the difficulty of the problem.

[0036] 2. This invention transforms the deep learning semantic segmentation problem into a deep learning image classification problem, which is simple to train, has high classification accuracy, high efficiency, and low computational cost.

[0037] 3. Based on the classification results of each small image, this invention stitches the small images together to obtain a predicted image of the same size as the original remote sensing image. The predicted image can represent the density information of different man-made buildings. The idea is clear, the method is efficient, and the feasibility is strong. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0039] Figure 1 This is a flowchart illustrating some embodiments of the deep learning-based remote sensing image target extraction method provided by the present invention;

[0040] Figure 2 This is a schematic diagram of the network structure of a deep learning model;

[0041] Figure 3 This is a schematic diagram of an application scenario of the remote sensing image target extraction method based on deep learning provided by the present invention;

[0042] Figure 4 This is a flowchart illustrating some embodiments of the deep learning-based remote sensing image annotation method provided by the present invention;

[0043] Figure 5 This is a schematic diagram of one embodiment for calculating the density of man-made structures in remotely sensed images;

[0044] Figure 6 These are schematic diagrams illustrating the structure of some embodiments of the remote sensing image target extraction device based on deep learning provided by the present invention.

[0045] Figure 7 This is a schematic diagram of the structure of an electronic device provided according to the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0047] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0048] It should be noted that the concepts of "first" and "second" mentioned in this invention are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0049] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0050] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0051] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0052] Please see Figure 1 , Figure 1 These are schematic flowcharts illustrating some embodiments of the deep learning-based remote sensing image target extraction method provided by this invention. For example... Figure 1 As shown, the method includes the following steps:

[0053] Step 101: Randomly crop the remote sensing image to obtain multiple images to be classified. Each image to be classified has a different size, and the remote sensing image contains the target to be extracted.

[0054] As an example, targets that need to be extracted from remote sensing images can include man-made buildings, forests, grasslands, rivers, mountains, and crowds.

[0055] In some optional implementations, the actual spatial distance corresponding to the width and / or height of the image to be classified is a random number between L1 and L2. As an example, L1 and L2 can be flexibly set according to the distribution of man-made structures in the remote sensing image to be processed. To achieve high accuracy, L1 and L2 can be set to smaller values; conversely, to achieve lower accuracy, L1 and L2 can be set to larger values. Generally, L1 can be, but is not limited to, 100m, and L2 can be, but is not limited to, 400m. When the spatial length corresponding to the width of the remote sensing image to be processed is less than L2, the image width direction does not need to be cropped; when the spatial length corresponding to the height of the remote sensing image is less than L2, the height direction does not need to be cropped. Setting L1 and L2 can balance accuracy and efficiency to a certain extent. Setting L1 and L2 as needed is equivalent to improving the model's generalization ability, enabling more accurate classification in different scenarios.

[0056] Step 102: After normalizing multiple images to be classified, multiple input images of the same size are obtained. The input images are then input into a deep learning model for classification to obtain a classification label for each input image.

[0057] As an example, the size of the image after normalizing multiple images to be classified can be the size required for the input of a deep learning network.

[0058] In some alternative implementations, the deep learning model may include: convolutional layers, max pooling layers, upper convolutional layers, concatenation layers, fully connected layers, and activation function layers.

[0059] In some optional implementations, the network structure of the deep learning model is as follows:

[0060] Layer 0 is the input layer, which receives the normalized image.

[0061] The first layer is a convolutional layer with a kernel size of 3*3 and a number of 64 kernels.

[0062] The second layer is a convolutional layer with a kernel size of 3*3 and a number of 64 kernels.

[0063] The third layer contains three max pooling operations with pooling window sizes of 2, 3, and 6 respectively.

[0064] The 4th and 5th layers are both convolutional layers, with a kernel size of 3*3 and a kernel count of 64.

[0065] The 6th layer is an upper convolutional layer, containing three types of upper convolution operations with kernel sizes of 2*2, 3*3 and 6*6, and the number of upper convolution kernels is 32.

[0066] The 7th layer is the splicing layer;

[0067] The 8th layer is the max pooling layer, and the pooling window for the max pooling operation is 2.

[0068] The 9th layer is a convolutional layer with a kernel size of 3*3 and 128 kernels.

[0069] The 10th layer is the max pooling layer, and the pooling window for the max pooling operation is 2.

[0070] The 11th layer is a convolutional layer with a kernel size of 3*3 and a number of 256 kernels.

[0071] The 12th layer is the max pooling layer, and the pooling window for the max pooling operation is 2.

[0072] The 13th layer is a convolutional layer with a kernel size of 3*3 and a number of 512 kernels.

[0073] The 14th layer is a fully connected layer and a ReLU layer, with an output size of 1*1*1000;

[0074] The 15th layer is a fully connected layer and a ReLU layer, with an output size of 1*1*3;

[0075] The 16th layer is the Softmax layer;

[0076] The 17th layer is the classification output layer.

[0077] As an example, refer to Figure 2 The specific network parameters that a deep learning model can have are as follows:

[0078] (1) The 0th layer of the network structure is the input image, which has a size of 224*224*3.

[0079] (2) The first layer of the network structure is a convolutional layer with a kernel size of 3*3, a number of kernels of 64, and an output data size of 224*224*64.

[0080] (3) The second layer of the network structure is a convolutional layer with a kernel size of 3*3 and a number of kernels of 64. The output data size is 224*224*64. The output of the second layer network is input to the third layer in three ways.

[0081] (4) The third layer of the network structure contains three max pooling operations, with pooling windows of 2, 3 and 6 respectively.

[0082] (5) The fourth and fifth layers of the network structure are both convolutional layers, with a kernel size of 3*3 and a kernel number of 64. In addition, the output of the third layer network will be added to the output of the fifth layer network.

[0083] (6) The sixth layer of the network structure is an upper convolutional layer, which contains three types of upper convolution operations with kernel sizes of 2*2, 3*3 and 6*6 respectively, and the number of upper convolution kernels is 32.

[0084] (7) The 7th layer of the network structure is a splicing layer, which splices the output of the 1st layer and the output of the 6th layer to form 224*224*160 data.

[0085] (8) The 8th layer of the network structure is the max pooling layer, and the pooling window of the max pooling operation is 2.

[0086] (9) The 9th layer of the network structure is a convolutional layer with a kernel size of 3*3 and a number of kernels of 128, outputting 112*112*128 data.

[0087] (10) The 10th layer of the network structure is the max pooling layer, and the pooling window of the max pooling operation is 2.

[0088] (11) The 11th layer of the network structure is a convolutional layer with a kernel size of 3*3 and a kernel number of 256, outputting 56*56*256 data.

[0089] (12) The 12th layer of the network structure is the max pooling layer, and the pooling window of the max pooling operation is 2.

[0090] (13) The 13th layer of the network structure is a convolutional layer with a kernel size of 3*3 and a number of kernels of 512, outputting 28*28*512 data.

[0091] (14) The 14th layer of the network structure is a fully connected and ReLU layer with an output size of 1*1*1000;

[0092] (15) The 15th layer of the network structure is a fully connected and ReLU layer with an output size of 1*1*3;

[0093] (16) The 16th layer of the network structure is the Soft max layer;

[0094] (17) The 17th layer of the network structure is the classification output layer.

[0095] In some optional implementations, the classification label for each input image includes 0, 1, and 2, where 0 indicates that there is no target in the input image corresponding to the classification label, 1 indicates that there are sparsely distributed targets in the input image corresponding to the classification label, and 2 indicates that there are densely distributed targets in the input image corresponding to the classification label. As an example, the classification labels can be divided into three categories based on the density of man-made structures in the remote sensing image: no man-made structures, sparse man-made structures, and dense man-made structures; the category of "no man-made structures" is represented by the Arabic numeral "0", the category of "sparse man-made structures" is represented by the Arabic numeral "1", and the category of "dense man-made structures" is represented by the Arabic numeral "2".

[0096] Step 103: According to the classification label, mark the corresponding input image, restore the size of the marked input image to the size of the corresponding image to be classified, and obtain multiple classified images. Each classified image has the same label as the corresponding input image. Then, stitch the multiple classified images according to the position of the corresponding image to be classified to obtain a stitched remote sensing image. The stitched remote sensing image shows the distribution of the target.

[0097] In some optional implementations, the step of marking the corresponding input image according to the classification label and restoring the size of the marked input image to the size of the corresponding image to be classified, thereby obtaining multiple classified images, includes: setting each pixel in the corresponding input image to a different color according to the classification label, with each color corresponding to a classification label, to obtain a marked input image; restoring the size of the marked input image to the size of the corresponding image to be classified, thereby obtaining multiple classified images, wherein the color of each pixel in the classified images is the same as the color of each pixel in the corresponding input image. As an example, when the input image is predicted to output the number "0", it is assumed that the spatial region corresponding to the input image is "free of man-made buildings". A new image with the same size as the randomly cropped input image is created (that is, the size of the labeled input image is restored to the size of the corresponding image to be classified, resulting in multiple classified images). The new image (i.e., the classified image) is a single white color. When the image is predicted to output the number "1", it is assumed that the spatial region corresponding to the image is "sparsely populated with man-made buildings". A new image with the same size as the randomly cropped input image is created. The new image is a single gray color. When the image is predicted to output the number "2", it is assumed that the spatial region corresponding to the image is "densely populated with man-made buildings". A new image with the same size as the randomly cropped input image is created. The new image is a single red color. After all the images cropped from the original remote sensing image have been predicted, the predicted images are stitched together according to their positions in the original remote sensing image to form a new image with the same size as the original image. In the new image, red areas represent areas with dense man-made structures in the remote sensing image, gray areas represent areas with sparse man-made structures, and white areas represent areas without man-made structures. This transforms the image segmentation problem into an image classification problem.

[0098] As an example, see reference Figure 3 , Figure 3 This is a schematic diagram illustrating an application scenario of the deep learning-based remote sensing image target extraction method provided by the present invention. It can be seen that... Figure 3 There are two processes involved:

[0099] 1. The image training process of deep learning models, in which (e.g.) Figure 3 In path 1), the images are first classified and labeled to create a training set for deep learning. Before inputting the images into the deep learning network, the images are transformed to the size required for deep learning. Then, the deep learning network is trained to obtain the deep learning model.

[0100] 2. The image classification process utilizing deep learning models, this process occurs during the deployment phase (i.e. Figure 3Path 2): Before inputting the image into the deep learning model for classification prediction, the image is transformed to the size required by the deep learning model. Image stitching is performed based on the classification prediction output of the deep learning deployment module. When the image prediction output is the number "0", a new image with the same size as the randomly cropped input image is created, and the new image is a single white color; when the image prediction output is the number "1", a new image with the same size as the randomly cropped input image is created, and the new image is a single gray color; when the image prediction output is the number "2", a new image with the same size as the randomly cropped input image is created, and the new image is a single red color. After all the images cropped from the original remote sensing image have been predicted, the obtained prediction results images are stitched together according to the position of each image in the original remote sensing image to form a new image with the same size as the original remote sensing image. The red areas in the new image are composed of multiple predicted small red images, representing areas with dense man-made structures in the remote sensing image; the gray areas are composed of multiple predicted small gray images, representing areas with sparse man-made structures in the remote sensing image; and the white areas are composed of multiple predicted small white images, representing areas without man-made structures in the remote sensing image.

[0101] The input image is labeled with the classification tag, and its size is restored to the size of the corresponding image to be classified, resulting in multiple classified images. Each classified image has the same tag as the corresponding input image. The multiple classified images are then stitched together according to the position of the corresponding image to be classified to obtain a stitched remote sensing image. This method transforms the complex deep learning semantic segmentation problem into a deep learning semantic classification problem, reduces the amount of data processing, and thus improves classification efficiency.

[0102] As an example, a platform can be designed based on the aforementioned deep learning-based remote sensing image target extraction method. This platform consists of three parts: an image manipulation module, a deep learning training module, and a deep learning deployment module. The image manipulation module includes random image cropping, image deformation, and image stitching. The deep learning training module includes classification labels and a deep learning network. The deep learning training module uses classification labels to train the deep learning network, and after training, a deep learning model is obtained. The deep learning deployment module uses the deep learning model trained by the deep learning training module to perform classification prediction on the input remote sensing image. In the process, the large input remote sensing image is first randomly cropped into multiple smaller images. All the smaller images are then input into the deep learning deployment module to obtain prediction results. Finally, based on the prediction results, the smaller images are stitched together to obtain a prediction result with the same size as the original input remote sensing image.

[0103] Before training the deep learning model described above, you need to prepare the dataset for training the model. Please refer to [link / reference]. Figure 4 , Figure 4 This is a flowchart illustrating some embodiments of the deep learning-based remote sensing image annotation method provided by the present invention. The dataset generated by the annotation method is used to train the deep learning model in the above annotation method. The annotation method includes the following steps:

[0104] Step 401: Obtain the dataset to be labeled. The dataset to be labeled is the data to be labeled after random cropping of a remote sensing image to be labeled. The remote sensing image includes the target to be labeled.

[0105] Step 402: For each image to be labeled, determine the first average value w of the pixel distance between the bounding rectangles of each target region in the width direction of the image to be labeled. ave And the number of pixels w in the width direction of the image to be labeled, based on the first average value w ave The first ratio α is determined by the ratio of the number of pixels w in the width direction of the image to be labeled.

[0106] As an example, the first ratio α can be expressed as:

[0107]

[0108] Where w represents the number of pixels in the width direction of the remote sensing image, and w is a known value; w ave α represents the average pixel distance between the bounding rectangles of each man-made building region along the width direction, and w represents the average distance between man-made buildings along the width direction. ave The ratio of w to .

[0109] Step 403: For each image to be labeled, determine the second average value h of the pixel distance between the bounding rectangles of each target region in the height direction of the image to be labeled. ave The number of pixels h in the height direction of the image to be labeled is used to determine the second ratio β based on the ratio of the second average value have to the number of pixels h in the height direction of the image to be labeled.

[0110] As an example, the second ratio β can be expressed as:

[0111]

[0112] Where h represents the number of pixels in the height direction of the remote sensing image, and h is a known value. ave β represents the average pixel distance between the bounding rectangles of each man-made building region in the height direction, and β represents the average distance h between man-made buildings in the width direction. ave The ratio of h to .

[0113] To calculate the values ​​of α and β, the key is to calculate the average pixel distance w along the width direction of the man-made building. ave The average h of pixel distances in the height direction ave The solution for these two parameters can be found by referring to the following steps, as an example. Figure 5 (Areas 1, 2, and 3 are multiple images to be classified obtained by random cropping):

[0114] Step 1: First, draw the bounding rectangles of each man-made structure in the remote sensing image. Assuming the number of man-made structure regions is n, then the number of distances in the width direction of the bounding rectangles between different regions is n*(n-1) / 2. (Use w...) ij Represents the shortest distance between the bounding rectangles of the i-th and j-th regions along their width; denoted by h. ij This represents the shortest distance between the bounding rectangles of the i-th and j-th regions along their height; for example, in... Figure 2 If there are 3 man-made building regions, then the number of distances along the width direction is 3*(3-1) / 2=3. The distance between different regions can be represented as w. 12 w 23 and w 13 .

[0115] Step 2: Based on the annotations from Step 1, the average pixel distance (wave) along the width direction can be calculated and expressed as follows:

[0116]

[0117] For example in Figure 2 In the given information, n=3, the average pixel distance w in the width direction is... ave It can be calculated as follows:

[0118]

[0119] Furthermore, when n=1, w ave It can be represented as the average pixel distance from the bounding rectangle of the man-made building to the left and right boundaries of the remote sensing image.

[0120] Step 3: Similar to step 2, we can calculate the average pixel distance h in the height direction. ave And expressed by the following formula:

[0121]

[0122] For example in Figure 2 In the given information, n=3, and the average pixel distance h in the width direction is... ave It can be calculated as follows:

[0123]

[0124] Furthermore, when n=1, h ave It can be represented as the average pixel distance from the circumscribed rectangle of the man-made building to the top and bottom boundaries of the remote sensing image.

[0125] Step 4: Based on formulas (1), (2), (3) and (5), the values ​​of α and β can be calculated.

[0126] Step 404: Label the target distribution in the image to be labeled according to the first ratio α and the second ratio β.

[0127] In some optional implementations, if the first ratio α satisfies a first preset condition and the second ratio β satisfies a second preset condition, then the target distribution in the image to be labeled is the first distribution; if the first ratio α satisfies a fourth preset condition or the second ratio β satisfies a fifth preset condition, then the target distribution in the image to be labeled is the second distribution; otherwise, the target distribution in the image to be labeled is the third distribution. For example, when α≤0.3 and β≤0.3, the remote sensing image is classified as "densely populated with man-made buildings"; when α∈(0.3,0.8] or β∈(0.3,0.8], the remote sensing image is classified as "sparsely populated with man-made buildings"; otherwise, the remote sensing image is classified as "without man-made buildings".

[0128] By following the steps above, multiple remote sensing images can be labeled for model training.

[0129] Please see Figure 6 , Figure 6 These are schematic diagrams illustrating the structure of some embodiments of the remote sensing image target extraction device based on deep learning provided by the present invention. As an implementation of the methods shown in the above figures, the present invention also provides some embodiments of the remote sensing image target extraction device based on deep learning. These device embodiments are similar to... Figure 1 The embodiments of some of the methods shown correspond to this, and the device can be applied to a variety of electronic devices.

[0130] like Figure 6As shown in some embodiments, a remote sensing image target extraction device based on deep learning includes a first processing module 601, a second processing module 602, and a third processing module 603: The first processing module 601 is used to randomly crop a remote sensing image to obtain multiple images to be classified, each image to be classified having a different size, and the remote sensing image contains targets to be extracted; the second processing module 602 is used to normalize the multiple images to be classified to obtain multiple input images of the same size, input the input images into a deep learning model for classification, and obtain a classification label for each input image; the third processing module 603 is used to mark the corresponding input images according to the classification labels, restore the size of the marked input images to the size of the corresponding images to be classified, obtain multiple classified images, each classified image having the same label as the corresponding input image, and stitch the multiple classified images according to the positions of the corresponding images to be classified to obtain a stitched remote sensing image, the stitched remote sensing image showing the distribution of the targets.

[0131] In some optional implementations of the embodiments, the deep learning model includes: convolutional layers, max pooling layers, upper convolutional layers, concatenation layers, fully connected layers, and activation function layers.

[0132] In some optional implementations of the embodiments, the network structure of the deep learning model is as follows:

[0133] Layer 0 is the input layer, which receives the normalized image.

[0134] The first layer is a convolutional layer with a kernel size of 3*3 and a number of 64 kernels.

[0135] The second layer is a convolutional layer with a kernel size of 3*3 and a number of 64 kernels.

[0136] The third layer contains three max pooling operations with pooling window sizes of 2, 3, and 6 respectively.

[0137] The 4th and 5th layers are both convolutional layers, with a kernel size of 3*3 and a kernel count of 64.

[0138] The 6th layer is an upper convolutional layer, containing three types of upper convolution operations with kernel sizes of 2*2, 3*3 and 6*6, and the number of upper convolution kernels is 32.

[0139] The 7th layer is the splicing layer;

[0140] The 8th layer is the max pooling layer, and the pooling window for the max pooling operation is 2.

[0141] The 9th layer is a convolutional layer with a kernel size of 3*3 and 128 kernels.

[0142] The 10th layer is the max pooling layer, and the pooling window for the max pooling operation is 2.

[0143] The 11th layer is a convolutional layer with a kernel size of 3*3 and a number of 256 kernels.

[0144] The 12th layer is the max pooling layer, and the pooling window for the max pooling operation is 2.

[0145] The 13th layer is a convolutional layer with a kernel size of 3*3 and a number of 512 kernels.

[0146] Layer 14 consists of a fully connected layer and a ReLU layer;

[0147] Layer 15 consists of a fully connected layer and a ReLU layer;

[0148] The 16th layer is the Softmax layer;

[0149] The 17th layer is the classification output layer.

[0150] In some optional implementations of the embodiments, the actual spatial distance corresponding to the width and / or height of the image to be classified is a random number between L1 and L2.

[0151] In some optional implementations of the embodiments, the classification label of each input image includes 0, 1, and 2, where 0 indicates that there is no target in the input image corresponding to the classification label, 1 indicates that there is a sparsely distributed target in the input image corresponding to the classification label, and 2 indicates that there is a densely distributed target in the input image corresponding to the classification label.

[0152] In some optional implementations of embodiments, the third processing module 603 is further configured to: set each pixel in the corresponding input image to a different color according to the classification label, with each color corresponding to a classification label, to obtain a labeled input image; restore the size of the labeled input image to the size of the corresponding image to be classified, to obtain multiple classified images, wherein the color of each pixel in the classified images is the same as the color of each pixel in the corresponding input image.

[0153] It is understandable that the modules described in this device are consistent with the reference. Figure 1 The steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the device and the modules and units contained therein, and will not be repeated here.

[0154] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions from the memory 730 to execute a deep learning-based remote sensing image target extraction method and / or a deep learning-based remote sensing image annotation method.

[0155] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the deep learning-based remote sensing image target extraction method and / or the deep learning-based remote sensing image annotation method provided by the above methods.

[0157] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the aforementioned deep learning-based remote sensing image target extraction methods and / or deep learning-based remote sensing image annotation methods.

[0158] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0159] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for target extraction from remote sensing images based on deep learning, characterized in that, include: Multiple images to be classified are obtained by randomly cropping remote sensing images. Each image to be classified has a different size. The remote sensing images contain targets that need to be extracted. After normalizing multiple images to be classified, multiple input images of the same size are obtained. These input images are then fed into a deep learning model for classification to obtain the classification label for each input image. The input image is labeled according to the classification label. The size of the labeled input image is restored to the size of the corresponding image to be classified, resulting in multiple classified images. Each classified image has the same label as the corresponding input image. The multiple classified images are then stitched together according to the position of the corresponding image to be classified to obtain a stitched remote sensing image. The stitched remote sensing image shows the distribution of the target. The classification label of each input image includes 0, 1, and 2, where 0 indicates that there is no target in the input image corresponding to the classification label, 1 indicates that there is a sparsely distributed target in the input image corresponding to the classification label, and 2 indicates that there is a densely distributed target in the input image corresponding to the classification label. The step involves marking the input image according to the classification label, restoring the size of the marked input image to the size of the corresponding image to be classified, and obtaining multiple classified images, including: Based on the classification labels, each pixel in the corresponding input image is set to a different color, and each color corresponds to a classification label, resulting in a labeled input image. The dimensions of the labeled input image are restored to the dimensions of the corresponding image to be classified, resulting in multiple classified images. The color of each pixel in the classified image is the same as the color of each pixel in the corresponding input image.

2. The remote sensing image target extraction method based on deep learning according to claim 1, characterized in that, The deep learning model includes: convolutional layers, max pooling layers, upper convolutional layers, concatenation layers, fully connected layers, and activation function layers.

3. The remote sensing image target extraction method based on deep learning according to claim 2, characterized in that, The network structure of the deep learning model is as follows: Layer 0 is the input layer, which receives the normalized image. The first layer is a convolutional layer with a kernel size of 3*3 and a number of 64 kernels. The second layer is a convolutional layer with a kernel size of 3*3 and a number of 64 kernels. The third layer contains three max pooling operations with pooling window sizes of 2, 3, and 6 respectively. The 4th and 5th layers are both convolutional layers, with a kernel size of 3*3 and a kernel count of 64. The 6th layer is an upper convolutional layer, containing three types of upper convolution operations with kernel sizes of 2*2, 3*3 and 6*6, and the number of upper convolution kernels is 32. The 7th layer is the splicing layer; The 8th layer is the max pooling layer, and the pooling window for the max pooling operation is 2. The 9th layer is a convolutional layer with a kernel size of 3*3 and a number of 128 kernels. The 10th layer is the max pooling layer, and the pooling window for the max pooling operation is 2. The 11th layer is a convolutional layer with a kernel size of 3*3 and a number of 256 kernels. The 12th layer is the max pooling layer, and the pooling window for the max pooling operation is 2. The 13th layer is a convolutional layer with a kernel size of 3*3 and a number of 512 kernels. Layer 14 consists of a fully connected layer and a ReLU layer; Layer 15 consists of a fully connected layer and a ReLU layer; The 16th layer is the Softmax layer; The 17th layer is the classification output layer.

4. The remote sensing image target extraction method based on deep learning according to claim 1, characterized in that, The actual spatial distance corresponding to the width and / or height of the image to be classified is a random number between L1 and L2; the smaller L1 and L2 are, the higher the accuracy of target extraction.

5. A remote sensing image annotation method based on deep learning, characterized in that, The dataset generated by the annotation method is used to train the deep learning model in the deep learning-based remote sensing image target extraction method as described in any one of claims 1 to 4, wherein the annotation method includes: Obtain the dataset to be labeled, which is the data to be labeled after random cropping of a remote sensing image to be labeled, and the remote sensing image includes the target to be labeled; For each image to be labeled, determine a first average value of the pixel distances between the bounding rectangles of each target region along the width direction of the image to be labeled. And the number of pixels w in the width direction of the image to be labeled, based on the first average value. The first ratio α is determined by the ratio of the number of pixels w in the width direction of the image to be labeled; For each image to be labeled, determine the second average value h of the pixel distances between the bounding rectangles of each target region in the height direction of the image to be labeled. ave And the number of pixels h in the height direction of the image to be labeled, based on the second average value h. ave The second ratio β is determined by the ratio of the number of pixels h in the height direction of the image to be labeled; The distribution of targets in the image to be labeled is determined based on the first ratio α and the second ratio β.

6. A target extraction device for remote sensing images based on deep learning, characterized in that, include: The first processing module is used to randomly crop the remote sensing image to obtain multiple images to be classified. Each image to be classified has a different size, and the remote sensing image contains a target that needs to be extracted. The second processing module is used to normalize multiple images to be classified to obtain multiple input images of the same size, and then input the input images into a deep learning model for classification to obtain the classification label of each input image. The third processing module is used to mark the corresponding input image according to the classification label, restore the size of the marked input image to the size of the corresponding image to be classified, obtain multiple classified images, each classified image has the same label as the corresponding input image, and stitch the multiple classified images according to the position of the corresponding image to be classified to obtain a stitched remote sensing image, the stitched remote sensing image shows the distribution of the target; The classification label of each input image includes 0, 1, and 2, where 0 indicates that there is no target in the input image corresponding to the classification label, 1 indicates that there is a sparsely distributed target in the input image corresponding to the classification label, and 2 indicates that there is a densely distributed target in the input image corresponding to the classification label. The step involves marking the input image according to the classification label, restoring the size of the marked input image to the size of the corresponding image to be classified, and obtaining multiple classified images, including: Based on the classification labels, each pixel in the corresponding input image is set to a different color, and each color corresponds to a classification label, resulting in a labeled input image. The dimensions of the labeled input image are restored to the dimensions of the corresponding image to be classified, resulting in multiple classified images. The color of each pixel in the classified image is the same as the color of each pixel in the corresponding input image.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the deep learning-based remote sensing image target extraction method as described in any one of claims 1 to 4.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the deep learning-based remote sensing image target extraction method as described in any one of claims 1 to 4.

Images