Target extraction method and device, electronic equipment and storage medium

Abstract

The application provides a target extraction method and device, electronic equipment and storage medium, the method relates to the field of image processing technology, and comprises the following steps: acquiring an image of a target to be extracted; determining edge features of the image; inputting the image and the edge features into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on sample images after training; the target extraction result is used to represent a building in the image; and the sample images are obtained based on initial sample images. The method provided by the application performs deep fusion on the image of the target to be extracted and the edge features corresponding to the image through the multi-scale edge constraint model, refines the boundary of the building in the image, and improves the precision of building extraction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a target extraction method and device, electronic equipment and storage medium. BACKGROUND

[0002] In recent years, with the rapid development of sensor technology, the spatial resolution of remote sensing images is continuously improved. Remote sensing images with higher spatial resolution have more abundant spatial detail information, making it possible to accurately extract buildings. Due to the very complex external environment of the buildings, such as building ancillary facilities, shadow shielding, etc., it is difficult to extract the features of the buildings; the contrast between the buildings and some non-buildings (parking lots, bare land, roads) is low, which easily interferes with the identification results of the buildings. In addition, the profiles, structures and materials of buildings in different regions differ greatly, so the fine extraction of building remote sensing images is still a challenging task.

[0003] In related technologies, a convolutional neural network is used to extract buildings in remote sensing images. Although the overall accuracy of building extraction is already very high, there is still a large error in the boundary area of the buildings. Therefore, how to improve the extraction accuracy of the boundary area of the buildings is a problem to be solved. SUMMARY

[0004] The present application provides a target extraction method, device, electronic equipment and storage medium to solve the problem of low extraction accuracy of the boundary area of the building in the prior art, which realizes the complete extraction of the building and improves the accuracy of building extraction.

[0005] The present application provides a target extraction method, which comprises:

[0006] obtaining an image of a target to be extracted;

[0007] determining the edge features of the image;

[0008] inputting the image and the edge features into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on training of a sample image; the target extraction result is used to represent a building in the image; and the sample image is obtained based on an initial sample image.

[0009] According to the target extraction method provided by the present application, the multi-scale edge constraint model comprises an encoder, a decoder and an edge constraint block.

[0010] The image and the edge feature are input into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model, including:

[0011] The image is input into the encoder to obtain at least one first feature information of the image output by the encoder; each first feature information is used to represent a feature of different dimensions corresponding to the image;

[0012] Each first feature information is respectively up-sampled to obtain at least one first up-sampling result after up-sampling;

[0013] Each first feature information and each first up-sampling result are input into the decoder to obtain at least one decoding result output by the decoder; each decoding result is used to represent feature information of different resolutions corresponding to the image;

[0014] The edge feature and a second up-sampling result corresponding to each decoding result are input into the edge constraint block to obtain a target extraction result of the image output by the edge constraint block.

[0015] According to the target extraction method provided by the application, the encoder includes at least one encoding unit connected in sequence;

[0016] The image is input into the encoder to obtain at least one first feature information of the image output by the encoder, including:

[0017] The image is input into a first encoding unit to obtain first dimension feature information output by the first encoding unit;

[0018] The first dimension feature information is down-sampled to obtain a first down-sampling result after sampling;

[0019] The first down-sampling result is input into a second encoding unit to obtain second dimension feature information output by the second encoding unit;

[0020] The second dimension feature information is down-sampled to obtain a second down-sampling result after sampling;

[0021] The second down-sampling result is input into a third encoding unit to obtain third dimension feature information output by the third encoding unit;

[0022] The third dimension feature information is down-sampled to obtain a third down-sampling result after sampling;

[0023] The third down-sampling result is input into a fourth encoding unit to obtain fourth dimension feature information output by the fourth encoding unit;

[0024] down-sampling the fourth dimension feature information to obtain a fourth down-sampling result after sampling;

[0025] inputting the fourth down-sampling result into a fifth encoding unit to obtain fifth dimension feature information output by the fifth encoding unit.

[0026] According to the target extraction method provided in the application, the decoder comprises at least one decoding unit;

[0027] The inputting of the first feature information and the first up-sampling result into the decoder to obtain at least one decoding result output by the decoder comprises:

[0028] inputting the first up-sampling result corresponding to the second dimension feature information and the third dimension feature information into a first decoding unit to obtain a first decoding result output by the first decoding unit;

[0029] inputting the first up-sampling result corresponding to the third dimension feature information and the fourth dimension feature information into a second decoding unit to obtain a second decoding result output by the second decoding unit;

[0030] inputting the first up-sampling result corresponding to the fourth dimension feature information and the fifth dimension feature information into a third decoding unit to obtain a third decoding result output by the third decoding unit;

[0031] determining at least one decoding result output by the decoder based on the first decoding result, the second decoding result and the third decoding result.

[0032] According to the target extraction method provided in the application, the edge constraint block comprises at least one edge constraint unit connected in sequence;

[0033] The inputting of the second up-sampling result corresponding to the edge feature and the decoding result into the edge constraint block to obtain the target extraction result of the image output by the edge constraint block comprises:

[0034] In the case that the edge constraint unit is a first edge constraint unit, inputting the first up-sampling result corresponding to the first dimension feature information and the second dimension feature information and the edge feature into the first edge constraint unit to obtain first fusion feature information output by the first edge constraint unit; the first fusion feature information is used to represent the fusion of the first up-sampling result corresponding to the first dimension feature information and the second dimension feature information and the edge feature;

[0035] In the case that the edge constraint unit is a non-first edge constraint unit, the edge feature, second fusion feature information corresponding to all edge constraint units before the non-first edge constraint unit and second up-sampling results corresponding to decoding results output by the decoder are input into the non-first edge constraint unit to obtain third fusion feature information output by the non-first edge constraint unit; and a result output by a last edge constraint unit is taken as a target extraction result of the image.

[0036] According to the target extraction method provided by the application, the first dimension feature information, the second dimension feature information and the edge feature are spliced to determine second feature information after splicing.

[0037] The first dimension feature information, the second dimension feature information and the edge feature are spliced to determine second feature information after splicing.

[0038] The second feature information after splicing is convoluted twice to determine third feature information after convolution.

[0039] The third feature information is input into the self-attention mechanism unit to obtain first fusion feature information output by the self-attention mechanism unit.

[0040] According to the target extraction method provided by the application, the edge feature, second fusion feature information corresponding to all edge constraint units before the non-first edge constraint unit and second up-sampling results corresponding to decoding results output by the decoder are input into the non-first edge constraint unit to obtain third fusion feature information output by the non-first edge constraint unit, including:

[0041] The edge feature, second fusion feature information corresponding to all edge constraint units before the non-first edge constraint unit and second up-sampling results corresponding to decoding results output by the decoder are spliced to determine fourth feature information after splicing.

[0042] The fourth feature information after splicing is convoluted twice to determine fifth feature information after convolution.

[0043] The fifth feature information is input into the self-attention mechanism unit to obtain third fusion feature information output by the self-attention mechanism unit.

[0044] According to the target extraction method provided by the application, the decoder unit comprises a self-attention mechanism unit.

[0045] The first up-sampling result corresponding to the second dimension feature information and the third dimension feature information is input into a first decoding unit to obtain a first decoding result output by the first decoding unit.

[0046] The first up-sampling result corresponding to the second dimension feature information and the third dimension feature information is spliced to determine sixth feature information after splicing;

[0047] The sixth feature information is convolved twice to determine seventh feature information after convolution;

[0048] The seventh feature information is input into the self-attention mechanism unit to obtain a first decoding result output by the self-attention mechanism unit.

[0049] According to the target extraction method provided by the application, the sample image is obtained based on an initial sample image, which comprises:

[0050] At least one initial sample image pair is selected; the initial sample image pair comprises a background image and a copied image;

[0051] The Fourier transform is performed on each pixel point in the background image and the copied image respectively to obtain a frequency domain image after transformation;

[0052] The frequency domain images corresponding to the background image and the copied image are compared respectively;

[0053] In the case that the low-frequency pixel points in the frequency domain image corresponding to the background image are less than the low-frequency pixel points in the frequency domain image corresponding to the copied image, the low-frequency pixel points in the frequency domain image corresponding to the copied image are replaced with the low-frequency pixel points in the frequency domain image corresponding to the background image, and the background image after replacement is determined as the sample image;

[0054] In the case that the low-frequency pixel points in the frequency domain image corresponding to the background image are not less than the low-frequency pixel points in the frequency domain image corresponding to the copied image, the copied image is taken as the sample image.

[0055] According to the target extraction method provided by the application, the method further comprises:

[0056] The copied image is operated with the corresponding mask, and the operation result is dilated;

[0057] In the case that the background image and the copied image have an overlapping area, the building in the overlapping area of the background image is deleted;

[0058] The low-frequency pixel points in the frequency domain image corresponding to the copied image are copied to the background image to obtain the sample image.

[0059] The application further provides a target extraction device, which comprises:

[0060] an acquisition module, configured to acquire an image of a target to be extracted;

[0061] a determination module, configured to determine an edge feature of the image;

[0062] an extraction module, configured to input the image and the edge feature into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on a sample image after training; the target extraction result is used to represent a building in the image; and the sample image is obtained based on an initial sample image.

[0063] The application further provides an electronic device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the target extraction method according to any one of the above when executing the program.

[0064] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the target extraction method according to any one of the above.

[0065] The application further provides a computer program product, which comprises a computer program, and the computer program is executable on a processor to implement the target extraction method according to any one of the above.

[0066] The target extraction method, device, electronic device and storage medium provided by the application acquire an image of a target to be extracted, determine an edge feature of the image, input the image and the edge feature into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model, the multi-scale edge constraint model is obtained based on a sample image after training, the target extraction result is used to represent a building in the image, and the sample image is obtained based on an initial sample image. The target extraction method provided by the application deep-fuses the image of the target to be extracted and the edge feature corresponding to the image through the multi-scale edge constraint model, refines the boundary of the building in the image, and improves the precision of building extraction. BRIEF DESCRIPTION OF DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the present application or prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0068] Figure 1 is one of the flow schematic diagrams of the target extraction method provided by the present application;

[0069] Figure 2 is the second flow schematic diagram of the target extraction method provided by the present application;

[0070] Figure 3 is the operation flow chart of the self-attention mechanism unit provided by the present application;

[0071] Figure 4 is the structural schematic diagram of the edge constraint unit provided by the present application;

[0072] Figure 5 is the structural schematic diagram of the multi-scale edge constraint model provided by the present application;

[0073] Figure 6 is the result schematic diagram of the sample image acquisition method provided by the present application;

[0074] Figure 7 is the result schematic diagram of the target extraction method provided by the present application;

[0075] Figure 8 is the structural schematic diagram of the target extraction device provided by the present application;

[0076] Figure 9 is the structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0077] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0078] The target extraction method provided by the present application will be described in detail below in combination with the drawings and some embodiments and application scenarios thereof.

[0079] The target extraction method provided by the application is suitable for a target extraction scene in remote sensing images, and comprises the following steps: acquiring an image of a target to be extracted; determining edge features of the image; inputting the image and the edge features into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on training of sample images; the target extraction result is used to represent a building in the image; and the sample images are obtained based on initial sample images. The method provided by the application performs deep fusion on the image of the target to be extracted and the edge features corresponding to the image by using the multi-scale edge constraint model, refines the boundary of the building in the image, and improves the accuracy of building extraction.

[0080] The target extraction method provided by the application is suitable for a target extraction scene in remote sensing images, and comprises the following steps: acquiring an image of a target to be extracted; determining edge features of the image; inputting the image and the edge features into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on training of sample images; the target extraction result is used to represent a building in the image; and the sample images are obtained based on initial sample images. The method provided by the application performs deep fusion on the image of the target to be extracted and the edge features corresponding to the image by using the multi-scale edge constraint model, refines the boundary of the building in the image, and improves the accuracy of building extraction. Figures 1-7 The target extraction method provided by the application is suitable for a target extraction scene in remote sensing images, and comprises the following steps: acquiring an image of a target to be extracted; determining edge features of the image; inputting the image and the edge features into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on training of sample images; the target extraction result is used to represent a building in the image; and the sample images are obtained based on initial sample images. The method provided by the application performs deep fusion on the image of the target to be extracted and the edge features corresponding to the image by using the multi-scale edge constraint model, refines the boundary of the building in the image, and improves the accuracy of building extraction.

[0081] Figure 1 The target extraction method provided by the application is suitable for a target extraction scene in remote sensing images, and comprises the following steps: acquiring an image of a target to be extracted; determining edge features of the image; inputting the image and the edge features into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on training of sample images; the target extraction result is used to represent a building in the image; and the sample images are obtained based on initial sample images. The method provided by the application performs deep fusion on the image of the target to be extracted and the edge features corresponding to the image by using the multi-scale edge constraint model, refines the boundary of the building in the image, and improves the accuracy of building extraction. Figure 1 The target extraction method provided by the application is suitable for a target extraction scene in remote sensing images, and comprises the following steps: acquiring an image of a target to be extracted; determining edge features of the image; inputting the image and the edge features into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on training of sample images; the target extraction result is used to represent a building in the image; and the sample images are obtained based on initial sample images. The method provided by the application performs deep fusion on the image of the target to be extracted and the edge features corresponding to the image by using the multi-scale edge constraint model, refines the boundary of the building in the image, and improves the accuracy of building extraction.

[0082] Step 101, acquiring an image of a target to be extracted.

[0083] It should be noted that the target extraction method provided by the application is suitable for a target extraction scene in remote sensing images. The execution subject of the method can be a target extraction device, such as an electronic device, or a control module in the target extraction device for executing the target extraction method.

[0084] Specifically, remote sensing images of a target region can be photographed by an airplane or a satellite, and the image of the target to be extracted can be acquired; for example, the target to be extracted can be a building or other targets.

[0085] Step 102, determining edge features of the image.

[0086] Specifically, a Sobel operator is used to convolve the Sobel operator with the image, and the gradient values obtained are calculated to obtain the edge features of the image.

[0087] The Sobel operator is a common first-order derivative edge detection operator, which uses two 3*3 matrices to convolve with the image to obtain the gradient values of horizontal Ex and vertical Ey, respectively, and uses formulas (1) and (2) respectively; according to the gradient values of horizontal Ex and vertical Ey, the edge features E are obtained, which are represented by formula (3):

[0088]

[0089]

[0090]

[0091] wherein, I represents an image, E x represents a horizontal gradient value, E y represents a vertical gradient value, E represents an edge feature.

[0092] Step 103, inputting the image and the edge feature into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on a sample image after training; the target extraction result is used to represent a building in the image; and the sample image is obtained based on an initial sample image.

[0093] Specifically, the multi-scale edge constraint model (MEC-Net) takes U-Net++ as a basic network and adopts a residual network (Resnet50) as a backbone for extracting buildings in a remote sensing image.

[0094] In practice, the image of the target to be extracted and the edge feature of the image are input into the multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model, wherein the target extraction result is used to represent a building in the image, that is, the building in the image is extracted by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on a sample image after training, wherein the sample image is obtained based on an initial sample image.

[0095] The target extraction method provided by the application comprises the following steps: obtaining an image of a target to be extracted; determining an edge feature of the image; inputting the image and the edge feature into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model. The method provided by the application deepens the fusion of the image of the target to be extracted and the corresponding edge feature of the image by the multi-scale edge constraint model, refines the boundary of the building in the image, and improves the precision of building extraction.

[0096] Figure 2 is a flowchart of the target extraction method provided by the application, as shown in Figure 2 The method comprises steps 201-208, wherein:

[0097] Step 201, obtaining an image of a target to be extracted;

[0098] Step 202, determining an edge feature of the image;

[0099] Step 203, input the image into the encoder to obtain at least one first feature information of the image output by the encoder; each first feature information is used to represent different dimensions of features corresponding to the image.

[0100] Specifically, the multi-scale edge constraint model comprises an encoder, a decoder and an edge constraint block; an image is input into the encoder to obtain at least one first feature information of the image output by the encoder; wherein each first feature information is used to represent different dimensions of features corresponding to the image.

[0101] Optionally, the encoder comprises at least one encoding unit connected in sequence; and the specific implementation of step 203 comprises the following steps:

[0102] Step 1), input the image into the first encoding unit to obtain first dimension feature information output by the first encoding unit.

[0103] Specifically, the image is input into the first encoding unit, the first encoding unit performs convolution and ReLU function activation on the image, obtains the result output by the ReLU function, and then performs convolution and ReLU function activation on the output result to obtain the first dimension feature information output by the first encoding unit; wherein the size and number of the convolution kernel can be set according to actual conditions, for example, the size of the convolution kernel is 3*3 and the number is 64.

[0104] Step 2), down-sampling the first dimension feature information to obtain a first down-sampling result after sampling.

[0105] Specifically, the first dimension feature information is down-sampled through a pooling kernel to obtain the first down-sampling result; for example, the size of the pooling kernel can be 2*2, and the first dimension feature information is down-sampled to half of the original dimension.

[0106] Step 3), input the first down-sampling result into the second encoding unit to obtain second dimension feature information output by the second encoding unit.

[0107] Specifically, the first down-sampling result is input into the second encoding unit, the second encoding unit performs convolution and ReLU function activation on the first down-sampling result, obtains the result output by the ReLU function, and then performs convolution and ReLU function activation on the output result to obtain the second dimension feature information output by the second encoding unit; wherein the number of convolution kernels is twice the number of convolution kernels in step 1), and the dimension of the second dimension feature information is higher than that of the first dimension feature information.

[0108] Step 4), down-sampling the second dimension feature information to obtain a second down-sampling result after sampling.

[0109] Specifically, the second dimension feature information is down-sampled by a pooling kernel to obtain a second down-sampling result.

[0110] Step 5) inputting the second down-sampling result into a third encoding unit to obtain third dimension feature information output by the third encoding unit.

[0111] Specifically, the second down-sampling result is input into the third encoding unit, the third encoding unit performs convolution and ReLU function activation on the second down-sampling result, obtains a result output by the ReLU function, and then performs convolution and ReLU function activation on the output result to obtain third dimension feature information output by the third encoding unit; wherein the number of convolution kernels is twice the number of convolution kernels in step 3), and the dimension of the third dimension feature information is higher than that of the second dimension feature information.

[0112] Step 6) down-sampling the third dimension feature information to obtain a third down-sampling result after sampling.

[0113] Specifically, the third dimension feature information is down-sampled by a pooling kernel to obtain a third down-sampling result.

[0114] Step 7) inputting the third down-sampling result into a fourth encoding unit to obtain fourth dimension feature information output by the fourth encoding unit.

[0115] Specifically, the third down-sampling result is input into the fourth encoding unit, the fourth encoding unit performs convolution and ReLU function activation on the third down-sampling result, obtains a result output by the ReLU function, and then performs convolution and ReLU function activation on the output result to obtain fourth dimension feature information output by the fourth encoding unit; wherein the number of convolution kernels is twice the number of convolution kernels in step 5), and the dimension of the fourth dimension feature information is higher than that of the third dimension feature information.

[0116] Step 8) down-sampling the fourth dimension feature information to obtain a fourth down-sampling result after sampling.

[0117] Specifically, the fourth dimension feature information is down-sampled by a pooling kernel to obtain a fourth down-sampling result.

[0118] Step 9) inputting the fourth down-sampling result into a fifth encoding unit to obtain fifth dimension feature information output by the fifth encoding unit.

[0119] Specifically, the fourth down-sampling result is input to a fifth encoding unit, the fifth encoding unit performs convolution and ReLU function activation on the fourth down-sampling result, and then performs convolution and ReLU function activation on the output result to obtain fifth-dimensional feature information output by the fifth encoding unit; wherein the number of convolution kernels is twice the number of convolution kernels in step 7), and the dimension of the fifth-dimensional feature information is higher than the dimension of the fourth-dimensional feature information.

[0120] In step 204, each of the first feature information is up-sampled to obtain at least one first up-sampling result after up-sampling.

[0121] Specifically, the at least one first feature information output by the encoder is deconvoluted, that is, each of the first feature information is up-sampled to obtain at least one first up-sampling result after up-sampling.

[0122] In step 205, each of the first feature information and each of the first up-sampling result is input to the decoder to obtain at least one decoding result output by the decoder; each of the decoding result is used to represent feature information of different resolutions corresponding to the image.

[0123] Optionally, the decoder comprises at least one decoding unit; and the specific implementation manner of step 205 comprises the following steps:

[0124] In step 1), the first up-sampling result corresponding to the second-dimensional feature information and the third-dimensional feature information is input to a first decoding unit to obtain a first decoding result output by the first decoding unit.

[0125] Specifically, the first up-sampling result corresponding to the second-dimensional feature information and the third-dimensional feature information output by the second encoding unit is input to a first decoding unit, the first decoding unit fuses the second-dimensional feature information and the first up-sampling result to obtain a first decoding result output by the first decoding unit.

[0126] Optionally, the decoding unit comprises a self-attention mechanism unit.

[0127] The first up-sampling result corresponding to the second-dimensional feature information and the third-dimensional feature information is input to a first decoding unit to obtain a first decoding result output by the first decoding unit, comprising:

[0128] The first up-sampling result corresponding to the second-dimensional feature information and the third-dimensional feature information is spliced to determine sixth feature information after splicing;

[0129] The sixth feature information is convoluted twice to determine seventh feature information after convolution;

[0130] The seventh feature information is input into the self-attention mechanism unit to obtain a first decoding result output by the self-attention mechanism unit.

[0131] Specifically, the decoder comprises at least one decoding unit, and each decoding unit comprises a self-attention mechanism unit, i.e., scSE. The self-attention mechanism can enable the model to learn to use attention, i.e., to focus on information or features of interest, so as to ignore unimportant information and enhance the use of more useful information. The self-attention mechanism unit scSE is a parallel combination of a channel attention mechanism unit (cSE) and a spatial attention mechanism unit (sSE), specifically, after passing through the sSE and the cSE respectively, the output results of the sSE and the cSE are added to obtain a more accurate corrected feature map.

[0132] It should be noted that the cSE module belongs to the channel attention mechanism and is used to redistribute the weights of the feature map in the channel dimension to obtain a channel weighted feature map. First, the shape of the feature map is converted from (C, H, W) to (C, 1, 1) through global average pooling, and then two 1x1 convolution blocks and an activation function (ReLU) are used to obtain a channel feature importance degree vector with a shape of (C, 1, 1). The channel feature importance degree vector is multiplied with the original feature map in a channel-wise manner to obtain a feature map corrected by attention in the channel. The sSE module belongs to the spatial attention mechanism and is used to redistribute the weights of the spatial information of the same feature map to obtain a feature map containing different spatial weight information. First, a 1x1 convolution block with a channel of C is directly used for channel compression of the feature map to convert its shape from (C, H, W) to (C, 1, 1), and then a sigmoid function is used for activation to obtain a spatial feature importance degree vector. The spatial feature importance degree vector is multiplied with the original feature map to obtain a feature map corrected by attention in the space.

[0133] Figure 3 is an operation flowchart of the self-attention mechanism unit provided by the present application, as shown in Figure 3As shown, the channel attention mechanism unit (cSE) passes the original feature map through global average pooling, then through two 1x1 convolution blocks and an activation function (ReLU), to obtain a channel feature importance degree vector with a shape of (C, 1, 1), and finally multiply the channel feature importance degree vector with the original feature map to obtain the feature map on the channel that has been corrected by attention; the spatial attention mechanism unit (sSE) passes the original feature map through a 1x1 convolution block with C channels for channel compression, so that the shape of the original feature map is converted from (C, H, W) to (C, 1, 1), and then a sigmoid function is used for activation to obtain a spatial feature importance degree map vector, and finally the spatial feature importance degree map vector is multiplied with the original feature map to obtain the feature map on the space that has been corrected by attention; the self-attention mechanism unit (scSE) inputs the original feature map into the cSE and the sSE to obtain the feature map on the channel that has been corrected by attention output by the cSE and the feature map on the space that has been corrected by attention output by the sSE, and adds the two feature maps to obtain a more accurately corrected feature map.

[0134] In practice, the first decoding unit concatenates the first up-sampling result corresponding to the second dimension feature information output by the second encoding unit and the third dimension feature information output by the third encoding unit after convolution, to obtain the sixth feature information after concatenation; then the sixth feature information is input into a convolution kernel and a ReLu function, and then input into a convolution kernel and a ReLu function again, to obtain the sixth feature information; the sixth feature information is input into the self-attention mechanism unit (scSE) to obtain the first decoding result output by the self-attention mechanism unit.

[0135] Step 2) input the first up-sampling result corresponding to the third dimension feature information and the fourth dimension feature information into the second decoding unit to obtain the second decoding result output by the second decoding unit.

[0136] Specifically, the second decoding unit concatenates the first up-sampling result corresponding to the third dimension feature information output by the third encoding unit and the fourth dimension feature information output by the fourth encoding unit after convolution, to obtain the eighth feature information after concatenation; then the eighth feature information is input into a convolution kernel and a ReLu function, and then input into a convolution kernel and a ReLu function again, to obtain the ninth feature information; the ninth feature information is input into the self-attention mechanism unit (scSE) to obtain the second decoding result output by the self-attention mechanism unit.

[0137] Step 3) input the first up-sampling result corresponding to the fourth dimension feature information and the fifth dimension feature information into a third decoding unit to obtain a third decoding result output by the third decoding unit.

[0138] Specifically, the third decoding unit concatenates the first up-sampling result corresponding to the fourth dimension feature information output by the fourth encoding unit and the fifth dimension feature information output by the fifth encoding unit after convolution to obtain tenth feature information after concatenation; then the tenth feature information is processed by a convolution kernel and a ReLu function, and then processed by a convolution kernel and a ReLu function again to obtain eleventh feature information; the eleventh feature information is input into a self-attention mechanism unit (scSE) to obtain a third decoding result output by the self-attention mechanism unit.

[0139] Step 4) determine at least one decoding result output by the decoder based on the first decoding result, the second decoding result and the third decoding result.

[0140] Specifically, according to the first decoding result output by the first decoding unit, the second decoding result output by the second decoding unit and the third decoding result output by the third decoding unit, the steps of decoding by the first decoding unit, the second decoding unit or the third decoding unit can obtain the decoding results output by other decoding units of the decoder respectively.

[0141] Step 206) input the second up-sampling result corresponding to each of the edge features and the decoding results into the edge constraint block to obtain a target extraction result of the image output by the edge constraint block.

[0142] Specifically, input the first up-sampling result corresponding to the first dimension feature information output by the first encoding unit and the first dimension feature information output by the second encoding unit, and the second up-sampling result corresponding to each of the decoding results into the edge constraint block to obtain a target extraction result of the image output by the edge constraint block.

[0143] The target extraction method provided by the application comprises the following steps: inputting an image into an encoder to obtain at least one first feature information of the image output by the encoder; performing upsampling on each first feature information to obtain at least one first up-sampling result after upsampling; inputting each first feature information and each first up-sampling result into a decoder to obtain at least one decoding result output by the decoder; inputting an edge feature and a second up-sampling result corresponding to each decoding result into an edge constraint block to obtain a target extraction result of the image output by the edge constraint block, thereby realizing the fusion of the image of the target to be extracted, the edge feature corresponding to the image and feature information of different scales through the encoder, the decoder and the edge constraint block, refining the boundary of a building in the image and improving the precision of building extraction.

[0144] Optionally, the edge constraint block comprises at least one edge constraint unit connected in series.

[0145] The edge constraint block comprises at least one edge constraint unit connected in series.

[0146] In the case that the edge constraint unit is the first edge constraint unit, the first dimension feature information, the second dimension feature information corresponding to the first up-sampling result and the edge feature are input into the first edge constraint unit to obtain first fusion feature information output by the first edge constraint unit; the first fusion feature information is used to represent the fusion of the first dimension feature information, the second dimension feature information corresponding to the first up-sampling result and the edge feature.

[0147] In the case that the edge constraint unit is the first edge constraint unit, the first dimension feature information, the second dimension feature information corresponding to the first up-sampling result and the edge feature are input into the first edge constraint unit to obtain first fusion feature information output by the first edge constraint unit; the first fusion feature information is used to represent the fusion of the first dimension feature information, the second dimension feature information corresponding to the first up-sampling result and the edge feature.

[0148] Specifically, the edge constraint units in the edge constraint block are described respectively.

[0149] 1) In the case that the edge constraint unit in the edge constraint block is the first edge constraint unit, the first dimension feature information, the second dimension feature information corresponding to the first up-sampling result and the edge feature are input into the first edge constraint unit to obtain first fusion feature information output by the first edge constraint unit, which comprises the following steps:

[0150] The first dimension feature information, the first up-sampling result corresponding to the second dimension feature information, and the edge feature are spliced to determine second feature information after splicing; the second feature information after splicing is convolved twice to determine third feature information after convolution; and the third feature information is input into a self-attention mechanism unit to obtain first fusion feature information output by the self-attention mechanism unit.

[0151] Specifically, the first edge constraint unit splices the first dimension feature information, the result after convolution of the first up-sampling result corresponding to the second dimension feature information, and the edge feature according to formula (4) to determine second feature information after splicing; the second feature information after splicing is convolved twice and activated by a ReLU function to obtain third feature information; and the third feature information is input into a self-attention mechanism unit to obtain first fusion feature information output by the self-attention mechanism unit according to formula (5); wherein:

[0152]

[0153]

[0154] wherein E represents the edge feature, represents the first dimension feature information, and k is 0, represents the second dimension feature information, represents the second feature information, and Output j represents the first fusion feature information, j is 1, and represents the first edge constraint unit.

[0155] 2) In the case where the edge constraint unit is not the first edge constraint unit, the edge feature, second fusion feature information corresponding to all edge constraint units before the non-first edge constraint unit, and second up-sampling result corresponding to the decoding result output by the decoder are input into the non-first edge constraint unit to obtain third fusion feature information output by the non-first edge constraint unit, including:

[0156] The edge feature, second fusion feature information corresponding to all edge constraint units before the non-first edge constraint unit, and second up-sampling result corresponding to the decoding result output by the decoder are spliced to determine fourth feature information after splicing; the fourth feature information after splicing is convolved twice to determine fifth feature information after convolution; and the fifth feature information is input into the self-attention mechanism unit to obtain third fusion feature information output by the self-attention mechanism unit.

[0157] Specifically, the non-first edge constraint unit can also use the above formula (4) to splice the edge feature, the second fusion feature information corresponding to all edge constraint units before the non-first edge constraint unit, and the result after the decoder output decoding result corresponding second up-sampling result convolution, and perform twice convolution and ReLU function activation on the spliced fourth feature information to obtain fifth feature information; then input the fifth feature information into the self-attention mechanism unit, that is, use formula (5) to obtain the third fusion feature information output by the self-attention mechanism unit, wherein the result output by the last edge constraint unit is taken as the target extraction result of the image. It should be noted that when the result output by the last edge constraint unit is taken as the target extraction result of the image, the result output by the last edge constraint unit is compared with the target threshold, and the pixel points corresponding to the results greater than the threshold are white, and the pixel points corresponding to the results less than the threshold are black. At the same time, at this time, the formula (4) and the formula (5) in the represents the second fusion feature information, k is greater than 0, represents the decoding result output by the decoder, represents the fourth feature information, Output j represents the third fusion feature information, j is greater than 1, and represents the non-first edge constraint unit.

[0158] Figure 4 is a structural schematic diagram of the edge constraint unit provided by the application, as shown in Figure 4 Edge_feature, Low_feature(0), Low_feature(1), …, Low_feature(j-1) and High_feature(j-1) are spliced by Concat after the results of the convolution of the corresponding second up-sampling results, then the spliced results are passed through convolution (Conv2d) and activation function (ReLU), then through convolution (Conv2d) and activation function (ReLU), and then through the self-attention mechanism unit (scSE) to obtain the fusion feature information output by the self-attention mechanism unit.

[0159] Figure 5 is a structural schematic diagram of the multi-scale edge constraint model provided by the application, as shown in Figure 5 X 0,0 , X 1,0 , X 2 ,0 , X 3,0 , X 4,0 comprise an encoder, wherein X 0,0 , X 1,0 , X 2,0 , X 3,0 , X 4,0respectively represent the first encoding unit, the second encoding unit, the third encoding unit, the fourth encoding unit, the fifth encoding unit; X 1,1 , X 1,2 , X 1,3 , X 2,1 , X 2,2 , X 3,1 constitute a decoder, wherein X 1,1 represents the first decoding unit, X 2,1 represents the second decoding unit, X 3,1 represents the third decoding unit, and X 1,2 , X 1,3 , X 2 ,2 all represent other decoding units; X 0,1 , X 0,2 , X 0,3 , X 0,4 constitute a MEC block, wherein X 0,1 represents the first MEC unit, X 0,2 , X 0,3 , X 0,4 all represent non-first MEC units, and X 0,4 the output result is compared with a target threshold value, and then taken as the target extraction result of the image.

[0160] Next, the target extraction method provided by the present application is described in detail according to the structural schematic diagram of the multi-scale edge constraint model shown in Figure 5

[0161] 1) an image of a target to be extracted is obtained;

[0162] 2) an edge feature of the image is determined using a Sobel operator;

[0163] ​3) input the image into a first encoding unit of the encoder, the first encoding unit performs convolution and ReLU function activation on the image, and then performs convolution and ReLU function activation again, that is, performs twice convolution and ReLU function activation on the image to obtain first dimension feature information output by the first encoding unit; the first dimension feature information is down-sampled, that is, the first dimension feature information is input into a pooling kernel to obtain a first down-sampled result after sampling; the first down-sampled result is input into a second encoding unit, the second encoding unit performs twice convolution and ReLU function activation on the first down-sampled result to obtain second dimension feature information output by the second encoding unit; the second dimension feature information is down-sampled, that is, the second dimension feature information is input into a pooling kernel to obtain a second down-sampled result after sampling; the second down-sampled result is input into a third encoding unit, the third encoding unit performs twice convolution and ReLU function activation on the second down-sampled result to obtain third dimension feature information output by the third encoding unit; the third dimension feature information is down-sampled, that is, the third dimension feature information is input into a pooling kernel to obtain a third down-sampled result after sampling; the third down-sampled result is input into a fourth encoding unit, the fourth encoding unit performs twice convolution and ReLU function activation on the third down-sampled result to obtain fourth dimension feature information output by the fourth encoding unit; the fourth dimension feature information is down-sampled, that is, the fourth dimension feature information is input into a pooling kernel to obtain a fourth down-sampled result after sampling; the fourth down-sampled result is input into a fifth encoding unit, the fifth encoding unit performs twice convolution and ReLU function activation on the fourth down-sampled result to obtain fifth dimension feature information output by the fifth encoding unit;

[0164] 4) input the first up-sampled result corresponding to the second dimension feature information and the third dimension feature information into a first decoding unit, the first decoding unit splices the result after convolution of the first up-sampled result corresponding to the second dimension feature information and the third dimension feature information to determine sixth feature information after splicing; the sixth feature information is subjected to twice convolution and ReLU function activation to determine seventh feature information after convolution; the seventh feature information is input into a self-attention mechanism unit to obtain a first decoding result output by the self-attention mechanism unit;

[0165] input the first up-sampled result corresponding to the third dimension feature information and the fourth dimension feature information into a second decoding unit, the second decoding unit splices the first up-sampled result corresponding to the third dimension feature information and the fourth dimension feature information to determine seventh feature information after splicing; the seventh feature information is subjected to twice convolution to determine eighth feature information after convolution; the eighth feature information is input into a self-attention mechanism unit to obtain a second decoding result output by the self-attention mechanism unit;

[0166] The first up-sampling result corresponding to the fourth-dimensional feature information and the fifth-dimensional feature information is input to a third decoding unit, the third decoding unit performs splicing on the first up-sampling result corresponding to the fourth-dimensional feature information and the fifth-dimensional feature information, and determines ninth feature information after splicing; the ninth feature information is convolved twice, and tenth feature information after convolution is determined; the tenth feature information is input to a self-attention mechanism unit, and a third decoding result output by the self-attention mechanism unit is obtained;

[0167] The second up-sampling result corresponding to the second-dimensional feature information, the first decoding result and the second decoding result is input to a fourth decoding unit, the fourth decoding unit performs splicing on the result after convolution of the second up-sampling result corresponding to the second-dimensional feature information, the first decoding result and the second decoding result, and determines eleventh feature information after splicing; the eleventh feature information is convolved twice and activated by a ReLU function, and twelfth feature information after convolution is determined; the twelfth feature information is input to a self-attention mechanism unit, and a fourth decoding result output by the self-attention mechanism unit is obtained;

[0168] The second up-sampling result corresponding to the third-dimensional feature information, the second decoding result and the third decoding result is input to a fifth decoding unit, the fifth decoding unit performs splicing on the result after convolution of the second up-sampling result corresponding to the third-dimensional feature information, the second decoding result and the third decoding result, and determines thirteenth feature information after splicing; the thirteenth feature information is convolved twice and activated by a ReLU function, and fourteenth feature information after convolution is determined; the fourteenth feature information is input to a self-attention mechanism unit, and a fifth decoding result output by the self-attention mechanism unit is obtained;

[0169] The second up-sampling result corresponding to the second-dimensional feature information, the first decoding result, the fourth decoding result and the fifth decoding result is input to a sixth decoding unit, the sixth decoding unit performs splicing on the result after convolution of the second up-sampling result corresponding to the second-dimensional feature information, the first decoding result, the fourth decoding result and the fifth decoding result, and determines fifteenth feature information after splicing; the fifteenth feature information is convolved twice and activated by a ReLU function, and sixteenth feature information after convolution is determined; the sixteenth feature information is input to a self-attention mechanism unit, and a sixth decoding result output by the self-attention mechanism unit is obtained; and at least one decoding result output by the decoder is obtained;

[0170] 5) input the first up-sampling result corresponding to the edge feature, the first dimension feature information and the second dimension feature information into the first edge constraint unit, splice the result after convolution of the first up-sampling result corresponding to the edge feature, the first dimension feature information and the second dimension feature information by the first edge constraint unit, and obtain the spliced second feature information; perform twice convolution and ReLU function activation on the spliced second feature information to determine the third feature information after convolution; input the third feature information into the self-attention mechanism unit to obtain the first fusion feature information output by the self-attention mechanism unit;

[0171] input the second fusion feature information corresponding to the edge feature and all edge constraint units before the non-first edge constraint unit and the second up-sampling result corresponding to the decoding result output by the decoder into the non-first edge constraint unit, splice the result after convolution of the second fusion feature information corresponding to the edge feature and all edge constraint units before the non-first edge constraint unit and the second up-sampling result corresponding to the decoding result output by the decoder by the non-first edge constraint unit to determine the spliced fourth feature information; perform twice convolution and ReLU function activation on the spliced fourth feature information to determine the fifth feature information after convolution; input the fifth feature information into the self-attention mechanism unit to obtain the third fusion feature information output by the self-attention mechanism unit;

[0172] 6) take the result output by the last non-first edge constraint unit as the target extraction result of the image. It should be noted that the result output by the last non-first edge constraint unit is compared with the target threshold, and the pixel points corresponding to the fusion feature information greater than or equal to the target threshold are white, and the pixel points corresponding to the fusion feature information less than the target threshold are black, so as to obtain the target extraction result.

[0173] Since the multi-scale edge constraint model is obtained based on the sample image after training, and the sample image is obtained based on the initial sample image, the acquisition method of the sample image, i.e., the building-building provided by the present application is described.

[0174] Optionally, the sample image is obtained based on the initial sample image, comprising:

[0175] select at least one set of initial sample image pairs; the initial sample image pair comprises a background image and a copied image;

[0176] perform Fourier transform on each pixel point in the background image and the copied image respectively to obtain the transformed frequency domain image;

[0177] compare the frequency domain images corresponding to the background image and the copied image respectively;

[0178] In a case where the low-frequency pixel points in the frequency domain image corresponding to the background image are less than the low-frequency pixel points in the frequency domain image corresponding to the copied image, the low-frequency pixel points in the frequency domain image corresponding to the copied image are replaced with the low-frequency pixel points in the frequency domain image corresponding to the background image, and the background image after replacement is determined as the sample image.

[0179] In a case where the low-frequency pixel points in the frequency domain image corresponding to the background image are not less than the low-frequency pixel points in the frequency domain image corresponding to the copied image, the copied image is taken as the sample image.

[0180] Specifically, in order to enable the data enhancement method to play a greater role in building extraction, the application proposes a data enhancement method for buildings (build-building), at least one initial sample image pair is selected from initial sample images, wherein the initial sample image pair includes a background image and a copied image; the background image is represented as a pasted image, and the copied image is represented as a copied image.

[0181] In practice, fast Fourier transform (FFT) is performed on each pixel point in the background image and the copied image respectively to obtain the frequency domain image after transformation; the FFT transformation is performed by using formula (6) and formula (7) respectively, wherein:

[0182]

[0183]

[0184] Wherein, F copy (u,v) represents the frequency domain image corresponding to the copied image, f copy (x,y) represents the copied image, F paste (u,v) represents the frequency domain image corresponding to the background image, f paste (x,y) represents the background image, M represents the width of the copied image or the background image, N represents the height of the copied image or the background image, and u and v both represent the frequency.

[0185] After the spatial domain of the copied image and the background image is converted into the frequency domain, the places where the gray value changes sharply correspond to high frequencies, and vice versa. That is, the high-frequency component is mainly a measure of the image edge and contour, and the low-frequency component is mainly a comprehensive measure of the intensity of the entire image.

[0186] After performing FFT transformation on the background image and the copied image, the frequency domain images corresponding to the background image and the copied image are compared respectively; in the case that the low-frequency pixel points in the frequency domain image corresponding to the background image are less than the low-frequency pixel points in the frequency domain image corresponding to the copied image, the low-frequency pixel points in the frequency domain image corresponding to the copied image are replaced with the low-frequency pixel points in the frequency domain image corresponding to the background image, as shown in formulas (8) and (9), the low-frequency part of F paste (u,v) is replaced with the low-frequency part of F copy (u,v) to obtain F copy-paste (u,v), and then converted into the spatial domain f copy-paste (x,y) through iFFT, that is, the replaced background image is determined as the sample image; in the case that the low-frequency pixel points in the frequency domain image corresponding to the background image are not less than the low-frequency pixel points in the frequency domain image corresponding to the copied image, the copied image is taken as the sample image.

[0187]

[0188]

[0189] wherein, F copy-paste (u,v) represents the frequency domain image after replacement, F copy (u,v) represents the frequency domain image corresponding to the copied image, F paste (u,v) represents the frequency domain image corresponding to the background image, f copy-paste (x,y) represents the spatial domain image after iFFT transformation of F copy-paste (u,v), M represents the width of the copied image or the background image, N represents the height of the copied image or the background image, and u and v both represent frequency.

[0190] Optionally, the method further comprises:

[0191] performing operation on the copied image and the corresponding mask, and performing dilation operation on the operation result;

[0192] in the case that the background image and the copied image have an overlapping area, deleting the building in the overlapping area in the background image;

[0193] copying the low-frequency pixel points in the frequency domain image corresponding to the copied image to the background image to obtain the sample image.

[0194] It should be noted that when constructing the dataset of building images, the top of the building is generally taken as the ground truth value corresponding thereto, and if the pixels corresponding to the top of the building are directly copied, the neighborhood features important for building extraction, such as inclined walls or shadows, are ignored, and therefore, the corresponding mask is dilated before being copied and pasted.

[0195] Specifically, the copied image and the corresponding mask are operated, and the operation result is dilated, and when the copied image is copied and pasted, the situation that a building covers other buildings may occur due to the existence of overlapping areas of two or more buildings, the covering result does not conform to the actual situation, and the correct direction of model learning is affected. Therefore, in the case that the background image and the copied image have overlapping areas, the buildings in the overlapping areas in the background image are deleted; then the low-frequency pixel points in the frequency domain image corresponding to the copied image are copied to the background image to obtain a sample image.

[0196] Figure 6 is a result schematic diagram of the sample image acquisition method provided by the present application, as shown in Figure 6 (a) shows the copied image, including the positive sample copied image and the true result graph corresponding to the positive sample copied image, (b) shows the background image, including the positive sample background image and the true result graph corresponding to the positive sample background image, (c) shows the result graph obtained by directly copying and pasting (a) to (b), (d) shows the result graph obtained by copying and pasting after FFT transformation of (a) and (b), (e) shows the result graph obtained by copying and pasting after FFT transformation of (a) and (b), and operating the corresponding mask during copying and pasting, and dilating the operation result, (f) shows the result graph obtained by copying and pasting after deleting the buildings in the overlapping areas in the background image in the case that the background image and the copied image have overlapping areas, that is, bulid-buliding. According to the results shown in (c) to (f), it can be seen that the effect of copying and pasting in (f) is the best.

[0197] The method provided by the application is trained on an Nvidia GeForce RTX 3090 GPU, a model is built using a pytorch (Paszke et al., 2019) framework, the batch size of the model training is set to 8, an Adam (Adam) optimizer with weight decay is used as the optimizer, wherein the decay coefficient is 0.001; the learning rate is adjusted by cosine annealing, the initial learning rate is set to 0.001, and the minimum learning rate is 0.0001; the training set is subjected to data enhancement with a random probability of 0.5, the loss function uses the sum of the soft cross-entropy loss and the Dice loss, the maximum number of generations of the model training is 125, and in the training process, the model parameters corresponding to the validation set are saved, and the model parameters corresponding to the best generation number are used as the final model parameters.

[0198] In order to further illustrate the performance of the model provided by the application in extracting buildings, four commonly used precision indicators are used for precision evaluation, including precision, recall, F1 index and IoU; wherein, the precision refers to the proportion of pixels actually being buildings among the pixels being classified as buildings, the recall indicates how many building pixels are correctly classified as buildings, the F1 index indicates a comprehensive index of precision and recall, and the IoU is the ratio of the intersection to the union of the building area in the prediction result and the ground truth.

[0199] In order to better measure the extraction effect of the building boundary area, an additional IoU calculation is performed on the buffer area within the building boundary radius of 2 pixels. The calculation of each index is shown in formula (10)-formula (13):

[0200]

[0201]

[0202]

[0203]

[0204] Wherein, TP is a positive class with correct classification, FP is a negative class wrongly classified as a positive class, and FN is a positive class wrongly classified as a negative class.

[0205] Table 1. Precision index comparison table

[0206] Method Precision (%) Recall (%) F1(％) IoU (%) IoU (boundary) (%) PSP-Net 92.60 93.83 93.21 87.28 58.88 Res-U-Net 94.00 94.86 94.43 89.45 66.00 DeeplabV3+ 94.30 94.50 94.40 89.39 94.93 HRNet 94.67 94.64 94.66 89.86 67.52 MEC-Net 94.70 96.03 95.36 91.13 68.52

[0207] The multi-scale edge constraint model (MEC-Net) provided by the application and the existing pyramid scene parsing network (PSP-Net), residual U network (Res-U-Net), semantic segmentation network (DeeplabV3+) and HRNet network all obtain results as shown in Table 1 on the WHU building dataset. The performance of the MEC-Net provided by the application reaches high IoU and f1 on the WHU building dataset; wherein the IoU reaches 91.13%, and the f1 reaches 95.36%.

[0208] Figure 7 is a result schematic diagram of the target extraction method provided by the application, as shown in 7, the small building in the first row, Deeplabv3+ and HR-Net cause missing extraction, and MEC-Net can be well extracted; the features of the parking lot in the second row are similar to the buildings, and other models all have false extraction, and MEC-Net effectively distinguishes the parking lot and the building and other two ground objects; the super large building in the third row, the result of Deeplabv3+ extraction is very disordered, and MEC-Net successfully extracts the building. Therefore, using the MEC-Net provided by the target extraction method provided by the application, the most complete building extraction result and the most prominent boundary details can be obtained.

[0209] The target extraction device provided by the application is described below, and the target extraction device described below can be correspondingly referred to the target extraction method described above.

[0210] Figure 8 is a structural schematic diagram of the target extraction device provided by the application, as shown in Figure 8 the target extraction device 800 comprises an acquisition module 801, a determination module 802 and an extraction module 803, wherein:

[0211] The acquisition module 801 is used for acquiring an image of a target to be extracted.

[0212] The determination module 802 is used for determining an edge feature of the image.

[0213] The extraction module 803 is used for inputting the image and the edge feature into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on training of a sample image; the target extraction result is used for representing a building in the image; and the sample image is obtained based on an initial sample image.

[0214] The target extraction device provided by the application comprises the following steps: obtaining an image of a target to be extracted; determining an edge feature of the image; inputting the image and the edge feature into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model. The method provided by the application performs deep fusion on the image of the target to be extracted and the edge feature corresponding to the image through the multi-scale edge constraint model, refines the boundary of the building in the image, and improves the precision of building extraction.

[0215] Optionally, the multi-scale edge constraint model comprises an encoder, a decoder and an edge constraint block; and the extraction module 803 is specifically configured to:

[0216] input the image into the encoder to obtain at least one first feature information of the image output by the encoder; and each first feature information is used to represent a feature of different dimensions corresponding to the image.

[0217] perform up-sampling on each first feature information to obtain at least one first up-sampling result after up-sampling;

[0218] input each first feature information and each first up-sampling result into the decoder to obtain at least one decoding result output by the decoder; and each decoding result is used to represent feature information of different resolutions corresponding to the image.

[0219] input a second up-sampling result corresponding to the edge feature and each decoding result into the edge constraint block to obtain a target extraction result of the image output by the edge constraint block.

[0220] Optionally, the encoder comprises at least one encoding unit connected in series; and the extraction module 803 is specifically configured to:

[0221] input the image into a first encoding unit to obtain first dimension feature information output by the first encoding unit;

[0222] perform down-sampling on the first dimension feature information to obtain a first down-sampling result after sampling;

[0223] input the first down-sampling result into a second encoding unit to obtain second dimension feature information output by the second encoding unit;

[0224] perform down-sampling on the second dimension feature information to obtain a second down-sampling result after sampling;

[0225] input the second down-sampling result into a third encoding unit to obtain third dimension feature information output by the third encoding unit;

[0226] down-sample the third dimension feature information to obtain a third down-sampling result after sampling;

[0227] input the third down-sampling result into a fourth encoding unit to obtain fourth dimension feature information output by the fourth encoding unit;

[0228] down-sample the fourth dimension feature information to obtain a fourth down-sampling result after sampling;

[0229] input the fourth down-sampling result into a fifth encoding unit to obtain fifth dimension feature information output by the fifth encoding unit.

[0230] Optionally, the decoder comprises at least one decoding unit; and the extraction module 803 is specifically configured to:

[0231] input the first up-sampling result corresponding to the second dimension feature information and the third dimension feature information into a first decoding unit to obtain a first decoding result output by the first decoding unit;

[0232] input the first up-sampling result corresponding to the third dimension feature information and the fourth dimension feature information into a second decoding unit to obtain a second decoding result output by the second decoding unit;

[0233] input the first up-sampling result corresponding to the fourth dimension feature information and the fifth dimension feature information into a third decoding unit to obtain a third decoding result output by the third decoding unit;

[0234] determine at least one decoding result output by the decoder based on the first decoding result, the second decoding result and the third decoding result.

[0235] Optionally, the edge constraint block comprises at least one edge constraint unit connected in sequence; and the extraction module 803 is specifically configured to:

[0236] in a case where the edge constraint unit is a first edge constraint unit, input the first up-sampling result corresponding to the first dimension feature information and the second dimension feature information and the edge feature into the first edge constraint unit to obtain first fusion feature information output by the first edge constraint unit; the first fusion feature information is used to represent fusion of the first up-sampling result corresponding to the first dimension feature information and the second dimension feature information and the edge feature;

[0237] In the case that the edge constraint unit is a non-first edge constraint unit, the edge feature, second fusion feature information corresponding to all edge constraint units before the non-first edge constraint unit, and second up-sampling results corresponding to the decoding result output by the decoder are input into the non-first edge constraint unit to obtain third fusion feature information output by the non-first edge constraint unit; and a result output by a last edge constraint unit is taken as the target extraction result of the image.

[0238] Optionally, the extraction module 803 is specifically configured to:

[0239] concatenate the first dimension feature information, first up-sampling results corresponding to the second dimension feature information, and the edge feature to determine second feature information after concatenation;

[0240] perform twice convolution on the second feature information after concatenation to determine third feature information after convolution;

[0241] input the third feature information into the self-attention mechanism unit to obtain first fusion feature information output by the self-attention mechanism unit.

[0242] Optionally, the extraction module 803 is specifically configured to:

[0243] concatenate the edge feature, second fusion feature information corresponding to all edge constraint units before the non-first edge constraint unit, and second up-sampling results corresponding to the decoding result output by the decoder to determine fourth feature information after concatenation;

[0244] perform twice convolution on the fourth feature information after concatenation to determine fifth feature information after convolution;

[0245] input the fifth feature information into the self-attention mechanism unit to obtain third fusion feature information output by the self-attention mechanism unit.

[0246] Optionally, the decoding unit includes a self-attention mechanism unit; and the extraction module 803 is specifically configured to:

[0247] concatenate first up-sampling results corresponding to the second dimension feature information and the third dimension feature information to determine sixth feature information after concatenation;

[0248] perform twice convolution on the sixth feature information to determine seventh feature information after convolution;

[0249] input the seventh feature information into the self-attention mechanism unit to obtain first decoding result output by the self-attention mechanism unit.

[0250] Optionally, the extraction module 803 is further configured to:

[0251] select at least one set of initial sample image pairs; the initial sample image pairs include a background image and a copied image;

[0252] perform Fourier transform on each pixel point in the background image and the copied image respectively to obtain a frequency domain image after transformation;

[0253] compare the frequency domain images corresponding to the background image and the copied image respectively;

[0254] in a case where the low-frequency pixel points in the frequency domain image corresponding to the background image are less than the low-frequency pixel points in the frequency domain image corresponding to the copied image, replace the low-frequency pixel points in the frequency domain image corresponding to the copied image with the low-frequency pixel points in the frequency domain image corresponding to the background image, and determine the background image after replacement as a sample image;

[0255] in a case where the low-frequency pixel points in the frequency domain image corresponding to the background image are not less than the low-frequency pixel points in the frequency domain image corresponding to the copied image, determine the copied image as a sample image.

[0256] Optionally, the target extraction device 800 further includes:

[0257] perform operation on the copied image and the corresponding mask, and perform dilation operation on the operation result;

[0258] in a case where the background image and the copied image have an overlapping area, delete the building in the overlapping area in the background image;

[0259] copy the low-frequency pixel points in the frequency domain image corresponding to the copied image to the background image to obtain the sample image.

[0260] Figure 9 is a schematic diagram of an entity structure of an electronic device provided by the present application, such as Figure 9As shown, the electronic device 900 can include a processor 910, a communications interface 920, a memory 930, and a communications bus 940, wherein the processor 910, the communications interface 920, and the memory 930 complete mutual communication through the communications bus 940. The processor 910 can invoke a logic instruction in the memory 930 to execute a target extraction method, which includes: acquiring an image of a target to be extracted; determining an edge feature of the image; inputting the image and the edge feature into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on a sample image after training; the target extraction result is used to represent a building in the image; and the sample image is obtained based on an initial sample image.

[0261] In addition, the logic instruction in the memory 930 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0262] On the other hand, the present application also provides a computer program product, which includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, so that the computer can execute the target extraction method provided by the above-mentioned methods, which includes: acquiring an image of a target to be extracted; determining an edge feature of the image; inputting the image and the edge feature into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on a sample image after training; the target extraction result is used to represent a building in the image; and the sample image is obtained based on an initial sample image.

[0263] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the target extraction method provided by the above method, and the method comprises: acquiring an image of a target to be extracted; determining an edge feature of the image; inputting the image and the edge feature into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on training of a sample image; the target extraction result is used to represent a building in the image; and the sample image is obtained based on an initial sample image.

[0264] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0265] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method described in each embodiment or some part of the embodiment.

[0266] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A target extraction method characterized by, The method comprises: acquiring an image of a target to be extracted; determining edge features of the image; inputting the image and the edge features into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on training of sample images; the target extraction result is used to represent a building in the image; the sample images are obtained based on initial sample images; the multi-scale edge constraint model comprises an encoder, a decoder and an edge constraint block; the inputting of the image and the edge features into the multi-scale edge constraint model to obtain the target extraction result of the image output by the multi-scale edge constraint model comprises: inputting the image into the encoder to obtain at least one first feature information of the image output by the encoder; each first feature information is used to represent a feature of different dimensions corresponding to the image; performing up-sampling on each first feature information to obtain at least one first up-sampling result after up-sampling; inputting each first feature information and each first up-sampling result into the decoder to obtain at least one decoding result output by the decoder; each decoding result is used to represent feature information of different resolutions corresponding to the image; inputting the edge features and second up-sampling results corresponding to each decoding result into the edge constraint block to obtain the target extraction result of the image output by the edge constraint block; the sample images are obtained based on initial sample images, comprising: selecting at least one initial sample image pair; the initial sample image pair comprises a background image and a copied image; performing Fourier transform on each pixel point in the background image and the copied image respectively to obtain frequency domain images after transformation; comparing the frequency domain images corresponding to the background image and the copied image respectively; in a case where low-frequency pixel points in the frequency domain image corresponding to the background image are less than low-frequency pixel points in the frequency domain image corresponding to the copied image, replacing the low-frequency pixel points in the frequency domain image corresponding to the copied image with the low-frequency pixel points in the frequency domain image corresponding to the background image to determine the background image after replacement as a sample image; in a case where the low-frequency pixel points in the frequency domain image corresponding to the background image are not less than the low-frequency pixel points in the frequency domain image corresponding to the copied image, taking the copied image as a sample image.

2. The object extraction method according to claim 1, characterized in that, the encoder comprises at least one encoding unit connected in series; the inputting of the image into the encoder to obtain at least one first feature information of the image output by the encoder comprises: inputting the image into a first encoding unit to obtain first dimension feature information output by the first encoding unit; performing down-sampling on the first dimension feature information to obtain a first down-sampling result after sampling; inputting the first down-sampling result into a second encoding unit to obtain second dimension feature information output by the second encoding unit; performing down-sampling on the second dimension feature information to obtain a second down-sampling result after sampling; inputting the second down-sampling result into a third encoding unit to obtain third dimension feature information output by the third encoding unit; down-sampling the third dimension feature information to obtain a third down-sampling result after sampling; inputting the third down-sampling result into a fourth encoding unit to obtain fourth dimension feature information output by the fourth encoding unit; down-sampling the fourth dimension feature information to obtain a fourth down-sampling result after sampling; inputting the fourth down-sampling result into a fifth encoding unit to obtain fifth dimension feature information output by the fifth encoding unit.

3. The object extraction method according to claim 2, characterized in that, The decoder comprises at least one decoding unit; The inputting of the first feature information and the first up-sampling result into the decoder to obtain at least one decoding result output by the decoder comprises: inputting the second dimension feature information and the third dimension feature information corresponding first up-sampling result into a first decoding unit to obtain a first decoding result output by the first decoding unit; inputting the third dimension feature information and the fourth dimension feature information corresponding first up-sampling result into a second decoding unit to obtain a second decoding result output by the second decoding unit; inputting the fourth dimension feature information and the fifth dimension feature information corresponding first up-sampling result into a third decoding unit to obtain a third decoding result output by the third decoding unit; determining at least one decoding result output by the decoder based on the first decoding result, the second decoding result and the third decoding result.

4. The object extraction method according to claim 2, characterized in that, The edge constraint block comprises at least one edge constraint unit connected in sequence; The inputting of the edge feature and the second up-sampling result corresponding to the decoding result into the edge constraint block to obtain the target extraction result of the image output by the edge constraint block comprises: in the case that the edge constraint unit is a first edge constraint unit, inputting the first dimension feature information, the second dimension feature information corresponding first up-sampling result and the edge feature into the first edge constraint unit to obtain first fusion feature information output by the first edge constraint unit; the first fusion feature information is used to represent the fusion of the first dimension feature information, the second dimension feature information corresponding first up-sampling result and the edge feature; in the case that the edge constraint unit is a non-first edge constraint unit, inputting the edge feature, the second fusion feature information corresponding to all edge constraint units before the non-first edge constraint unit respectively and the second up-sampling result corresponding to the decoding result output by the decoder into the non-first edge constraint unit to obtain third fusion feature information output by the non-first edge constraint unit; wherein the result output by the last edge constraint unit is taken as the target extraction result of the image.

5. The object extraction method according to claim 4, characterized in that, The inputting of the first dimension feature information, the second dimension feature information corresponding first up-sampling result and the edge feature into the first edge constraint unit to obtain first fusion feature information output by the first edge constraint unit comprises: The first dimension feature information, the first up-sampling result corresponding to the second dimension feature information, and the edge feature are spliced to determine second feature information after splicing; The second feature information after splicing is convolved twice to determine third feature information after convolution; The third feature information is input into a self-attention mechanism unit to obtain first fusion feature information output by the self-attention mechanism unit.

6. The object extraction method according to claim 4, characterized in that, The edge feature, second fusion feature information corresponding to all edge constraint units before the non-first edge constraint unit, and second up-sampling result corresponding to the decoding result output by the decoder are input into the non-first edge constraint unit to obtain third fusion feature information output by the non-first edge constraint unit, including: The edge feature, second fusion feature information corresponding to all edge constraint units before the non-first edge constraint unit, and second up-sampling result corresponding to the decoding result output by the decoder are spliced to determine fourth feature information after splicing; The fourth feature information after splicing is convolved twice to determine fifth feature information after convolution; The fifth feature information is input into a self-attention mechanism unit to obtain third fusion feature information output by the self-attention mechanism unit.

7. The object extraction method according to claim 3, characterized by, The decoding unit includes a self-attention mechanism unit; The first up-sampling result corresponding to the second dimension feature information and the third dimension feature information is input into a first decoding unit to obtain first decoding result output by the first decoding unit, including: The first up-sampling result corresponding to the second dimension feature information and the third dimension feature information is spliced to determine sixth feature information after splicing; The sixth feature information is convolved twice to determine seventh feature information after convolution; The seventh feature information is input into the self-attention mechanism unit to obtain first decoding result output by the self-attention mechanism unit.

8. The object extraction method of claim 1, wherein, The method further includes: Performing an operation on the copied image and the corresponding mask, and performing a dilation operation on the operation result; In the case where the background image and the copied image have an overlapping region, deleting the building in the overlapping region in the background image; Copying the low-frequency pixel points in the frequency domain image corresponding to the copied image to the background image to obtain the sample image.

9. An object extraction apparatus characterized by comprising: The device includes: An acquisition module configured to acquire an image of a target to be extracted; A determination module configured to determine an edge feature of the image; An extraction module configured to input the image and the edge feature into a multi-scale edge constraint model to obtain a target extraction result of the image output by the multi-scale edge constraint model; the multi-scale edge constraint model is obtained based on a sample image; the target extraction result is used to represent a building in the image; and the sample image is obtained based on an initial sample image; The multi-scale edge constraint model includes an encoder, a decoder, and an edge constraint block; and the extraction module is specifically configured to: inputting the image into the encoder to obtain at least one first feature information of the image output by the encoder, each of the first feature information being used to represent a feature of different dimensions corresponding to the image; respectively performing up-sampling on each of the first feature information to obtain at least one first up-sampling result after up-sampling; inputting each of the first feature information and each of the first up-sampling result into the decoder to obtain at least one decoding result output by the decoder, each of the decoding result being used to represent a feature information of different resolutions corresponding to the image; inputting the edge feature and each of the second up-sampling result corresponding to each of the decoding result into the edge constraint block to obtain a target extraction result of the image output by the edge constraint block; selecting at least one initial sample image pair, the initial sample image pair including a background image and a copied image; respectively performing Fourier transform on each of the pixels in the background image and the copied image to obtain a frequency domain image after transformation; respectively comparing the frequency domain images corresponding to the background image and the copied image; in a case where the low-frequency pixel points in the frequency domain image corresponding to the background image are less than the low-frequency pixel points in the frequency domain image corresponding to the copied image, replacing the low-frequency pixel points in the frequency domain image corresponding to the copied image with the low-frequency pixel points in the frequency domain image corresponding to the background image, and determining the background image after replacement as a sample image; in a case where the low-frequency pixel points in the frequency domain image corresponding to the background image are not less than the low-frequency pixel points in the frequency domain image corresponding to the copied image, determining the copied image as a sample image.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the target extraction method according to any one of claims 1 to 8.

11. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the target extraction method according to any one of claims 1 to 8.

12. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the target extraction method according to any one of claims 1 to 8.

Images