Farmland parcel extraction method based on multi-type double-branch convolutional neural network

Abstract

The invention provides a farmland plot extraction method based on a multi-type double-branch convolutional neural network, and the method comprises the steps: creating a double-branch multi-task network DBMNet based on a multi-type convolution module, and designing a double-branch feature extraction module DBFE to replace a convolution layer of an original network, a dynamic linear constraint convolution module LCDC and a multi-scale deformable convolution feature extraction module MDFE are used to comprehensively extract edge features of an image and field block region features of different sizes and shapes, the perception ability of a network to information of different scales is improved, and thus the segmentation precision of the network is improved. The network uses boundary detection and distance regression tasks to improve the precision extraction of the farmland plot masks, so that the network has a better generalization effect. According to the method, the multi-type convolutional network is applied to the farmland plot extraction task, the experimental result shows that compared with other most advanced algorithms, the method has the best segmentation performance, discriminative features of the farmland plot can be effectively extracted, the segmentation precision is remarkably improved, the farmland plot is accurately extracted, and the method has great application potential and prospect.

Description

Technical Field

[0001] The present invention relates to the field of remote sensing image farmland plot segmentation, and in particular to a method for extracting farmland plot information through a deep convolutional network, constructing a multi-type convolutional network through an improved multi-scale deformable convolution module and a dynamic linear constraint convolution module, comprehensively and fully extracting edge features and regional features of remote sensing image farmland plots, and accurately extracting remote sensing image farmland plots. Background Art

[0002] Arable land is the material basis for social development and human survival, and agricultural land is the basic unit for agricultural practice and application.

[0003] In such a large agricultural production context, using scientific and technological means to supervise and assist the transformation, development and production links of farmland will be a very important way to improve efficiency and increase production.

[0004] With the development of science and technology, there are more and more ways to obtain field boundaries, and remote sensing technology is widely popular as one of the most convenient and quickest means. Remote sensing is a non-contact method of obtaining information about distant targets, mainly using artificial satellites or aircraft, drones and other equipment to collect data. Remote sensing equipment can transmit and receive specific electromagnetic wave information of ground targets at high altitudes, and analyze and apply it. Remote sensing images have the characteristics of wide range, dynamic monitoring and real-time performance. Therefore, applying remote sensing technology to agricultural production can quickly and accurately understand the distribution status, area size and spatial geographical location information of farmland plots, and provide important basic data for land supervision and agricultural production.

[0005] With the continuous development of ground remote sensing observation platforms, it is no longer a problem to repeatedly obtain high-definition ground images in a short period of time, but it is still a difficult problem to extract valuable plot information from these high-quality images. The early remote sensing image farmland plot extraction relied on manual experience to extract and classify information, which was inefficient and consumed a lot of manpower and material resources. In addition, there are rich types of crops planted on cultivated land, and there are large differences in the cultivation methods and maturity periods of different crops. There are also uncultivated fallow cultivated land. These factors make cultivated land of different crop types show large feature differences in remote sensing images, and the phenomenon of different spectra of the same object is relatively serious, resulting in low accuracy of manual extraction methods. Therefore, in the face of the huge amount of information and complex image features of remote sensing images, it is of great significance to realize the automatic extraction of farmland plots in remote sensing images with the help of computer technology. The automation of this process can reduce manual participation and improve the accuracy of plot segmentation.

[0006] Machine learning methods are widely used methods for automatic classification and extraction of remote sensing images. Traditional machine learning methods work well in extracting cultivated land from low- and medium-resolution remote sensing images. However, this method requires manual participation in parameter selection and feature selection, and can only extract shallow features. Therefore, for high-resolution remote sensing images containing complex land object information, traditional methods cannot effectively extract the deep features of various land objects, and cannot accurately extract farmland plots. With the continuous development of digital agriculture and the enhancement of remote sensing technology. The remote sensing images collected by satellites are clearer, with richer color, contour and texture features, and even contain a large number of deep features such as spatial location features and information entropy change features. Therefore, the development of refined agriculture and smart agriculture has higher and higher requirements for the data quality of plot information. The more complex and huge remote sensing image information sources and the more stringent and high-standard plot information requirements make it an urgent problem to efficiently and accurately extract farmland plots from high-resolution images.

[0007] Applying deep learning technology to the task of semantic segmentation of remote sensing image plots can identify and classify plots at the pixel level. While realizing the automatic segmentation of farmland plots in remote sensing images, the accuracy and processing speed are improved, and the visual segmentation result image is directly output. In recent years, with the development of deep learning, the Convolutional Neural Network (CNN) technology has achieved good results in the field of image processing. CNN can automatically learn and extract features in images and has good robustness. Therefore, more and more scholars use CNN for high-resolution remote sensing image classification and extraction. However, when CNN is directly transferred from natural images to remote sensing images, it is found that compared with natural images, remote sensing images contain more band data such as near infrared and short-wave infrared in addition to red, green, and blue band data. Therefore, how to give full play to the rich information advantages of remote sensing images is of great research significance. In addition, the sizes and shapes of farmland plots in remote sensing images vary greatly, and the features vary greatly. However, most of the existing farmland plot extraction methods based on CNN use traditional convolutional features of fixed size to extract features, which cannot fully capture the diverse feature information of farmland plots. Therefore, it is crucial to design a farmland plot extraction method based on convolutional neural networks that can more efficiently process complex high-resolution remote sensing images, extract complex and diverse farmland plot features, and significantly improve the plot extraction accuracy. Summary of the invention

[0008] The purpose of the present invention is to solve the problem of single feature extraction in existing methods. A dual-branch multi-task network DBMNet based on multi-type convolution modules is proposed. The network uses multi-type convolution to comprehensively and fully extract the diverse features of farmland plots in remote sensing images, thereby improving the segmentation effect of high-resolution remote sensing plot images.

[0009] The technical solution of the present invention provides a method for extracting farmland plots based on a multi-type dual-branch convolutional neural network, comprising the following steps:

[0010] Step 1: Construct a dual-branch multi-task network DBMNet based on multi-type convolutional modules. Take the U-Net network as the baseline network and design a dual-branch feature extraction module DBFE to replace the convolutional layer of the original network, so that the network can comprehensively extract the edge features of the image and the regional features of fields of different sizes and shapes, increasing the network depth while improving the network's perception of information at different scales.

[0011] Step 2: For the DBFE module in step 1, design edge feature extraction branches and regional feature extraction branches. For edge feature extraction, design the dynamic linear constraint convolution module LCDC to enhance the network's perception of slender linear structures and adaptively focus on slender and tortuous local structures to accurately capture the edge features of field boundaries. For regional feature extraction, design the multi-scale deformable convolution feature extraction module MDFE to perform deformable convolution operations on input data at different scales, adaptively focus on the shape structure of field plots, and accurately capture the regional feature information of field plots of different sizes and shapes;

[0012] In step 3, the network finally outputs the boundary, mask and boundary distance of the field block respectively, among which boundary detection and distance regression are used to improve the accuracy of field block mask extraction, so that the network has a better generalization effect.

[0013] Furthermore, in step 1, the multi-band input data extracted from the remote sensing image Where H×W is the size of the input sample, B is the number of band channels of the sample; the data is processed by the first layer of 3×3 convolution module and becomes H1, where It represents the feature map output by the i-th layer of the network, h×w is the size of the feature map, and b is the number of channels of the feature map. H1 is taken as the input data of the first layer of DBFE and is further encoded to obtain H2, and so on until the last layer. The stacked DBFE module extracts multi-scale, shallow-to-deep deep farmland plot features from the original multi-band remote sensing image data and outputs them.

[0014] Further, the stacked DBFE module includes m DBFE modules.

[0015] Furthermore, in step 2, the input data H of each layer of the DBFE module i , i=1,2,...,m are input into two different branches to extract edge features and regional features of farmland plots, respectively, where m is the number of DBFE modules. The edge feature extraction branch of DBFE consists of an LCDC module and a traditional 3×3 convolution, and the output is H i+1,i=1,2,...,m. For the LCDC module, it is a deformable convolution with arbitrary parameters N. The convolution regards the point in the upper left corner as the sampling origin (0,0). Let R represent the sampling grid, and R is expressed as follows:

[0016] R={(-1,-1),(-1,0),...,(0,1),(1,1)} (1)

[0017] The initial sampling position p of the convolution n It can be customized or randomly generated. p0 is the output image h i,1 The value of each pixel in the output image is calculated by the following formula:

[0018]

[0019] Among them, w(p n ) indicates p n The size of the convolution kernel weight of the point. n +offset n ) represents p0+p n +offset n The eigenvalue of the point, offset n is the offset, but if the model learns the deformation offset completely freely, the receptive field will often deviate from the target, especially when dealing with slender linear structures. Therefore, a convolution kernel of size k is considered, with the point in the upper left corner as the sampling origin C0, and the remaining k-1 sampling points adopt an iterative strategy when offsetting: each C i , i=1,2,...,k-1 points are relative to their previous point C i-1 ,i=1,2,...,k-1 adds an offset Δ={δ|δ∈[-1,1]}, and the offsets are accumulated in sequence to ensure that the convolution kernel conforms to the linear morphological structure. The formula is as follows:

[0020]

[0021] Since the offset is usually a decimal, bilinear interpolation is used to calculate the pixel value of the offset point. The formula is as follows:

[0022]

[0023] Where q represents the point on the feature map with the nearest four integers around point p, x(q) represents the pixel value of point q, and x(p) represents the value calculated after the offset point p, p = p0 + p n +offset n .

[0024] Furthermore, in step 2, the regional feature extraction branch of DBFE is composed of MDFE modules. i , i = 1, 2, ..., m, and evenly divide it into s feature map subsets h according to the channel i,j ,j=1,2,...,s. Except h i,1 Without going through the convolutional layer, it is a quick connection. i,j ,j=2,3,...,s all have corresponding 5×5 deformable convolutional layers, using D i,j (·),j=2,3,...,s means, D i,j (·) is composed of 3×3 deformable convolution, BatchNormalization (BN) and ReLU activation function. i,j Indicates D i,j The output of each branch j is the output of the branch h. i,j Splice the output y of the previous branch i,j-1 After D i,j (·) is obtained, so y i,j can be written as:

[0025]

[0026] in, represents the concatenation operation, each 5×5 deformable convolution operator D i,j (·) It is possible to split h from all features i,j ,j≤i receives feature information, so the receptive field of each branch output result increases successively. This paper obtains larger convolution and receptive field by repeatedly stacking small-sized convolutions, and because two convolution layers use one more activation function than one convolution layer, the nonlinear ability is enhanced.

[0027] Furthermore, in step 3, the mask and boundary prediction tasks are binary classification tasks, so the negative log-likelihood (NLL) loss is used, while the distance map estimation task is a regression task and the mean square error (MSE) loss is used.

[0028] Let L cls is the pixel classification loss (i.e., NLL), x is the pixel position in the image space R, and the loss function for the mask and boundary prediction task for N input instances is defined as:

[0029]

[0030] Among them, P mc (x; l mc (x)) represents the true label l activated by SoftMax mc The predicted probability of .

[0031] In order to obtain the MSE loss for the distance map estimation task (i.e., L reg ), this paper uses the predicted distance Calculate the deviation from the actual distance D(x):

[0032]

[0033] For multiple tasks, the loss values ​​of different subtasks are summed up to get the total loss:

[0034] L total =w1·L cls1 +w2·L cls2 +w3·L reg (8)

[0035] Among them, L cls1 , L cls2 , L reg It refers to the loss values ​​of the three subtasks, corresponding to mask, boundary and distance prediction respectively, and w1, w2 and w3 are the corresponding weight coefficients.

[0036] Furthermore, the initial learning rate used in the training phase is 10 -2 and weight decay is 10 -4 The Apollo optimizer is used to train the mask prediction and distance map estimation tasks, and the initial learning rate is 10 -8 and weight decay is 10 -4 The stochastic gradient descent (SGD) optimizer is used to train the edge prediction task.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] The dual-branch multi-task network DBMNet based on multi-type convolutional modules proposed in the present invention is based on the original U-Net network, and a dual-branch feature extraction module is designed to replace the convolution layer of the original network, so that the network can comprehensively extract the edge features of the image and the regional features of the fields of different sizes and shapes, while increasing the network depth and improving the network's perception of information of different scales, thereby improving the network's segmentation accuracy. The network finally outputs the boundaries, masks and boundary distances of the fields, and boundary detection and distance regression are conducive to improving the accuracy of the mask extraction of the farmland plots, so that the network has a better generalization effect.

[0039] The dual-branch feature extraction module DBFE proposed in the present invention designs an edge feature extraction branch and a regional feature extraction branch. For the extraction of edge features, a dynamic linear constraint convolution module LCDC is designed to enhance the network's perception of slender linear structures, adaptively focus on slender and tortuous local structures, and accurately capture the edge features of field boundaries. For the extraction of regional features, a multi-scale deformable convolution feature extraction module MDFE is designed to perform deformable convolution operations on input data at different scales, adaptively focus on the shape structure of field plots, accurately capture regional feature information of field plots of different sizes and shapes, and improve the generalization ability of the model. Therefore, the farmland plot extraction method based on multi-type dual-branch convolutional neural network not only has very important academic value, but also has important practical significance and application potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is the overall network structure diagram of the present invention.

[0041] Figure 2 It is the structural diagram of the dynamic linear constrained convolution module LCDC.

[0042] Figure 3 It is a structural diagram of the multi-scale deformable convolutional feature extraction module MDFE. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention more clear, the following is a detailed description of the farmland plot extraction method based on a multi-type dual-branch convolutional neural network with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0044] The invention mainly aims at the problem that traditional convolutional neural networks usually only extract edge features or regional features of farmland plots, use a fixed-size receptive field, and lack the mining of diversified features of farmland plots. A dual-branch multi-task network DBMNet based on multi-type convolution modules is constructed, and a dual-branch feature extraction module DBFE is designed: a multi-scale deformable feature extraction module mines multi-scale information of farmland plots in remote sensing images, and adaptively focuses on the shape structure of the field plots to accurately capture the regional feature information of field plots of different sizes and shapes; a dynamic linear constraint convolution module enhances the network's perception of slender linear structures, adaptively focuses on slender and tortuous local structures, and accurately captures the edge features of the field boundary; a multi-task network framework is built: the network finally outputs the boundary, mask and boundary distance of the field plot respectively, wherein the boundary detection and distance regression are conducive to improving the precision extraction of the mask of the farmland plot, so that the network has a better generalization effect.

[0045] Step 1: Construct a dual branch multi-task network based on multi-type convolutional modules (Dual Branch Multi-type Convolutional Network, DBMNet), take the U-Net network as the baseline network, and design a dual branch feature extraction module (Dual Branch Feature Extraction Module, DBFE) to replace the convolutional layer of the original network, so that the network can comprehensively extract the edge features of the image and the regional features of fields of different sizes and shapes, increasing the network depth while improving the network's perception of information at different scales;

[0046] Step 2: Design edge feature extraction branches and regional feature extraction branches for the DBFE module in step 1. For edge feature extraction, design a Line Constrained Dynamic Convolution Module (LCDC) to enhance the network's perception of slender linear structures and adaptively focus on slender and tortuous local structures to accurately capture the edge features of field boundaries. For regional feature extraction, design a Multi-scale Deformable Feature Extraction Module (MDFE) to perform deformable convolution operations on input data at different scales, adaptively focus on the shape structure of field plots, and accurately capture regional feature information of field plots of different sizes and shapes.

[0047] In step 3, the network finally outputs the boundary, mask and boundary distance of the field block respectively, among which boundary detection and distance regression are used to improve the accuracy of field block mask extraction, so that the network has a better generalization effect.

[0048] Furthermore, in step 1, the multi-band input data extracted from the remote sensing image Where H×W is the size of the input sample, B is the number of band channels of the sample; the data is processed by the first layer of 3×3 convolution module and becomes H1, where It represents the feature map output by the i-th layer of the network, h×w is the size of the feature map, and b is the number of channels of the feature map. H1 is taken as the input data of the first layer of DBFE and is further encoded to obtain H2, and so on until the last layer. The stacked DBFE module extracts multi-scale, shallow-to-deep deep farmland plot features from the original multi-band remote sensing image data and outputs them.

[0049] Furthermore, in this embodiment, the stacked DBFE module includes m DBFE modules, and the input data H of each layer of the DBFE module i, i=1,2,...,m are input into two different branches to extract edge features and regional features of farmland plots, respectively, where m is the number of DBFE modules. The edge feature extraction branch of DBFE consists of an LCDC module and a traditional 3×3 convolution, and the output is H i+1 ,i=1,2,...,m. For the LCDC module, it is a deformable convolution with arbitrary parameters N. The convolution regards the point in the upper left corner as the sampling origin (0,0). Let R represent the sampling grid, and R is expressed as follows:

[0050] R={(-1,-1),(-1,0),...,(0,1),(1,1)} (1)

[0051] The initial sampling position p of the convolution n It can be customized or randomly generated. p0 is the output image h i,1 The value of each pixel in the output image is calculated by the following formula:

[0052]

[0053] Among them, w(p n ) indicates p n The size of the convolution kernel weight of the point. n +offset n ) represents p0+p n +offset n The eigenvalue of the point, offset n is the offset, but if the model learns the deformation offset completely freely, the receptive field will often deviate from the target, especially when dealing with slender linear structures. Therefore, a convolution kernel of size k is considered, with the point in the upper left corner as the sampling origin C0, and the remaining k-1 sampling points adopt an iterative strategy when offsetting: each C i , i=1,2,...,k-1 points are relative to their previous point C i-1 ,i=1,2,...,k-1 adds an offset Δ={δ|δ∈[-1,1]}, and the offsets are accumulated in sequence to ensure that the convolution kernel conforms to the linear morphological structure. The formula is as follows:

[0054]

[0055] Since the offset is usually a decimal, bilinear interpolation is used to calculate the pixel value of the offset point. The formula is as follows:

[0056]

[0057] Where q represents the point on the feature map with the nearest four integers around point p, x(q) represents the pixel value of point q, and x(p) represents the value calculated after the offset point p, p = p0 + p n +offset n .

[0058] Furthermore, the regional feature extraction branch of DBFE is composed of MDFE modules. i , i = 1, 2, ..., m, and evenly divide it into s feature map subsets h according to the channel i,j ,j=1,2,...,s. Except h i,1 Without going through the convolutional layer, it is a quick connection. i,j ,j=2,3,...,s all have corresponding 3×3 deformable convolutional layers, using D i,j (·),j=2,3,...,s means, D i,j (·) is composed of 3×3 deformable convolution, BatchNormalization (BN) and ReLU activation function. i,j Indicates D i,j The output of each branch j is the output of the branch h. i,j Splice the output y of the previous branch i,j-1 After D i,j (·) is obtained, so y i,j can be written as:

[0059]

[0060] in, represents the concatenation operation, each 5×5 deformable convolution operator D i,j (·) It is possible to split h from all features i,j ,j≤i receives feature information, so the receptive field of each branch output result increases successively. This paper obtains larger convolution and receptive field by repeatedly stacking small-sized convolutions, and because two convolution layers use one more activation function than one convolution layer, the nonlinear ability is enhanced.

[0061] Furthermore, in step 3, the mask and boundary prediction tasks are binary classification tasks, so the negative log-likelihood (NLL) loss is used, while the distance map estimation task is a regression task and the mean square error (MSE) loss is used.

[0062] Let L cls is the pixel classification loss (i.e., NLL), x is the pixel position in the image space R, and the loss function for the mask and boundary prediction task for N input instances is defined as:

[0063]

[0064] Among them, P mc (x; l mc (x)) represents the true label l activated by SoftMax mc The predicted probability of .

[0065] In order to obtain the MSE loss for the distance map estimation task (i.e., L reg ), this paper uses the predicted distance Calculate the deviation from the actual distance D(x):

[0066]

[0067] For multiple tasks, the loss values ​​of different subtasks are summed up to get the total loss:

[0068] L total =w1·L cls1 +w2·L cls2 +w3·L reg (8)

[0069] Among them, L cls1 , L cls2 , L reg Refers to the loss values ​​of the three subtasks, corresponding to mask, boundary and distance prediction, respectively. w1, w2 and w3 are the corresponding weight coefficients. The initial learning rate used in the training phase is 10 -2 and weight decay is 10 -4 The Apollo optimizer is used to train the mask prediction and distance map estimation tasks, and the initial learning rate is 10 -8 and weight decay is 10 -4 The stochastic gradient descent (SGD) optimizer is used to train the edge prediction task.

[0070] The present invention can be implemented by computer software technology. Figure 1 , Figure 2 and Figure 3 The specific steps for implementing the farmland plot extraction method based on multi-type dual-branch convolutional neural network are described in detail.

[0071] Step 1: Construct a dual branch multi-task network based on multi-type convolutional modules (Dual Branch Multi-type Convolutional Network, DBMNet), take the U-Net network as the baseline network, and design a dual branch feature extraction module (Dual Branch Feature Extraction Module, DBFE) to replace the convolutional layer of the original network, so that the network can comprehensively extract the edge features of the image and the regional features of fields of different sizes and shapes, increasing the network depth while improving the network's perception of information at different scales;

[0072] In the example, Figure 1As shown in the network diagram, DBMNet consists of two parts, the encoder part and the decoder part. The encoder part consists of two layers of traditional 3×3 convolution blocks and two layers of DBFE modules, and the decoder part consists of two layers of traditional 3×3 convolution blocks and one layer of DBFE modules. The input data size of the network is (4, 320, 320), each sample contains 4 channels of size (320, 320), and after the first layer of 3×3 convolution processing in the encoder, the output feature map size is (32, 320, 320), that is, the number of channels becomes 32. The feature map output by the first layer is further used as the input of the second-layer DBFE module, and the new feature map output size is (64, 320, 320). After the 2×2 maximum pooling operation, the output feature map is (64, 160, 160). This operation is repeated until the fourth-layer DBFE module, and the output feature map is (256, 80, 80). After the 2×2 maximum pooling operation, the output feature map is (256, 40, 40). After that, the output feature map obtained by the 2×2 upsampling operation is (256, 80, 80), which is concatenated with the output feature map of the third 3×3 convolutional layer in the encoder to obtain a feature map of (384, 80, 80), which is used as the input of the first 3×3 convolutional layer in the decoder, and the new feature map output size is (128, 80, 80), and then it is subjected to a 2×2 upsampling operation, and the output feature map is (128, 160, 160), which is concatenated with the output feature map of the second layer DBFE module in the encoder to obtain a feature map of (192, 160, 160), which is used as the input of the second layer DBFE module in the decoder to obtain a new feature map output size of (64, 160, 160), and this operation is repeated until the third layer DBFE module in the decoder, and the output feature map is (32, 320, 320). Then, 1×1 convolution is used to obtain the output feature maps corresponding to the three tasks of mask, boundary prediction, and distance regression. The feature map size corresponding to mask and boundary prediction is (2, 320, 320), which is passed through the softmax function to obtain the predicted label. The feature map size corresponding to the distance regression task is (1, 320, 320), which is passed through the sigmoid function to obtain the regression result. The initial learning rate used in the training phase is 10 -2 and weight decay is 10 -4 The Apollo optimizer is used to train the mask prediction and distance map estimation tasks, and the initial learning rate is 10 -8 and weight decay is 10 -4 The stochastic gradient descent (SGD) optimizer is used to train the edge prediction task. The training batch is selected as 2 and the training lasts for 150 iterations.

[0073] Step 2: The DBFE module in step 1 consists of two branches, where the edge feature extraction branch consists of a dynamic linear constrained convolution module (Line Constrained Dynamic Convolution Module, LCDC) and a traditional 3×3 convolution module, and the region feature extraction branch consists of a multi-scale deformable convolution feature extraction module (Multi-scale Deformable Feature Extraction Module, MDFE). The input feature map of the DBFE module is input into the two branches respectively, and features are extracted by the LCDC module and the MDFE module respectively. Finally, the output feature maps of the two branches are added to obtain the output result of the DBFE module.

[0074] In the example, Figure 2 As shown in the structure diagram, the LCDC module is used to enhance the network's perception of slender linear structures, adaptively focusing on slender and tortuous local structures to accurately capture the edge features of the field boundary. The LCDC module is a deformable convolution with a parameter of 9. The convolution regards the point in the upper left corner as the sampling origin (0, 0), and the initial sampling position of the convolution is p n is set to be along the diagonal position, i.e. p n ={(0,0),(1,1),...,(8,8)}. Consider a convolution kernel of size 9, with the upper left corner as the sampling origin C0, and the remaining 8 sampling points adopt an iterative strategy when offsetting: each C i , i=1,2,..., all 8 points are relative to their previous point C i-1 ,i=1,2,...,8 adds an offset Δ={δδ∈[-1,1]}, that is, each point can be offset within a 3×3 range around its previous point, and the offsets are accumulated in sequence to ensure that the convolution kernel conforms to the linear morphological structure. Since the offset is usually a decimal, bilinear interpolation is used to calculate the pixel value of the offset point.

[0075] In the example, Figure 3 As shown in the structure diagram, MDFE is used to perform deformable convolution operations on the input data at different scales, adaptively focusing on the shape structure of the field blocks, and accurately capturing the regional feature information of field blocks of different sizes and shapes. The MDFE module divides the input data H into 4 feature map subsets h evenly by channel i , i = 1, 2, 3, 4. Except h1 which does not pass through the convolution layer and is a shortcut connection, the other h i , i=2,3,4 all have corresponding 5×5 deformable convolutional layers. i represents the output of each branch, then the output y i , i=3,4 are all input h of this branch iSplice the output y of the previous branch i After that, it is obtained through a 5×5 deformable convolution layer. Finally, the outputs of the four branches are concatenated to obtain the final output y of the MDFE module

[0076] In this example, the loss function weight coefficients of the three tasks are all 1, and the initial learning rate of the mask prediction and distance map estimation tasks is 10 -2 , using a weight decay of 10 -4 The Apollo optimizer is trained with an initial learning rate of 10 for edge prediction tasks. -8 , using a weight decay of 10 -4 The training batch size is 2 and the training lasts for 150 iterations.

[0077] Those skilled in the art can understand that the present invention looks at the cross-regional crop mapping problem from the perspective of deep feature distribution for the first time, effectively learns the feature distribution of crop samples in different phenological regions without using target domain labels, effectively aligns the deep phenological features corresponding to crop samples in different regions, makes the invalid deep model discriminative again, significantly improves the mapping accuracy, and has the potential to embed multi-scale feature alignment modules into other deep networks for cross-phenological crop mapping.

[0078] It should be noted and understood that various modifications and improvements can be made to the present invention described in detail above without departing from the spirit and scope of the present invention required by the appended claims. Therefore, the scope of the technical solution claimed is not limited by any specific exemplary teaching given.

Claims

1. A method for extracting farmland plots based on a multi-type dual-branch convolutional neural network, characterized in that: The following steps are involved: Step 1: Construct a dual-branch multi-task network DBMNet based on multi-type convolutional modules. Take the U-Net network as the baseline network and design a dual-branch feature extraction module DBFE to replace the convolutional layer of the original network, so that the network can comprehensively extract the edge features of the image and the regional features of fields of different sizes and shapes, increasing the network depth while improving the network's perception of information at different scales. Step 2: For the DBFE module in step 1, design edge feature extraction branches and regional feature extraction branches. For edge feature extraction, design the dynamic linear constraint convolution module LCDC to enhance the network's perception of slender linear structures and adaptively focus on slender and tortuous local structures to accurately capture the edge features of field boundaries. For regional feature extraction, design the multi-scale deformable convolution feature extraction module MDFE to perform deformable convolution operations on input data at different scales, adaptively focus on the shape structure of field plots, and accurately capture the regional feature information of field plots of different sizes and shapes; In step 3, the network finally outputs the boundary, mask and boundary distance of the field block respectively, among which boundary detection and distance regression are used to improve the accuracy of field block mask extraction, so that the network has a better generalization effect.

2. The method for extracting farmland plots based on a multi-type dual-branch convolutional neural network as claimed in claim 1, characterized in that: In step 1, the multi-band input data extracted from the remote sensing image Where H×W is the size of the input sample, B is the number of band channels of the sample; the data is processed by the first layer of 3×3 convolution module and becomes H1, where It represents the feature map output by the i-th layer of the network, h×w is the size of the feature map, and b is the number of channels of the feature map. H1 is taken as the input data of the first layer of DBFE and is further encoded to obtain H2, and so on until the last layer. The stacked DBFE module extracts multi-scale, shallow-to-deep deep farmland plot features from the original multi-band remote sensing image data and outputs them.

3. The method for extracting farmland plots based on a multi-type dual-branch convolutional neural network as claimed in claim 1, characterized in that: The stacked DBFE module includes m DBFE modules.

4. The method for extracting farmland plots based on a multi-type dual-branch convolutional neural network as claimed in claim 1, characterized in that: In step 2, the input data H of each layer of the DBFE module i , i=1,2,...,m are input into two different branches to extract edge features and regional features of farmland plots, respectively, where m is the number of DBFE modules. The edge feature extraction branch of DBFE consists of an LCDC module and a traditional 3×3 convolution, and the output is H i+1 =h i,1 +h i,2 ,i=1,2,...,m. For the LCDC module, it is a deformable convolution with arbitrary parameters N. The convolution regards the point in the upper left corner as the sampling origin (0,0). Let R represent the sampling grid, and R is expressed as follows: R＝{(-1,-1),(-1,0),...,(0,1),(1,1)} (1) The initial sampling position p of the convolution n It can be customized or randomly generated. p0 is the output image h i,1 The value of each pixel in the output image is calculated by the following formula: Among them, w(p n ) indicates p n The size of the convolution kernel weight of the point. n +offset n ) represents p0+p n +offset n The eigenvalue of the point, offset n is the offset, but if the model learns the deformation offset completely freely, the receptive field will often deviate from the target, especially when dealing with slender linear structures. Therefore, a convolution kernel of size k is considered, with the point in the upper left corner as the sampling origin C0, and the remaining k-1 sampling points adopt an iterative strategy when offsetting: each C i , i=1,2,...,k-1 points are relative to their previous point C i-1 ,i=1,2,...,k-1 adds an offset Δ={δ|δ∈[-1,1]}, and the offsets are accumulated in sequence to ensure that the convolution kernel conforms to the linear morphological structure. The formula is as follows: Since the offset is usually a decimal, bilinear interpolation is used to calculate the pixel value of the offset point. The formula is as follows: Where q represents the point on the feature map with the nearest four integers around point p, x(q) represents the pixel value of point q, and x(p) represents the value calculated after the offset point p, p = p0 + p n +offset n .

5. The method for extracting farmland plots based on a multi-type dual-branch convolutional neural network as claimed in claim 4, characterized in that: In step 2, the regional feature extraction branch of DBFE is composed of a multi-scale deformable convolutional feature extraction module MDFE. i , i = 1, 2, ..., m, and evenly divide it into s feature map subsets h according to the channel i,j ,j=1,2,...,s. Except h i,1 Without going through the convolutional layer, it is a quick connection. i,j ,j=2,3,...,s all have corresponding 5×5 deformable convolutional layers, using D i,j (·),j=2,3,...,s means, D i,j (·) is composed of 5×5 deformable convolution, BatchNormalization (BN) and ReLU activation function in sequence. i,j Indicates D i,j The output of each branch j is the output of the branch h. i,j Splice the output y of the previous branch i,j-1 After D i,j (·) is obtained, so y i,j can be written as: in, represents the concatenation operation, each 5×5 deformable convolution operator D i,j (·) It is possible to split h from all features i,j ,j≤i receives feature information, so the receptive field of each branch output result increases successively. This paper obtains larger convolution and receptive field by repeatedly stacking small-sized convolutions, and because two convolution layers use one more activation function than one convolution layer, the nonlinear ability is enhanced.

6. The method for extracting farmland plots based on a multi-type dual-branch convolutional neural network as claimed in claim 1, characterized in that: In step 3, the mask and boundary prediction tasks are binary classification tasks, so the negative log-likelihood (NLL) loss is used, while the distance map estimation task is a regression task and the mean square error (MSE) loss is used. Let L cls is the pixel classification loss (i.e., NLL), x is the pixel position in the image space R, and the loss function for the mask and boundary prediction task for N input instances is defined as: Among them, P mc (x; l mc (x)) represents the true label l activated by SoftMax mc The predicted probability of . In order to obtain the MSE loss for the distance map estimation task (i.e., L reg ), this paper uses the predicted distance Calculate the deviation from the actual distance D(x): For multiple tasks, the loss values ​​of different subtasks are summed up to get the total loss: L total =w1·L cls1 +w2·L cls2 +w3·L reg (8) Among them, L cls1 , L cls2 , L reg It refers to the loss values ​​of the three subtasks, corresponding to mask, boundary and distance prediction respectively, and w1, w2 and w3 are the corresponding weight coefficients.

7. The method for extracting farmland plots based on a multi-type dual-branch convolutional neural network as claimed in claim 1, characterized in that: The initial learning rate used in the training phase is 10 -2 and weight decay is 10 -4 The Apollo optimizer is used to train the mask prediction and distance map estimation tasks, and the initial learning rate is 10 -8 and weight decay is 10 -4 The stochastic gradient descent (SGD) optimizer is used to train the edge prediction task. The training batch is selected as 2 and the training lasts for 150 iterations.

Images