Erosion gully segmentation method based on multi-data target prototype fusion framework

Abstract

The invention discloses an erosion gully segmentation method based on a multi-data target prototype fusion framework, relates to the technical field of remote sensing image target extraction, and aims to solve the problem of low segmentation accuracy caused by interference of similar textures and spectral features in the existing segmentation method. According to the method, the digital elevation data for providing topographic information and the visible light image capable of providing visual features such as colors and textures are effectively fused, so that the interference of textures and spectral features of easy-to-confuse ground features such as weeds is relieved, and finally the accuracy of erosion gully segmentation is improved.

Description

Technical Field

[0001] The invention relates to the technical field of remote sensing image target extraction, and in particular to an erosion gully segmentation method based on a multi-data target prototype fusion framework. Background Art

[0002] Erosion gullies are gullies formed when the surface soil is eroded by wind and water, which can lead to soil loss, land degradation, ecosystem destruction and other disasters. Therefore, taking effective detection measures for erosion gullies can effectively protect land resources and maintain the sustainable development of the ecological environment.

[0003] At present, the most advanced method for extracting gullies is the semantic segmentation method based on deep learning using visible light data. However, for gullies, weeds and other objects inside and outside the gullies often have similar texture and spectral characteristics. Therefore, the existing segmentation methods have the problem of low segmentation accuracy due to the interference of similar texture and spectral characteristics. Summary of the invention

[0004] The purpose of the present invention is to propose an erosion gully segmentation method based on a multi-data target prototype fusion framework to address the problem that the existing segmentation methods have low segmentation accuracy due to the interference of similar texture and spectral characteristics.

[0005] The technical solution adopted by the present invention to solve the above technical problems is:

[0006] A gully segmentation method based on a multi-data target prototype fusion framework comprises the following steps:

[0007] Step 1: Obtain visible light data and digital elevation data in the study area;

[0008] Step 2: Extracting deep semantic features of visible light data and deep semantic features of digital elevation data;

[0009] Step 3: Use the deep semantic features of visible light data to construct a visible light erosion gully target prototype, and use the deep semantic features of digital elevation data to construct a digital elevation erosion gully target prototype;

[0010] Step 4: Perform dot multiplication on the deep semantic features of the visible light data and the target prototype of the visible light erosion gully to obtain the visible light enhancement feature, perform dot multiplication on the deep semantic features of the digital elevation data and the target prototype of the digital elevation erosion gully to obtain the digital elevation enhancement feature, and finally coordinate the visible light enhancement feature and the digital elevation enhancement feature to obtain the visible light data coordination feature and the digital elevation data coordination feature;

[0011] Step 5: After splicing the collaborative features of the visible light data and the collaborative features of the digital elevation data, input them into the segmentation encoder to obtain the output gully segmentation result.

[0012] Furthermore, the specific steps of step three are:

[0013] Step 31: Based on visible light data I R and digital elevation data I D , using the SLIC algorithm to construct a superpixel set S R and S D ;

[0014] Step 32: Get superpixel sets S respectively R and S D The binary mask of the nth superpixel in and And build them separately and The prototype vector P R n and P D n , the prototype vector P R n and P D n Respectively expressed as:

[0015]

[0016] Among them, f MAP (·) represents mask average pooling, MF R Represents the deep semantic features of visible light data, MF D Representing deep semantic features of digital elevation data;

[0017] Step 33: Using the prototype vector P R n Building the prototype block PB R , using the prototype vector P D n Building the prototype block PB D ;

[0018] Step 3 and 4: Prototype block PB R and prototype block PB D Selective sampling was performed to construct the visible light erosion gully target prototype TB. R and digital elevation erosion gully target prototype TB D , expressed as:

[0019]

[0020] in, represents dot product, f MLP(·) represents a multi-layer perceptron block for compressing dimensions, and fsigmoid(·) represents the sigmoid function.

[0021] Furthermore, the specific steps of step 4 are:

[0022] Step 41: TB R and MF R Perform point multiplication to obtain the visible light enhancement feature F R , TB D and MF D Multiply the rows by points to get the digital elevation enhancement feature F D , expressed as:

[0023]

[0024] in, represents dot product;

[0025] Step 42: Use three 1×1 convolutional layers with different weights to process F separately R and F D , and get the feature F R The corresponding query vector Q R , the vector K to be checked R and content vector V R , and feature F D The corresponding query vector Q D , the vector K to be checked D and content vector V D ;

[0026] Step 43: Get Q R The vector of pixel position x in Then use the pixel position x to extract K in the same spatial column R The yth element of and obtain and The degree of relevance

[0027] Step 44: Get Q D The vector of pixel position x in Then use the pixel position x to extract K in the same spatial column D The yth element of and obtain and The degree of relevance

[0028] Step 45: Repeat step 43 to get Q R The correlation degree of each pixel position in the , and then construct the vector Q R and vector KR The correlation matrix G R ;

[0029] Step 46: Repeat step 44 to get Q D The correlation degree of each pixel position in the , and then construct the vector Q D and vector K D The correlation matrix G D ;

[0030] Step 47: Based on the correlation matrix G R And the correlation matrix G D , and use the softmax function of the channel dimension to get the attention map AM R and AM D ;

[0031] Step 48: Using Attention Map AM D 、V R and F R , get the elevation feature F D The feature value of pixel position u in the coordinated feature

[0032] Step 49: Using Attention Map AM R 、V D and F D , get the elevation feature F R The feature value of pixel position u in the coordinated feature

[0033] Step 410: Repeat step 48 for F R After coordinating each pixel position in the image, the visible light data collaborative feature is obtained.

[0034] Step 411: Repeat step 49 for F D After coordinating each pixel position in the image, the collaborative features of the digital elevation data are obtained.

[0035] Furthermore, the specific steps of step five are:

[0036] Visible light data collaborative features Collaborative features with digital elevation data The concatenation is performed in the channel dimension and input into the segmentation encoder to obtain the output erosion gully segmentation result.

[0037] Furthermore, the degree of correlation It is expressed as:

[0038]

[0039] Furthermore, the degree of correlation It is expressed as:

[0040]

[0041] Furthermore, the characteristic value It is expressed as:

[0042]

[0043] in, Indicates that the elevation feature F has been accepted D The eigenvalue of position u in the coordinated features, Representation feature F R The eigenvalue of position u in the middle, H and W represent F R The length and width of Indicates V R The vth eigenvector in the same row or column as position u in Represents the attention map AM D The feature vector at position u and the attention value of the vth feature vector in .

[0044] Furthermore, the characteristic value It is expressed as:

[0045]

[0046] in, Indicates that the elevation feature F has been accepted R The eigenvalue of position u in the coordinated features, Representation feature F D The eigenvalue of position u in the middle, H and W represent F D The length and width of Indicates V D The vth eigenvector in the same row or column as position u in Represents the attention map AM R The feature vector at position u and the attention value of the vth feature vector in .

[0047] Furthermore, in the step 2, the deep semantic features of the visible light data and the deep semantic features of the digital elevation data are extracted by a visible light encoder and a digital elevation encoder respectively.

[0048] Furthermore, the segmentation encoder in step five is transformer, cnn or TransUnet.

[0049] The beneficial effects of the present invention are:

[0050] This application effectively integrates digital elevation data that provides terrain information and visible light images that can provide visual features such as color and texture, thereby alleviating the interference of texture and spectral characteristics of easily confused objects such as weeds, and ultimately improving the accuracy of gully segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a schematic diagram of the prototype fusion framework based on multiple data targets;

[0052] Figure 2 This is a schematic diagram of the prototype of the erosion gully target;

[0053] Figure 3 This is a schematic diagram of the overall structure of step 4;

[0054] Figure 4 Schematic diagram of erosion gully. DETAILED DESCRIPTION

[0055] Specific implementation method 1: The erosion gully segmentation method based on a multi-data target prototype fusion framework described in this implementation method includes the following steps:

[0056] S1. Obtain visible light data and digital elevation data in the study area;

[0057] S2, input the two data in step into the corresponding encoders respectively to extract the deep features of different data;

[0058] S3 inputs the two data into the corresponding encoders to extract deep semantic features, represented as MF R and MF D .

[0059] S4. Input the deep features extracted in step S3 into the target prototype module of the corresponding data to obtain target prototypes of different types of data.

[0060] The specific implementation method of step S4 includes the following steps:

[0061] S4.1, using SLIC algorithm to separate data I R and I D Construct a superpixel prototype set S R and S D , where Mask n ( or ) is the binary mask of the nth superpixel in the corresponding set;

[0062] S4.2. Constructing Mask n The prototype P n , the calculation expression is:

[0063] Pn =f MAP (MF,Mask n ))

[0064] Among them, f MAP (·) represents mask average pooling, and MF represents the corresponding deep semantic features, that is, MF R or MF D ;

[0065] S4.3, using step S4.2 to calculate the prototype of each superpixel and construct a prototype block PB;

[0066] S4.4. Selectively sample the prototype block PB to construct the target prototype TB. The calculation expression is:

[0067]

[0068] in, is the dot product, f MLP (·) is a multilayer perceptron block for compressing dimensionality, and fsigmoid(·) is a sigmoid function.

[0069] S5, combining the deep features obtained in step S3 and the target prototype obtained in step S4 and inputting them into a multi-data fusion module to obtain a multi-data activation feature that emphasizes the erosion gully area;

[0070] The specific implementation method of step S5 includes the following steps:

[0071] S5.1, using the visible light target prototype block TB obtained in step S4.4 R and elevation target prototype block TB D To activate the feature MF R and MF D The target feature in is calculated as:

[0072]

[0073] in, is the dot product, F R is a multi-level feature of the visible light data that has been activated by the target feature, and F D It is a multi-level feature of digital elevation data that has been activated by the target feature.

[0074] S5.2. Using the cross attention mechanism to implement feature F R and feature F D The specific steps of cross-modal interaction are as follows:

[0075] S5.2.1. Use three 1×1 convolutional layers to process feature F separately R and feature FD , get Q R , K R and V R and Q D , K D and V D ;

[0076] S5.2.2 Calculation of Q R and K R The correlation degree is calculated as:

[0077]

[0078] in, yes and The degree of correlation between It's Q R The x position vector in , is the position x in the same spatial column as K R The y-th element in the eigenvector of ;

[0079] S5.2.3. Calculation of Q D and K D The correlation degree is calculated as:

[0080]

[0081] in, yes and The degree of correlation between It's Q D The x position vector in , is the position x in the same spatial column as K D The y-th element in the eigenvector of ;

[0082] S5.2.4. Calculate each position using steps S5.2.2 and S5.2.3 to obtain G R and G D ;

[0083] S5.2.5, respectively, for G in step S5.2.4 R and G D , apply the softmax function of the channel dimension to obtain the attention map AM R and AM D ;

[0084] S5.2.6. Use elevation feature F D To supplement the visible light feature F R , the calculation expression is:

[0085]

[0086] Among them, H and W represent F R The length and width of By V R The y-th eigenvector in the same row or column as position x;

[0087] S5.2.7. Using visible light feature F R To supplement the elevation feature F D , the calculation expression is:

[0088]

[0089] Among them, H and W represent F R The length and width of By V D The y-th eigenvector in the same row or column as position x;

[0090] S5.2.8. By performing steps S5.2.6 and S5.2.7 for each position, the corresponding cross-modal activation features can be obtained. and

[0091] S6, inputting the multi-data activation features obtained in step S5 into an encoder to obtain an erosion gully segmentation result;

[0092] Step S6 converts the cross-modal activation features in step S5 into and Perform channel connection and input into the encoder to obtain the erosion gully segmentation result.

[0093] S7, analyzing the segmentation result obtained in step S6, and outputting remote sensing accuracy analysis coefficients, including precision rate Pre, recall rate Rec and F1 index.

[0094] Step S7 performs remote sensing data accuracy analysis on the gully segmentation result obtained in step S6, and obtains the remote sensing accuracy analysis coefficient by comparing the output result with the semantic label of the image, including the accuracy rate Pre, the recall rate Rec and the F1 index. The calculation expression is:

[0095]

[0096] Among them, TP is the number of pixels correctly predicted for erosion gullies, TN is the number of pixels correctly predicted for non-erosion gullies, FP is the number of pixels predicted for non-erosion gullies as erosion gullies, and FN is the number of pixels predicted for erosion gullies as non-erosion gullies.

[0097] Example:

[0098] The data used in this implementation is data taken by a drone in Bin County, Harbin City, Heilongjiang Province. The data includes visible light data and digital elevation data, as shown in the attached Figure 4 As shown in the figure, the processed data images were cut into 2213 slices of 384*384, and the processed data images were labeled. The labeled categories included erosion gullies and non-erosion gullies. Finally, the 2213 images were randomly divided into training set, validation set and test set in a ratio of 7:1:2.

[0099] On the Bin County dataset, this implementation uses a variety of advanced algorithms for comparison, and the following results are obtained as shown in Table 1. It can be seen from the results that the method proposed in the present invention shows great superiority. In particular, F1 reaches 67.84%, which is 7.4% higher than SA-Gate. Therefore, this application can more effectively fuse digital elevation data that provides terrain information and visible light images that can provide visual features such as color and texture, thereby showing great practical value.

[0100] Table 1: Comparison results

[0101]

[0102] This application designs a gully segmentation method based on a multi-data target prototype fusion framework. This application effectively fuses digital elevation data that can provide terrain information and visible light images that can provide visual features such as color and texture, which can improve the ability to distinguish confusing objects such as weeds, thereby reliably separating gullies from the background. This method provides an effective solution for timely monitoring of gullies.

[0103] It should be noted that the specific implementation is only an explanation and description of the technical solution of the present invention, and cannot be used to limit the scope of protection of the rights. Any partial changes made according to the claims and description of the present invention should still fall within the scope of protection of the present invention.

Claims

1. A gully segmentation method based on a multi-data target prototype fusion framework, characterized in that The following steps are involved: Step 1: Obtain visible light data and digital elevation data in the study area; Step 2: Extracting deep semantic features of visible light data and deep semantic features of digital elevation data; Step 3: Use the deep semantic features of visible light data to construct a visible light erosion gully target prototype, and use the deep semantic features of digital elevation data to construct a digital elevation erosion gully target prototype; Step 4: Perform dot multiplication on the deep semantic features of the visible light data and the prototype of the visible light erosion gully target to obtain the visible light enhancement feature, perform dot multiplication on the deep semantic features of the digital elevation data and the prototype of the digital elevation erosion gully target to obtain the digital elevation enhancement feature, and finally coordinate the visible light enhancement feature and the digital elevation enhancement feature to obtain the visible light data coordination feature and the digital elevation data coordination feature; Step 5: After splicing the collaborative features of the visible light data and the collaborative features of the digital elevation data, input them into the segmentation encoder to obtain the output gully segmentation result.

2. The erosion gully segmentation method based on a multi-data target prototype fusion framework according to claim 1 is characterized in that The specific steps of step three are: Step 31: Based on visible light data I R and digital elevation data I D , using the SLIC algorithm to construct a superpixel set S R and S D ; Step 32: Get superpixel sets S respectively R and S D The binary mask of the nth superpixel in and And build separately and The prototype vector P R n and P D n , prototype vector P R n and P D n Respectively expressed as: Among them, f MAP (·) represents mask average pooling, MF R Represents the deep semantic features of visible light data, MF D Representing deep semantic features of digital elevation data; Step 33: Using the prototype vector P R n Building the prototype block PB R , using the prototype vector P D n Building the prototype block PB D ; Step 3 and 4: Prototype block PB R and prototype block PB D Selective sampling was performed to construct the visible light erosion gully target prototype TB. R and digital elevation erosion gully target prototype TB D , expressed as: in, represents dot product, f MLP (·) represents a multi-layer perceptron block for compressing dimensions, and fsigmoid(·) represents the sigmoid function.

3. The erosion gully segmentation method based on a multi-data target prototype fusion framework according to claim 2 is characterized in that The specific steps of step 4 are: Step 41: TB R and MF R Perform point multiplication to obtain the visible light enhancement feature F R , TB D and MF D Multiply the rows by points to get the digital elevation enhancement feature F D , expressed as: in, represents dot product; Step 42: Use three 1×1 convolutional layers with different weights to process F separately R and F D , and get the feature F R The corresponding query vector Q R , the vector K to be checked R and content vector V R , and feature F D The corresponding query vector Q D , the vector K to be checked D and content vector V D ; Step 43: Get Q R The vector of pixel position x in Then use the pixel position x to extract K in the same spatial column R The yth element of and obtain and The degree of relevance Step 44: Get Q D The vector of pixel position x in Then use the pixel position x to extract K in the same spatial column D The yth element of and obtain and The degree of relevance Step 45: Repeat step 43 to get Q R The correlation degree of each pixel position in the , and then construct the vector Q R and vector K R The correlation matrix G R ; Step 46: Repeat step 44 to get Q D The correlation degree of each pixel position in the , and then construct the vector Q D and vector K D The correlation matrix G D ; Step 47: Based on the correlation matrix G R And the correlation matrix G D , and use the softmax function of the channel dimension to get the attention map AM R and AM D ; Step 48: Using Attention Map AM D 、V R and F R , get the elevation feature F D The feature value of pixel position u in the coordinated feature Step 49: Using Attention Map AM R 、V D and F D , get the elevation feature F R The feature value of pixel position u in the coordinated feature Step 410: Repeat step 48 for F R After coordinating each pixel position in the image, the visible light data collaborative feature is obtained. Step 411: Repeat step 49 for F D After coordinating each pixel position in the image, the digital elevation data coordination feature is obtained.

4. The erosion gully segmentation method based on a multi-data target prototype fusion framework according to claim 3 is characterized in that The specific steps of step five are: Visible light data collaborative features Collaborative features with digital elevation data The concatenation is performed in the channel dimension and input into the segmentation encoder to obtain the output erosion gully segmentation result.

5. The erosion gully segmentation method based on a multi-data target prototype fusion framework according to claim 4 is characterized in that The degree of relevance It is expressed as:

6. The erosion gully segmentation method based on a multi-data target prototype fusion framework according to claim 5 is characterized in that The degree of relevance It is expressed as:

7. The erosion gully segmentation method based on a multi-data target prototype fusion framework according to claim 6 is characterized in that The characteristic value It is expressed as: in, Indicates that the elevation feature F has been accepted D The eigenvalue of position u in the coordinated features, Representation feature F R The eigenvalue of position u in the middle, H and W represent F R The length and width of Indicates V R The vth eigenvector in the same row or column as position u in Represents the attention map AM D The feature vector at position u and the attention value of the vth feature vector in .

8. The erosion gully segmentation method based on a multi-data target prototype fusion framework according to claim 7 is characterized in that The characteristic value It is expressed as: in, Indicates that the elevation feature F has been accepted R The eigenvalue of position u in the coordinated features, Representation feature F D The eigenvalue of position u in the middle, H and W represent F D The length and width of Indicates V D The vth eigenvector in the same row or column as position u in Represents the attention map AM R The feature vector at position u and the attention value of the vth feature vector in .

9. The erosion gully segmentation method based on a multi-data target prototype fusion framework according to claim 8, characterized in that In the step 2, the deep semantic features of the visible light data and the deep semantic features of the digital elevation data are extracted by a visible light encoder and a digital elevation encoder respectively.

10. The erosion gully segmentation method based on a multi-data target prototype fusion framework according to claim 9, characterized in that The segmentation encoder in step 5 is transformer, cnn or TransUnet.

Images