Method, device, medium and equipment for detecting winter wheat under complex terrain conditions

Abstract

The invention discloses a method, a device, a medium and equipment for detecting winter wheat under complex terrain conditions, and relates to the technical field of image analysis. The method comprises the steps that digital elevation model data is used for reflecting topographic information of a preset winter wheat area; constructing a fusion segmentation network of the multi-modal data; the encoder is composed of two residual modules and four multi-modal feature fusion modules of a double-branch structure. The double-branch structure comprises an optical branch and an elevation branch; the decoder is formed by connecting four decoding modules, and each decoding module comprises an up-sampling module and a distribution feature enhancement module; the inputs of the last three up-sampling modules in the decoder are respectively connected with the outputs of the first three optical branches in the encoder; training the fusion segmentation network through the multi-modal training data set to obtain a winter wheat detection model; and finally outputting a detection result through the winter wheat detection model. The method can accurately detect the winter wheat distribution condition in a complex terrain area.

Description

Technical Field

[0001] The present invention relates to the technical field of image analysis, and in particular to a method, device, medium and equipment for detecting winter wheat under complex terrain conditions. Background Art

[0002] The development of satellite remote sensing technology has greatly promoted the dynamic monitoring of global surface changes and accumulated a large amount of remote sensing satellite data, which provides a reliable basis for surface coverage and utilization. In the agricultural field, remote sensing images can play an important role in assisting decision-making in resource management, agricultural production and food security policy making. In particular, as winter wheat is an important food crop in my country, monitoring its growth and distribution is crucial to ensuring national food supply.

[0003] In the prior art, semantic segmentation technology is usually used to classify surface targets at the pixel level using image processing technology on remote sensing images of winter wheat to monitor their growth distribution. In addition to the traditional method of simply distinguishing ground objects based on edge or texture information of the image and a fixed spectral threshold, most of the current semantic segmentation methods are based on the spectral information of a single phase optical image.

[0004] However, due to the influence of complex terrain, existing semantic segmentation methods are prone to lose the spatial distribution information of winter wheat when identifying winter wheat, which greatly reduces the accuracy of winter wheat distribution detection in complex terrain, resulting in serious false alarms and missed detections, especially in hilly and mountainous areas where the distribution of winter wheat is relatively fragmented and irregular, making it difficult to accurately capture small and scattered crop areas. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device, medium and equipment for detecting winter wheat under complex terrain conditions in response to the above-mentioned technical problems. This method can accurately detect the distribution of winter wheat in complex terrain areas.

[0006] The present invention adopts the following technical solutions:

[0007] Acquire a multimodal training data set; the multimodal training data set includes sample dual-phase optical remote sensing images and sample digital elevation model data corresponding to a preset winter wheat area; the digital elevation model data is used to reflect the terrain information of the preset winter wheat area;

[0008] According to the encoder and decoder, a fusion segmentation network for multimodal data is constructed; the encoder consists of two residual modules and four multimodal feature fusion modules with a dual-branch structure; the dual-branch structure includes an optical branch and an elevation branch; the decoder is composed of four decoding modules connected, each of which includes an upsampling module and a distribution feature enhancement module; the inputs of the last three upsampling modules in the decoder are respectively connected to the outputs of the first three optical branches in the encoder;

[0009] The fusion segmentation network is trained using a multimodal training dataset to obtain a winter wheat detection model.

[0010] Obtain dual-temporal optical remote sensing images and digital elevation model data of the winter wheat area to be inspected;

[0011] The dual-phase optical remote sensing image and digital elevation model data are input into the winter wheat detection model, and two residual modules are used to unify the number of channels of the dual-phase optical remote sensing image and the digital elevation model data to obtain optical features and elevation features; the optical features and elevation features are extracted through four multimodal feature fusion modules in turn, and the optical features and elevation features extracted through the fourth multimodal feature fusion module are spliced ​​to obtain fused features; the fused features and the features output by the corresponding optical branches are used as the input of the decoder, and the fused features and the optical branches are enhanced through the decoding modules in turn to obtain the detection results of the winter wheat area to be detected output by the decoder; the detection result is the pixel-level classification of the growth distribution of winter wheat in the winter wheat area to be detected.

[0012] Preferably, the acquiring of a multimodal training data set includes:

[0013] Obtain sample optical remote sensing images and sample digital elevation model data of the preset winter wheat area at two time points;

[0014] The two sample optical remote sensing images are band-fused to obtain a sample dual-phase optical remote sensing image;

[0015] The sample dual-temporal optical remote sensing images and sample digital elevation model data are used as training samples;

[0016] The training samples are divided into multimodal training data set, multimodal test data set and multimodal verification data set according to preset proportions.

[0017] Preferably, the method further comprises:

[0018] After obtaining the winter wheat detection model, the winter wheat detection model is tested using a validation data set to obtain test results;

[0019] If the test result does not meet the preset test conditions, the fusion segmentation network will continue to be trained through the multimodal training data set until the winter wheat detection model is obtained.

[0020] Preferably, the fusion segmentation network is trained by a multimodal training data set to obtain a winter wheat detection model, including:

[0021] Input the multimodal training dataset into the fusion segmentation network to obtain the predicted pixel-level classification of the multimodal training dataset;

[0022] The predicted pixel-level classification and the actual pixel-level classification corresponding to the multimodal training dataset are input into the cross entropy loss function to obtain the difference;

[0023] Iteratively update the fusion segmentation network based on difference, Adam optimizer and cosine annealing mechanism;

[0024] The iteratively updated fusion segmentation network is verified through the multimodal validation set, and the fusion segmentation network with the smallest loss function value on the multimodal validation set is determined as the winter wheat detection model.

[0025] Preferably, the optical branch of the multimodal feature fusion module is composed of a maximum pooling submodule, a deformable convolutional network block and a residual network, and the elevation branch is composed of a maximum pooling submodule and a convolutional block, and the output of the convolutional block is connected to the output of the deformable convolutional network block along the channel direction as the input of the residual network;

[0026] The specific features for any multimodal feature fusion module include:

[0027] The optical features and elevation features are input into the maximum pooling submodule, and the maximum pooling operation is used to perform 2 times downsampling in the height and width directions to obtain the first pooled optical features and the first pooled elevation features;

[0028] The pooled elevation feature is extracted through the convolution block to obtain the second pooled elevation feature. The number of channels of the second pooled elevation feature is twice that of the first pooled elevation feature.

[0029] The first pooled optical feature is extracted through a deformable convolutional network block to obtain an intermediate optical feature; the number of channels of the intermediate optical feature is twice that of the first pooled optical feature;

[0030] The second pooled elevation features and the intermediate optical features are connected along the channel direction and input into the residual network to obtain the second pooled optical features; the second pooled optical features and the second pooled elevation features are the optical features and elevation features input into the next multimodal feature fusion module.

[0031] Preferably, the deformable convolutional network block includes a convolution layer, a batch normalization layer, a nonlinear rectification activation function layer and a variable convolution block; the convolution block consists of a standard 3×3 convolution with a padding of 1 and a stride of 1, a batch normalization layer and a nonlinear rectification activation function layer.

[0032] Preferably, the distribution feature enhancement module includes a global maximum pooling submodule, a global average pooling submodule, a first activation function, a convolutional layer, and a second activation function;

[0033] For any distribution feature enhancement module, it specifically includes:

[0034] The upsampled features are passed through the global maximum pooling submodule and the global average pooling submodule respectively, and the features after global maximum pooling and global average pooling are connected along the channel direction to obtain the intermediate weight matrix;

[0035] Pass the intermediate weight matrix through the convolution layer and the first activation function to obtain the spatial attention weight matrix;

[0036] The upsampled features are passed through the second activation function to obtain enhanced features;

[0037] The enhanced features, spatial attention weight matrix and sampled features are point-multiplied in turn to obtain enhanced features; the enhanced features are the input of the next decoding module.

[0038] The present invention provides a detection device for winter wheat under complex terrain conditions, comprising:

[0039] The training set acquisition module is used to acquire a multimodal training data set; the multimodal training data set includes sample dual-phase optical remote sensing images and sample digital elevation model data corresponding to a preset winter wheat area; the digital elevation model data is used to reflect the terrain information of the preset winter wheat area;

[0040] A construction module is used to construct a fusion segmentation network for multimodal data according to an encoder and a decoder; the encoder is composed of two residual modules and four multimodal feature fusion modules of a dual-branch structure; the dual-branch structure includes an optical branch and an elevation branch; the decoder is composed of four decoding modules connected, each decoding module includes an upsampling module and a distribution feature enhancement module; the inputs of the last three upsampling modules in the decoder are respectively connected to the outputs of the first three optical branches in the encoder;

[0041] A training module is used to train the fusion segmentation network through a multimodal training data set to obtain a winter wheat detection model;

[0042] A data acquisition module is used to acquire dual-phase optical remote sensing images and digital elevation model data of the winter wheat area to be inspected;

[0043] The detection module is used to input the dual-phase optical remote sensing image and the digital elevation model data into the winter wheat detection model, and use two residual modules to unify the number of channels of the dual-phase optical remote sensing image and the digital elevation model data to obtain optical features and elevation features; the optical features and elevation features are extracted in turn through four multimodal feature fusion modules, and the feature map obtained by the fourth multimodal feature fusion module is spliced ​​to obtain fusion features; the fusion features and the features output by the corresponding optical branches are used as inputs of the decoder, and the fusion features and the optical branches are enhanced in turn through the decoding modules to obtain the detection results of the winter wheat area to be detected output by the decoder; the detection result is a pixel-level classification of the growth distribution of winter wheat in the winter wheat area to be detected.

[0044] The present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method for detecting winter wheat under complex terrain conditions is implemented.

[0045] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for detecting winter wheat under complex terrain conditions is implemented.

[0046] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0047] In the winter wheat detection method under complex terrain conditions provided by the present invention, optical features and elevation features are extracted and fused through a dual-branch structure of four multimodal feature fusion modules, so that the model can simultaneously utilize data from different sources, providing more comprehensive feature support for semantic segmentation; and, in the decoder, the outputs of the first three optical branches are connected with the inputs of the last three upsampling modules in the decoder, so that semantic reasoning of features can be realized at different levels. This combination effectively realizes the accurate detection of complex terrain areas such as multi-scale and irregular crop areas, improves the effect of semantic segmentation, and thus improves the accuracy of winter wheat detection in complex areas.

[0048] Furthermore, the dual-phase optical remote sensing images and digital elevation model data respectively fuse the information obtained by different sensors, and make full use of the complementary information of the dual-phases and the terrain characteristics of the winter wheat area. The spatial distribution characteristics of the winter wheat area can be integrated into the model to provide a more comprehensive and accurate scene understanding, enabling the model to understand the research object more efficiently and deeply, thereby improving the accuracy of the winter wheat detection model and thus improving the accuracy of winter wheat detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0050] Figure 1 A schematic flow chart of a method for detecting winter wheat under complex terrain conditions provided by the present invention;

[0051] Figure 2 A schematic diagram of the structure of a fusion segmentation network provided by the present invention;

[0052] Figure 3 A schematic diagram of the structure of a multimodal feature fusion module provided by the present invention;

[0053] Figure 4 A structural schematic diagram of a deformable convolutional network block provided by the present invention;

[0054] Figure 5 A schematic diagram of the structure of a distribution feature enhancement module provided by the present invention;

[0055] Figure 6 A schematic flow chart of another method for detecting winter wheat under complex terrain conditions provided by the present invention;

[0056] Figure 7 A schematic diagram of a winter wheat detection device under complex terrain conditions provided by the present invention;

[0057] Figure 8 A schematic diagram of a computer device for implementing a method for detecting winter wheat under complex terrain conditions provided by the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0059] In order to solve the limitations in the detection of complex terrain and crop distribution, it is urgent to explore semantic segmentation methods based on multimodal data fusion. By combining multimodal data, the detection accuracy of crop distribution in complex terrain areas can be improved, and the interference of factors such as lighting conditions, atmospheric conditions and terrain undulations on the classification results can be overcome, and false alarms and missed detections can be reduced. The problem of traditional crop segmentation methods losing spatial distribution information of winter wheat under complex terrain conditions and difficulty in accurately capturing small areas can be solved, thereby improving the applicability of winter wheat semantic segmentation models in complex terrain and improving the accuracy of winter wheat detection in complex terrain areas.

[0060] Based on this, the present invention provides a method for detecting winter wheat under complex terrain conditions. The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0061] The execution subject of the winter wheat detection method under complex terrain conditions in the present invention can be a server, and the server can be a server set up on a business platform, or a device such as a desktop computer, a laptop computer, etc. that can execute the solution of the present invention.

[0062] Figure 1 The present invention is a flow chart of a method for detecting winter wheat under complex terrain conditions, which specifically includes the following steps:

[0063] S101, obtaining a multimodal training dataset.

[0064] The multimodal training dataset includes sample dual-phase optical remote sensing images and sample digital elevation model data corresponding to a preset winter wheat area; the digital elevation model data is used to reflect the terrain information of the preset winter wheat area.

[0065] Preferably, obtaining a multimodal training data set includes: obtaining sample optical remote sensing images and sample digital elevation model data of a preset winter wheat area at two time points; and performing band fusion on the two sample optical remote sensing images to obtain sample dual-phase optical remote sensing images; and then using the sample dual-phase optical remote sensing images and the sample digital elevation model data as training samples; and finally dividing the training samples into a multimodal training data set, a multimodal test data set, and a multimodal verification data set according to a preset ratio.

[0066] The preset winter wheat region may be any region where winter wheat grows; the two time points may be winter wheat and spring wheat; for example, December 2017 and March 2018. The sample optical remote sensing images may be optical remote sensing images collected in the preset winter wheat region at the corresponding two time points, and the sample digital elevation model images may be terrain information of the preset winter wheat region.

[0067] The two sample optical remote sensing images are subjected to band fusion to obtain a sample dual-phase optical remote sensing image, specifically comprising: combining the two sample optical remote sensing images in the band dimension, stacking them in the band order, and obtaining a sample dual-phase optical remote sensing image containing two time band combinations; for example, the two sample optical remote sensing images are I1 and I2, each sample optical remote sensing image includes C bands, and the image size is H×W.

[0068] In order to make the multimodal input data containing dual-phase information in the multimodal training data set, first, I1 and I2 are band fused, specifically, the spectral information of each pixel in I1 and I2 is cross-stacked to obtain a sample dual-phase optical remote sensing image. The optical remote sensing image after band fusion changes from the original (C, H, W) to (2C, H, W). The image size of the sample dual-phase optical remote sensing image is

[0069] Preferably, after obtaining the band fusion of the two sample optical remote sensing images, the sample optical remote sensing images after the band fusion can also be subjected to graphic operations; the graphic operations are to balance the data samples through image enhancement technology, including operations such as flipping, rotating, scaling, translating and cropping, to generate data samples of various perspectives and proportions, and to balance the number of data samples of rare types or sparsely distributed areas. The sample optical remote sensing images after the image operations can be used as sample dual-phase optical remote sensing images.

[0070] The sample digital elevation model (DEM) data is a spatial data model that describes the undulating morphological characteristics of the surface. It is a matrix composed of the elevation values ​​of regular grid points on the ground, forming a grid structure data set. As another type of data in the multimodal training data set, it does not participate in the band fusion process, and the shape of the DEM data remains unchanged.

[0071] Therefore, the sample dual-time optical remote sensing images and sample digital elevation model data obtained above can be used as training samples.

[0072] Among them, the preset ratio can be 6:2:2, that is, the training samples are divided into a multimodal training data set, a multimodal test data set and a multimodal verification data set according to the ratio of 6:2:2.

[0073] S102, constructing a fusion segmentation network for multimodal data based on the encoder and decoder; the encoder consists of two residual modules and four multimodal feature fusion modules with a dual-branch structure; the dual-branch structure includes an optical branch and an elevation branch; the decoder is composed of four decoding modules connected together, each decoding module includes an upsampling module and a distribution feature enhancement module; the inputs of the last three upsampling modules in the decoder are respectively connected to the outputs of the first three optical branches in the encoder.

[0074] like Figure 2 As shown, Figure 2 It can be a structural diagram of the fusion segmentation network; the fusion segmentation network consists of two parts: an encoder and a decoder. The encoder is responsible for extracting the features of multimodal data, and the decoder is used to restore the resolution and enhance the feature expression; wherein the multimodal data includes dual-phase optical remote sensing images and digital elevation model data.

[0075] The encoder consists of residual module 1, residual module 2, multimodal feature fusion module 1, multimodal feature fusion module 2, multimodal feature fusion module 3 and multimodal feature fusion module 4. The input of residual module 1 is digital elevation model data, and the input of residual module 2 is dual-phase optical remote sensing image. Each multimodal feature fusion module (Multi-ModalFeature Fusion Module, MFFM) has a dual-branch structure: an optical branch and an elevation branch, which respectively extracts the features of dual-phase optical remote sensing images and digital elevation model data. The features corresponding to dual-phase optical remote sensing images can be called optical features, and the features corresponding to digital elevation model data can be called elevation features.

[0076] The multimodal feature fusion module 1 extracts features from the optical features and elevation features output by the residual module 1 and the residual module 2, respectively; the multimodal feature fusion module 2 extracts features from the optical features and elevation features output by the multimodal feature fusion module 1, respectively; the multimodal feature fusion module 3 extracts features from the optical features and elevation features output by the multimodal feature fusion module 2, respectively; the multimodal feature fusion module 4 extracts features from the optical features and elevation features output by the multimodal feature fusion module 3, respectively; finally, the optical features and elevation features output by the multimodal feature fusion module 4 may be feature fused, specifically, the optical features and elevation features output by the multimodal feature fusion module 4 are concatenated, i.e., channel connected, to obtain fused features.

[0077] The multimodal feature fusion module combines optical images and digital elevation model data to extract and fuse the features of both during the encoding process. The optical features of the optical image are extracted by deformable convolution (DCN) to capture the multi-scale and irregular crop distribution characteristics. Assume that the optical feature is F opt , the elevation feature of the digital elevation model data is Fdem , then the feature fusion representation that can be obtained is: F fusion =concat(DCN(F opt ),Conv(F dem )).

[0078] The decoder is composed of four decoding modules: decoding module 1, decoding module 2, decoding module 3 and decoding module 4. Each decoding module includes an upsampling module and a distribution feature enhancement module. The input of the upsampling module is the input of the corresponding decoding module, the output of the upsampling module is the input of the distribution feature enhancement module, and the output of the distribution feature enhancement module is the output of the corresponding decoding module; the input of decoding module 1 is the fusion feature output in the encoder, the input of decoding module 2 is the output of decoding module 1, the input of decoding module 3 is the output of decoding module 2, and the input of decoding module 4 is the output of decoding module 3.

[0079] Among them, the inputs of the last three upsampling modules in the decoder are respectively connected to the outputs of the first three optical branches in the encoder, specifically: the output of the optical branch in the multimodal feature fusion module 1 is the input of the upsampling module in the decoding module 4, the output of the optical branch in the multimodal feature fusion module 2 is the input of the upsampling module in the decoding module 3, and the output of the optical branch in the multimodal feature fusion module 4 is the input of the upsampling module in the decoding module 2.

[0080] S103, training the fusion segmentation network using the multimodal training data set to obtain a winter wheat detection model.

[0081] Preferably, the fused segmentation network is trained by a multimodal training data set to obtain a winter wheat detection model, including: inputting the multimodal training data set into the fused segmentation network to obtain a predicted pixel-level classification of the multimodal training data set; inputting the predicted pixel-level classification and the true pixel-level classification corresponding to the multimodal training data set into a cross-entropy loss function to obtain a difference; based on the difference, Adam optimizer and cosine annealing mechanism, the fused segmentation network is iteratively updated to obtain a winter wheat detection model.

[0082] The multimodal training dataset is input into the fusion segmentation network. The optical and elevation features of the crops are first captured by the multimodal feature fusion module in the encoder. Then, the response to the sparsely broken winter wheat area is further enhanced by the distribution feature enhancement module in the decoder stage. Finally, the enhanced features output by the decoder are obtained, which is the predicted pixel-level classification of the multimodal training dataset.

[0083] During the training process, the cross entropy loss function is used to optimize the fusion segmentation network to reduce the error of the model. In each training iteration, the predicted pixel-level classification and the actual pixel-level classification corresponding to the multimodal training data set are input into the cross entropy loss function, and the loss function calculates the prediction error of each pixel, that is, the difference; then based on the difference, Adam optimizer and warm up plus cosine annealing mechanism, the fusion segmentation network is iteratively updated; finally, the iteratively updated fusion segmentation network is verified through the multimodal validation set, and the fusion segmentation network with the smallest loss function value on the multimodal validation set is determined as the winter wheat detection model.

[0084] The cross entropy loss function calculates the true pixel-level classification y for each pixel i Pixel-level classification with prediction The cross entropy loss function is shown in formula (1).

[0085]

[0086] Among them, L CE is the difference between the predicted pixel-level classification and the actual pixel-level classification, and N is the number of pixels.

[0087] The fused segmentation network after each iterative update is verified through the multimodal validation set to obtain the loss function value of each iterative update, and the fused segmentation network with the smallest loss function value on the multimodal validation set is determined as the winter wheat detection model.

[0088] Optionally, the fusion segmentation network is iteratively trained using the multimodal training data set until the fusion segmentation network reaches a preset convergence condition, and the training is stopped. The convergence condition can be set to the number of iterations satisfying a preset number of iterations.

[0089] In an exemplary embodiment, after obtaining the winter wheat detection model, the performance of the winter wheat can also be evaluated, specifically including: after obtaining the winter wheat detection model, testing the winter wheat detection model through a validation data set to obtain a test result; if the test result does not meet the preset test condition, continuing to train the fusion segmentation network through a multimodal training data set until the winter wheat detection model is obtained. The preset test condition can be that the test accuracy reaches a preset accuracy threshold.

[0090] S104, obtaining dual-phase optical remote sensing images and digital elevation model data of the winter wheat area to be detected.

[0091] The winter wheat area to be detected may be an area where the winter wheat planting distribution situation needs to be detected, and the digital elevation model data is the terrain information data of the winter wheat area to be detected.

[0092] Optical remote sensing images collected at two time points in the winter wheat area to be detected can be first obtained. The optical remote sensing images collected at the two time points can be wheat growth images of the winter wheat area to be detected in winter and wheat growth images in the spring of the following year.

[0093] The two optical remote sensing images are band-fused to obtain a dual-phase optical remote sensing image. It should be noted that the band fusion method is the same as the method of obtaining the sample dual-phase optical remote sensing image by band fusion in the above embodiment, and the embodiments of the present application will not be repeated here.

[0094] S105, input the dual-phase optical remote sensing image and the digital elevation model data into the winter wheat detection model, use two residual modules to unify the number of channels of the dual-phase optical remote sensing image and the digital elevation model data respectively, and obtain optical features and elevation features; extract the optical features and elevation features through four multimodal feature fusion modules in turn, and splice the optical features and elevation features extracted by the fourth multimodal feature fusion module to obtain fused features; use the fused features and the features output by the corresponding optical branch as the input of the decoder, and enhance the features of the fused features and the optical branches through the decoding modules in turn to obtain the detection result of the winter wheat area to be detected output by the decoder.

[0095] The detection result is a pixel-level classification of the growth distribution of winter wheat in the winter wheat area to be detected. The pixel-level classification can be a pixel-level segmentation map showing the distribution of winter wheat in the winter wheat area to be detected.

[0096] Specifically, the dual-phase optical remote sensing images and digital elevation model data are input into the winter wheat detection model. First, the number of channels of the dual-phase optical remote sensing images and digital elevation model data are changed to 64 respectively through the residual module, and the size remains unchanged; then the features are extracted through the multimodal feature fusion module.

[0097] like Figure 3 As shown, Figure 3 Schematic diagram of the structure of the multimodal feature fusion module. The optical branch of the multimodal feature fusion module consists of a maximum pooling submodule, a deformable convolutional network block and a residual network. The elevation branch consists of a maximum pooling submodule and a convolutional block. The output of the convolutional block is connected to the output of the deformable convolutional network block along the channel direction as the input of the residual network. Taking the i-th layer multimodal feature fusion module as an example, the input optical feature is The elevation feature is The output optical characteristics are The elevation feature is The convolutional block consists of a standard 3×3 convolution with padding of 1 and stride of 1, a batch normalization layer, and a nonlinear rectification activation function layer.

[0098] The specific steps for any multimodal feature fusion module include: inputting the optical features and elevation features into the maximum pooling submodule, performing 2-fold downsampling in the height and width directions through the maximum pooling operation, and obtaining the first pooled optical features and the first pooled elevation features; extracting the features of the pooled elevation features through the convolution block to obtain the second pooled elevation features, and the number of channels of the second pooled elevation features is twice that of the first pooled elevation features; then extracting the features of the first pooled optical features through the deformable convolutional network block to obtain the intermediate optical features; the number of channels of the intermediate optical features is twice that of the first pooled optical features; finally, connecting the second pooled elevation features and the intermediate optical features along the channel direction and inputting them into the residual network to obtain the second pooled optical features; the second pooled optical features and the second pooled elevation features are the optical features and elevation features input to the next multimodal feature fusion module.

[0099] Taking the i-th layer multimodal feature fusion module as an example, assume that the elevation branch input of the i-th layer multimodal feature fusion module is a feature map of shape (C, H, W) After the maximum pooling operation (maxpool) is performed with 2 times downsampling in width and height, it becomes the first pooled elevation feature of (C, H / 2, W / 2), which is then sent to the convolution block (Conv block) to extract features, and the second pooled elevation feature is output through the convolution block module. The number of channels of becomes twice as much as before, and the size becomes (2C, H / 2, W / 2). The multimodal feature fusion module is combined with the residual module to maintain the consistency of features, where the output of the elevation branch is:

[0100]

[0101] The maxpool() in the formula has a pooling kernel of 2 and a stride of 2; the Convblock block consists of a standard 3×3 convolution with a padding of 1 and a stride of 1, a batch normalization (BN) layer, and a nonlinear rectification (ReLU) activation function layer.

[0102] The optical branch consists of a maximum pooling submodule, a deformable convolutional network block (DCN block) and a residual network. The deformable convolutional network block includes a convolutional layer, a batch normalization layer, a nonlinear rectification activation function layer and a variable convolutional block; specifically, Figure 4 As shown, Figure 4: is a schematic diagram of the structure of a deformable convolutional network block, which includes a 3×3 convolution layer, a batch normalization (Batch Normalization, BN) layer, a nonlinear rectified linear unit (Rectified Linear Unit, ReLU), a deformable convolutional block (Deformable Convolutional Networks version 2, DCNv2) and a batch normalization layer and a nonlinear activation function; for example, the input of the deformable convolutional network block is The output is

[0103] Taking the i-th layer multimodal feature fusion module as an example, let the input feature map of the optical branch of the i-th layer multimodal feature fusion module be The maximum pooling operation is used to perform 2x downsampling in width and height to obtain the first pooled optical feature. Then, the first pooled optical feature is extracted in the deformable convolutional network block to extract the remote sensing image features, and the number of channels of the remote sensing image feature map is doubled to obtain the intermediate optical feature. With the second pooled elevation feature from the elevation branch After connecting along the channel direction, it is sent to the residual structure, and the final output is the second pooled optical feature The calculation formula for the optical branch is as follows:

[0104]

[0105] The calculation formulas of DCNblock and Resblock are as follows; among them, DCNv2 is a deformable convolution.

[0106] DCNblock(x)=ReLU(BN(DCNv2(x))) (4)

[0107] Resblock(x)=x+Convblock(Convblock(x)) (5)

[0108] For any upsampling module, the upsampling module performs a 2-fold upsampling of the input feature map through bilinear interpolation to restore the resolution of the input image. It should be noted that the inputs of upsampling module 2, upsampling module 3, and upsampling module 4 are the output of the previous distribution feature enhancement module and the output of the corresponding optical branch. Therefore, before upsampling, the upsampling module can first connect the two input features through channels to obtain a feature map, and then upsample the feature map.

[0109] The Distribution Feature Enhancement Module (DFEM) uses the weighted attention mechanism in the decoding stage to enhance the feature expression of winter wheat distribution area. Figure 5 As shown, the distribution feature enhancement module includes a global maximum pooling submodule, a global average pooling submodule, a first activation function, a convolutional layer and a second activation function.

[0110] For any distributed feature enhancement module, it specifically includes: passing the upsampled features through the global maximum pooling submodule and the global average pooling submodule respectively, and connecting the features after the global maximum pooling and the global average pooling along the channel direction to obtain an intermediate weight matrix; passing the intermediate weight matrix through the convolution layer and the first activation function to obtain a spatial attention weight matrix; passing the upsampled features through the second activation function to obtain enhanced features; performing point multiplication on the enhanced features, the spatial attention weight matrix and the sampled features in turn to obtain enhanced features; the enhanced features are the input of the next decoding module.

[0111] Specifically, suppose the feature map x of the i-th layer input i The size of is (C, H, W), and after upsampling, it becomes (C, 2H, 2W), and then it is concatenated with the feature map of the same size as the decoder, and the output feature map size becomes (2C, 2H, 2W).

[0112] The above sampled features are x i For example, the upsampled features are respectively passed through global max pooling and global mean pooling, and the features after global max pooling and global mean pooling are connected along the channel direction to obtain the intermediate weight matrix; then the intermediate weight matrix passes through a 3×3 convolution layer and the first activation function sigmold to obtain the spatial attention weight matrix W i sam , the spatial attention weight matrix W i sam Used to enhance the distribution information of wheat.

[0113] W i sam =sigmoid(Conv(concat(globalmaxpool(x i ),globalavgpool(x i ))))(6)

[0114] The distribution feature enhancement module converts x i The feature strength is adjusted by the second activation function tanh()+1, and the enhanced features generated by the second activation function tanh()+1 are and the spatial attention weight matrix Perform point multiplication to generate the distribution feature enhancement weight matrix W i DFEM , strengthen the wheat area, and then W i DFEM With the upsampled feature x i Perform dot multiplication to obtain the enhanced feature x i+1 The calculation process is as follows:

[0115]

[0116] x i+1 =W i DFEM × i (9)

[0117] It should be noted that the enhanced features output by the distribution feature enhancement module 4 are the pixel-level classification of the winter wheat area to be detected output by the decoder.

[0118] In an exemplary embodiment, the fusion segmentation architecture design based on multimodal data integrates data characteristics of different modalities by constructing a multimodal feature fusion module and a distribution feature enhancement module to enhance the feature expression capability of the crop area.

[0119] Winter wheat detection based on multimodal fusion segmentation network is mainly divided into three parts: input, training and output; Figure 6 As shown, Figure 6 A flow chart of a method for detecting winter wheat under complex terrain conditions, the method comprising:

[0120] S601, multimodal dataset construction.

[0121] The fused dual-temporal optical remote sensing images and digital elevation model data were obtained, and divided into multimodal training dataset, multimodal test dataset and multimodal validation dataset in the ratio of 6:2:2.

[0122] S602, multimodal feature fusion.

[0123] S603, feature enhancement and restoration.

[0124] According to the encoder and decoder, a fusion segmentation network for multimodal data is constructed; the encoder consists of two residual modules and four multimodal feature fusion modules with a dual-branch structure; the dual-branch structure includes an optical branch and an elevation branch; the decoder consists of four decoding modules connected, each of which includes an upsampling module and a distribution feature enhancement module; the inputs of the last three upsampling modules in the decoder are respectively connected to the outputs of the first three optical branches in the encoder. See Figure 2-Figure 5 .

[0125] S604, model training and loss optimization.

[0126] The multimodal training dataset is input into the fusion segmentation network, and first passes through the multimodal feature fusion module in the encoder to capture the spectral characteristics (DCN) and terrain elevation information (DEM) of the crop. Then, in the decoder stage, the distribution feature enhancement module is used to further enhance the response to the sparsely broken winter wheat area. During the training process, the cross entropy loss function is used for optimization. The error of the model is gradually reduced. In each training iteration, the loss function calculates the prediction error of each pixel point, and continuously updates the parameters of the model through back propagation to optimize the segmentation performance. As the training progresses, the model's detection accuracy of the winter wheat area is gradually improved, and the error is gradually reduced, ensuring that small areas in complex terrain can also be accurately captured.

[0127] S605, result generation and performance evaluation.

[0128] The model with the smallest loss on the multimodal validation dataset is saved as the best model, and then the model is tested on the multimodal test dataset. After passing through the decoder output layer, the interval result of winter wheat is generated. The result shows the distribution of winter wheat under complex terrain in the form of a pixel-level segmentation map.

[0129] This application fully considers the spatial distribution characteristics of winter wheat under complex terrain, utilizes the complementary advantages of multimodal data, and solves the problem that traditional crop segmentation methods are difficult to capture small crop areas and lose spatial distribution information in complex terrain by constructing a multimodal feature fusion module and a distribution feature enhancement module. It effectively realizes the precise detection of multi-scale and irregular crop areas, and ultimately significantly improves the accuracy and stability of winter wheat distribution detection under complex terrain conditions.

[0130] In the winter wheat detection method under complex terrain conditions provided by the present invention, a fusion segmentation structure is designed, and the dual-phase optical remote sensing image and digital elevation model data are used as a multimodal training data set to train the fusion segmentation structure to obtain a winter wheat detection model, so as to determine the detection result of the winter wheat area to be identified through the winter wheat detection model.

[0131] The fusion segmentation structure includes an encoder and a decoder. In the encoder, two residual modules can effectively extract the features of dual-phase optical remote sensing images and digital elevation model data, which enhances the model's expression ability. The dual-branch structure of the multimodal feature fusion module effectively fuses the optical features and elevation features, allowing the model to use data from different sources at the same time.

[0132] Moreover, in the decoder, the upsampling module and the distributed feature enhancement module can better restore the spatial resolution and enhance the feature expression ability. At the same time, the output of the optical branch is connected with the input of the upsampling module of the decoder, so that the model can fuse data at different levels and utilize the features of each level to make the feature transfer more comprehensive, effectively realizing the precise identification of multi-scale and irregular crop areas, thereby improving the accuracy of winter wheat detection results in complex terrain areas.

[0133] In addition, since the dual-phase optical remote sensing images and digital elevation model data respectively fuse the information obtained by different sensors and make full use of the complementary information of the dual-phase, they can provide a more comprehensive and accurate scene understanding, enabling the model to understand the research object more efficiently and deeply, thereby improving the accuracy of the winter wheat detection model and thus improving the accuracy of the winter wheat detection results.

[0134] The above is a method for detecting winter wheat under complex terrain conditions provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding device for detecting winter wheat under complex terrain conditions, such as Figure 7 shown.

[0135] Figure 7 A schematic diagram of a winter wheat detection device under complex terrain conditions provided by the present invention, the device 700 comprises:

[0136] The training set acquisition module 701 is used to acquire a multimodal training data set; the multimodal training data set includes sample dual-phase optical remote sensing images and sample digital elevation model data corresponding to a preset winter wheat area; the digital elevation model data is used to reflect the terrain information of the preset winter wheat area;

[0137] A construction module 702 is used to construct a fusion segmentation network for multimodal data according to an encoder and a decoder; the encoder is composed of two residual modules and four multimodal feature fusion modules of a dual-branch structure; the dual-branch structure includes an optical branch and an elevation branch; the decoder is composed of four decoding modules connected, each decoding module includes an upsampling module and a distribution feature enhancement module; the inputs of the last three upsampling modules in the decoder are respectively connected to the outputs of the first three optical branches in the encoder;

[0138] A training module 703 is used to train the fusion segmentation network through a multimodal training data set to obtain a winter wheat detection model;

[0139] The data acquisition module 704 is used to acquire dual-phase optical remote sensing images and digital elevation model data of the winter wheat area to be detected;

[0140] The detection module 705 is used to input the dual-phase optical remote sensing image and the digital elevation model data into the winter wheat detection model, and use two residual modules to unify the number of channels of the dual-phase optical remote sensing image and the digital elevation model data to obtain optical features and elevation features; extract the optical features and elevation features through four multimodal feature fusion modules in turn, and splice the feature map obtained by the fourth multimodal feature fusion module to obtain fusion features; use the fusion features and the features output by the corresponding optical branches as inputs of the decoder, and enhance the features of the fusion features and the optical branches through the decoding modules in turn to obtain the detection result of the winter wheat area to be detected output by the decoder; wherein the detection result is a pixel-level classification of the growth distribution of winter wheat in the winter wheat area to be detected.

[0141] For the specific definition of the detection device for winter wheat under complex terrain conditions, please refer to the definition of the detection method for winter wheat under complex terrain conditions mentioned above, which will not be repeated here. Each module in the above-mentioned detection device for winter wheat under complex terrain conditions can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0142] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 Provided is a detection method for winter wheat under complex terrain conditions.

[0143] The present invention also provides Figure 8 The structural diagram of the computer device shown in FIG. Figure 8 As shown in the figure, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 Provided is a detection method for winter wheat under complex terrain conditions.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0145] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A method for detecting winter wheat under complex terrain conditions, characterized in that: include: Obtain a multimodal training dataset; The multimodal training dataset includes sample dual-phase optical remote sensing images and sample digital elevation model data corresponding to the preset winter wheat area; Digital elevation model data is used to reflect the terrain information of the preset winter wheat area; According to the encoder and decoder, a fusion segmentation network of multimodal data is constructed; the encoder consists of two residual modules and four multimodal feature fusion modules with a dual-branch structure; the dual-branch structure includes an optical branch and an elevation branch; The decoder is composed of four decoding modules connected together, each of which includes an upsampling module and a distribution feature enhancement module; the inputs of the last three upsampling modules in the decoder are respectively connected to the outputs of the first three optical branches in the encoder; The fusion segmentation network is trained using a multimodal training dataset to obtain a winter wheat detection model. Obtain dual-temporal optical remote sensing images and digital elevation model data of the winter wheat area to be inspected; The dual-phase optical remote sensing image and the digital elevation model data are input into the winter wheat detection model, and two residual modules are used to unify the number of channels of the dual-phase optical remote sensing image and the digital elevation model data to obtain optical features and elevation features; the optical features and elevation features are sequentially extracted through four multimodal feature fusion modules, and the optical features and elevation features extracted through the fourth multimodal feature fusion module are spliced ​​to obtain fused features; the fused features and the features output by the corresponding optical branches are used as inputs of the decoder, and the fused features and the optical branches are feature enhanced through the decoding modules in turn to obtain the detection results of the winter wheat area to be detected output by the decoder; the detection results are pixel-level classification of the growth distribution of winter wheat in the winter wheat area to be detected.

2. The method according to claim 1, characterized in that The obtaining of a multimodal training data set includes: Obtain sample optical remote sensing images and sample digital elevation model data of the preset winter wheat area at two time points; The two sample optical remote sensing images are band-fused to obtain a sample dual-phase optical remote sensing image; The sample dual-temporal optical remote sensing images and sample digital elevation model data are used as training samples; The training samples are divided into multimodal training data set, multimodal test data set and multimodal verification data set according to preset proportions.

3. The method according to claim 2, characterized in that The method further comprises: After obtaining the winter wheat detection model, the winter wheat detection model is tested using a validation data set to obtain test results; If the test result does not meet the preset test conditions, the fusion segmentation network will continue to be trained through the multimodal training data set until the winter wheat detection model is obtained.

4. The method according to claim 1, characterized in that: The fusion segmentation network is trained by the multimodal training data set to obtain a winter wheat detection model, including: Input the multimodal training dataset into the fusion segmentation network to obtain the predicted pixel-level classification of the multimodal dataset; The predicted pixel-level classification and the actual pixel-level classification corresponding to the multimodal training dataset are input into the cross entropy loss function to obtain the difference; Iteratively update the fusion segmentation network based on difference, Adam optimizer and cosine annealing mechanism; The iteratively updated fusion segmentation network is verified through the multimodal validation set, and the fusion segmentation network with the smallest loss function value on the multimodal validation set is determined as the winter wheat detection model.

5. The method according to claim 1, characterized in that The optical branch of the multimodal feature fusion module consists of a maximum pooling submodule, a deformable convolutional network block and a residual network, and the elevation branch consists of a maximum pooling submodule and a convolutional block. The output of the convolutional block is connected to the output of the deformable convolutional network block along the channel direction as the input of the residual network. The specific features for any multimodal feature fusion module include: The optical features and elevation features are input into the maximum pooling submodule, and the maximum pooling operation is used to perform 2 times downsampling in the height and width directions to obtain the first pooled optical features and the first pooled elevation features; The pooled elevation feature is extracted through the convolution block to obtain the second pooled elevation feature. The number of channels of the second pooled elevation feature is twice that of the first pooled elevation feature. The first pooled optical feature is extracted through a deformable convolutional network block to obtain an intermediate optical feature; the number of channels of the intermediate optical feature is twice that of the first pooled optical feature; The second pooled elevation features and the intermediate optical features are connected along the channel direction and input into the residual network to obtain the second pooled optical features; the second pooled optical features and the second pooled elevation features are the optical features and elevation features input into the next multimodal feature fusion module.

6. The method according to claim 5, characterized in that The deformable convolutional network block includes a convolutional layer, a batch normalization layer, a nonlinear rectification activation function layer, and a variable convolution block; the convolutional block consists of a standard 3×3 convolution with a padding of 1 and a stride of 1, a batch normalization layer, and a nonlinear rectification activation function layer.

7. The method according to claim 1, characterized in that The distribution feature enhancement module includes a global maximum pooling submodule, a global average pooling submodule, a first activation function, a convolutional layer, and a second activation function; For any distribution feature enhancement module, it specifically includes: The upsampled features are passed through the global maximum pooling submodule and the global average pooling submodule respectively, and the features after global maximum pooling and global average pooling are connected along the channel direction to obtain the intermediate weight matrix; Pass the intermediate weight matrix through the convolution layer and the first activation function to obtain the spatial attention weight matrix; The upsampled features are passed through the second activation function to obtain enhanced features; The enhanced features, spatial attention weight matrix and sampled features are point-multiplied in turn to obtain enhanced features; the enhanced features are the input of the next decoding module.

8. A detection device for winter wheat under complex terrain conditions, characterized in that: include: A training set acquisition module is used to acquire a multimodal training data set; The multimodal training dataset includes sample dual-phase optical remote sensing images and sample digital elevation model data corresponding to the preset winter wheat area; Digital elevation model data is used to reflect the terrain information of the preset winter wheat area; A construction module is used to construct a fusion segmentation network of multimodal data based on an encoder and a decoder; the encoder is composed of two residual modules and four multimodal feature fusion modules of a dual-branch structure; the dual-branch structure includes an optical branch and an elevation branch; The decoder is composed of four decoding modules connected together, each of which includes an upsampling module and a distribution feature enhancement module; the inputs of the last three upsampling modules in the decoder are respectively connected to the outputs of the first three optical branches in the encoder; A training module is used to train the fusion segmentation network through a multimodal training data set to obtain a winter wheat detection model; A data acquisition module is used to acquire dual-phase optical remote sensing images and digital elevation model data of the winter wheat area to be inspected; The detection module is used to input the dual-phase optical remote sensing image and the digital elevation model data into the winter wheat detection model, and use two residual modules to unify the number of channels of the dual-phase optical remote sensing image and the digital elevation model data respectively to obtain optical features and elevation features; the optical features and elevation features are sequentially extracted through four multimodal feature fusion modules, and the feature map obtained by the fourth multimodal feature fusion module is spliced ​​to obtain fusion features; the fusion features and the features output by the corresponding optical branches are used as inputs of the decoder, and the fusion features and the optical branches are sequentially enhanced through the decoding module to obtain the detection result of the winter wheat area to be detected output by the decoder; the detection result is a pixel-level classification of the growth distribution of winter wheat in the winter wheat area to be detected.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images