A method for extracting spatial distribution of lodging winter wheat based on dual-modal feature fusion

Abstract

The present application discloses a method for extracting the spatial distribution of lodging winter wheat using dual-modal feature fusion, which relates to the field of agricultural remote sensing monitoring technology. The method comprises the following steps: establishing a dual-stream semantic segmentation network DMFCFNet and training the network using a constructed sample data set; inputting pre-processed remote sensing images and DSM elevation images into the network in parallel to obtain semantic texture feature information of remote sensing images of different scales and spatial structure feature information of DSM elevation images; performing feature correction and fusion on the acquired semantic texture feature information of remote sensing images and spatial structure feature information of DSM elevation images; and performing segmentation prediction on the fused feature information to extract the spatial distribution information of lodging areas of winter wheat. By correcting and fusing the semantic texture features of remote sensing images and DSM elevation features, combined with multi-scale information aggregation processing, the accuracy of extracting lodging areas of winter wheat is effectively improved, and the problems of missed detection of small targets and blurred boundaries are improved.

Description

Technical Field

[0001] The present application relates to the field of agricultural remote sensing monitoring technology, and in particular to a method for extracting the spatial distribution of lodged winter wheat by fusion of dual-modal features. Background Art

[0002] Winter wheat is one of my country's major grain crops, and its stable and high yields are of strategic importance to national food security and the optimization of agricultural resource allocation. In actual production, lodging of winter wheat is a major constraint on its high, stable, and high-quality yields. Using unmanned aerial vehicle (UAV) remote sensing imagery to quickly and accurately obtain spatial distribution data on lodging has become a matter of great concern to agricultural professionals.

[0003] Unmanned aerial vehicle (UAV) remote sensing, with its sub-meter spatial resolution and flexible acquisition capabilities, can offset the shortcomings of satellite remote sensing in terms of spatial and temporal resolution, providing a crucial data foundation for precision agriculture research. Early methods for extracting the spatial distribution of fallen winter wheat from UAV imagery primarily relied on features such as color, texture, and shape, using image processing algorithms such as threshold-based segmentation, watershed algorithms, and edge detection. However, these algorithms rely on manually selected features, have low automation, and are susceptible to interference from environmental factors such as texture and terrain, resulting in limitations. With the development of machine learning, researchers have applied algorithms such as support vector machines, random forests, and decision trees to UAV remote sensing image segmentation. By learning existing features from the data and building classification models based on the training data, they have improved image segmentation accuracy. However, with advances in remote sensing technology, the semantic information in images has become richer, leading to more prominent issues such as the same object having different spectra and scale imbalance between classes. This can lead to missegmentation when dealing with complex terrain or areas with similar features.

[0004] In recent years, convolutional neural networks have made breakthrough progress in the field of remote sensing image processing due to their excellent feature learning and extraction capabilities. Through a multi-level feature abstraction mechanism, key semantic features such as texture, spectrum, and geometry in remote sensing images can be more effectively extracted, improving the accuracy of pixel-by-pixel segmentation of images. However, when dealing with the extraction of winter wheat lodging areas in remote sensing images, existing methods still face problems such as small lodging target area, discontinuous distribution, and significant scale changes. They fail to fully consider the nonlinear correlation characteristics of lodging areas between different modalities, resulting in low efficiency of cross-modal feature interaction, which affects the recognition performance of lodging areas. Moreover, the current multi-scale feature extraction network has difficulty in dynamically adjusting the receptive field range, and there is a problem of insufficient integration of local detail features and global context information, which leads to the gradual dilution of small-scale lodging area features in the deep network, making the segmentation results prone to regional breaks, blurred boundaries, and missed detection of small targets. Summary of the Invention

[0005] In order to solve the above technical problems, this application proposes the following technical solutions:

[0006] In a first aspect, the present invention provides a method for extracting the spatial distribution of lodged winter wheat using bimodal feature fusion, comprising:

[0007] Establish a two-stream semantic segmentation network DMFCFNet and use the constructed training sample dataset to train DMFCFNet. Then, obtain the optimal DMFCFNet after testing it with the test sample dataset.

[0008] The pre-processed remote sensing images and DSM elevation images are input into DMFCFNet in parallel to obtain the semantic texture feature information of remote sensing images of different scales and the spatial structure feature information of DSM elevation images;

[0009] Correcting and fusing the semantic texture feature information of the remote sensing image and the spatial structure feature information of the DSM elevation image to obtain fused feature information;

[0010] The fused feature information of multiple scales is segmented and predicted to extract the spatial distribution information of the winter wheat lodging area.

[0011] In one possible implementation, the DMFCFNet includes: an encoder and a decoder, the encoder includes a remote sensing image branch and a DSM elevation image branch, the remote sensing image branch and the DSM elevation image branch include multiple stage residual modules and multiple cross-module feature correction modules, each stage residual module is connected to a corresponding cross-module feature correction module, and the output of the cross-module feature correction module is connected to an adaptive bimodal feature fusion module;

[0012] The remote sensing image and the DSM elevation image are input in parallel to the residual modules of the remote sensing image branch and the DSM elevation image branch, and then feature alignment and complementation are achieved through a cross-module feature correction module;

[0013] The adaptive dual-modal feature fusion module adaptively fuses the corrected remote sensing image features and the DSM elevation image features;

[0014] The decoder includes a multi-scale aggregation decoding module and a classification module. The multi-scale aggregation decoding module is connected to each adaptive bimodal feature fusion module in the encoder and is used to receive the fusion features output by different adaptive bimodal feature fusion modules. The multi-scale aggregation decoding module is connected to the classification module. The decoder is used to perform segmentation prediction on the fused feature information of different scales to achieve classification.

[0015] In a possible implementation, establishing a dual-stream semantic segmentation network DMFCFNet and training the DMFCFNet using the constructed training sample dataset includes:

[0016] Determine the hyperparameters during training and initialize each parameter in the network;

[0017] The selected training image-DSM data sample-label pair is input into DMFCFNet as training sample data;

[0018] Use DMFCFNet to perform a forward calculation on the current training data and calculate the loss according to the Loss function;

[0019] The SGD algorithm is used to update the parameters of DMFCFNet until the loss function is less than the specified expected value or the loss value no longer changes.

[0020] In one possible implementation, the calculation formula of the Loss loss function is:

[0021]

[0022]

[0023]

[0024] in, is the cross entropy loss, is the Dice loss, and is a hyperparameter, n is the number of samples, is the label value of the i-th sample, is the model's predicted value for the i-th sample, and N is the total number of pixels.

[0025] In one possible implementation, the pre-processed remote sensing image and the DSM elevation image are inputted into DMFCFNet in parallel to obtain semantic texture feature information of remote sensing images of different scales and spatial structure feature information of DSM elevation images, including:

[0026] After the pre-processed remote sensing image and DSM elevation image are input into DMFCFNet in parallel, the remote sensing image branch input layer uses large kernel convolution to construct a wide receptive field, captures the low-frequency texture features in the input remote sensing image, and reduces the size of the remote sensing image by downsampling;

[0027] The DSM elevation image branch adjusts the number of channels of the input layer convolution kernel for single-channel elevation data, reveals the spatial correlation between terrain undulation and crop lodging through elevation gradient features, and reduces the size of the DSM elevation image through downsampling;

[0028] The first-stage residual module retains the high-resolution details of remote sensing images and DSM elevation images through skip connections, compresses the feature map with maximum pooling at the end, and extracts deep semantic features by expanding the residual unit.

[0029] In a possible implementation, the semantic texture feature information of the remote sensing image and the spatial structure feature information of the DSM elevation image are feature corrected and fused to obtain fused feature information, including:

[0030] Global average pooling and maximum pooling are performed on the input remote sensing image and DSM elevation image respectively to capture the channel-level mean information and extreme value response, and then a nonlinear transformation is performed through the convolution module to generate channel attention weights;

[0031] The semantic clues of the two pools are fused by element-by-element addition, and finally the adaptive weighting of the channel dimension is achieved through the sigmoid activation function, and the output generates the corrected features. and ;

[0032] The rectified features are subjected to multi-scale depth-wise separable convolution using standard convolution and asymmetric convolution kernels. and to process;

[0033] The corrected features and Dynamic complementarity is achieved through cross-modal attention modules;

[0034] The adaptive bimodal feature fusion module realizes the adaptive fusion of remote sensing image and DSM elevation image features through structured cross-modal interaction and dynamic spatial fusion mechanism.

[0035] In a possible implementation, the corrected feature and Dynamic complementarity is achieved through cross-modal attention modules, including:

[0036] Stitching the dual-modal features of remote sensing images and DSM elevation images;

[0037] The cross-modal association is extracted by 3×3 convolution and RELU activation function, and the spatial weight is generated by 1×1 convolution. and ;

[0038] Using two learnable parameters and The bimodal features are weighted and fused to generate output features. The calculation formula is:

[0039]

[0040]

[0041] in, 、 Represent remote sensing images and DSM elevation images respectively, and represents the learnable parameters, 、 Represents the corrected remote sensing image features and the corrected DSM elevation image features, and represents the spatial weight, Represents pixel-by-pixel multiplication.

[0042] In one possible implementation, the adaptive bimodal feature fusion module realizes the adaptive fusion of remote sensing image and DSM elevation image features through structured cross-modal interaction and dynamic spatial fusion mechanism, including:

[0043] The adaptive dual-modal feature fusion module adopts a dual-branch parallel architecture to process the corrected remote sensing image features and DSM elevation image features respectively;

[0044] Each branch uses an independent convolution module to expand the input channel dimension and divide it into three groups of feature maps along the channel dimension: guidance feature, correlation feature and content feature;

[0045] The guiding features of the remote sensing image are spliced ​​with the correlation features of the DSM elevation image and the content features of the DSM elevation image along the channel dimension to form the first fusion feature;

[0046] At the same time, the guiding features of the DSM elevation image are reversely stitched with the correlation features and content features of the remote sensing image to form the second fusion feature;

[0047] Compressing the first fused feature and the second fused feature to the hidden channel dimension respectively through convolution to generate a first compact representation and a second compact representation;

[0048] The first compact representation and the second compact representation are mapped back to the original number of channels through the projection layer, and are added pixel by pixel with the input rectified remote sensing image features and DSM elevation image features respectively;

[0049] A spatial dynamic weight map is generated through a spatial adaptive feature fusion network to achieve adaptive fusion of remote sensing image and DSM elevation image features.

[0050] In a possible implementation, the segmentation and prediction of the fused feature information at multiple scales to extract the spatial distribution information of the winter wheat lodging area includes:

[0051] The fused feature information of multiple inputs at different scales is aligned in the channel dimension, and the features of each layer are uniformly mapped using parallel convolutional layers;

[0052] After eliminating cross-modal feature distribution differences through group normalization and GELU activation function, a top-down information fusion path is adopted, starting from the highest-level semantic features, upsampling is achieved through parameterized transposed convolution, and the output features are channel-wise concatenated with the adjacent low-level features, and cross-level feature fusion is completed through the convolution kernel;

[0053] A cascaded structure is used to process the output of each layer, unifying all layer features to the original input resolution through bilinear interpolation, and densely splicing along the channel dimension to form aggregated features;

[0054] A Softmax function is set before the final output layer to normalize the channel dimension and generate a pixel-level probability distribution map;

[0055] The fully connected information of each layer feature is retained according to the pixel-level probability distribution map, so as to realize the extraction of the spatial distribution information of the lodging winter wheat area.

[0056] Compared with the prior art, the present invention has the following advantages:

[0057] This application constructs a convolutional neural network model based on the encoder-decoder structure, and introduces a cross-module feature correction module and an adaptive bimodal feature fusion module in the encoder. The cross-module feature correction module corrects the remote sensing image and the DSM elevation image through the complementary features between the modalities, and reduces the loss of small-scale lodging area features in the deep network by dynamically adjusting the receptive field range, thereby improving the segmentation accuracy. The adaptive bimodal feature fusion module independently extracts the guiding features of the UAV remote sensing image and the DSM elevation image, and adopts a cross-grafting strategy to splice and reorganize the spectral guiding features and the elevation response features to achieve adaptive mixing of bimodal features, enhance the discrimination ability of the fusion features, and improve the extraction accuracy of the lodging winter wheat area. The multi-scale aggregation decoding module restores spatial details step by step through learnable deconvolution operations, uses channel alignment and group normalization to eliminate the differences in feature distribution at each scale, and adopts a progressive fusion strategy to aggregate semantic information and texture features at different levels, simultaneously strengthening global context perception and local boundary sharpening, improving the fuzzy phenomenon of lodging area boundary extraction, and thus achieving accurate reconstruction of the lodging area from coarse to fine. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic flow chart of a method for extracting the spatial distribution of lodged winter wheat using dual-modal feature fusion provided in an embodiment of the present application;

[0059] Figure 2 The DMFCFNet network structure diagram provided in the embodiment of this application;

[0060] Figure 3 Flowchart of the DMFCFNet training method provided in the embodiment of the present application;

[0061] Figure 4 A structural diagram of a cross-module feature correction module provided in an embodiment of the present application;

[0062] Figure 5 A structural diagram of the adaptive dual-modal feature fusion module provided in an embodiment of the present application;

[0063] Figure 6 A structural diagram of a multi-scale aggregation decoding module provided in an embodiment of the present application;

[0064] Figure 7 Schematic diagram of comparison results of different models provided in the embodiments of the present application;

[0065] Figure 8 The model test result diagram provided in the embodiment of this application. DETAILED DESCRIPTION

[0066] The present invention will be described below with reference to the accompanying drawings and specific implementation methods.

[0067] See also Figure 1 The method for extracting the spatial distribution of lodging winter wheat by fusing dual-modal features in this embodiment includes:

[0068] S101, establish a two-stream semantic segmentation network DMFCFNet and use the constructed training sample dataset to train DMFCFNet, and obtain the optimal DMFCFNet after testing with the test sample dataset.

[0069] The DMFCFNet network of this application is designed to extract the spatial distribution information of lodging winter wheat. It constructs an end-to-end segmentation framework consisting of an adaptive bimodal feature fusion module and a multi-scale aggregation decoder. The overall framework is as follows: Figure 2 As shown in the figure, it can be divided into four parts: bimodal input, encoder, decoder and output. Among them, the encoder includes a remote sensing image branch and a DSM elevation image branch. The remote sensing image branch and the DSM elevation image branch include multiple stage residual modules and multiple cross-module feature correction modules. The end of each residual module is embedded with a convolution layer, a maximum pooling layer, a batch normalization layer and a RELU activation function. Each stage residual module is connected to a cross-module feature correction module. The output of the cross-module feature correction module is connected to the adaptive bimodal feature fusion module. The decoder includes a multi-scale aggregation decoding module and a classification module. The multi-scale aggregation decoding module is connected to each adaptive bimodal feature fusion module in the encoder to receive the fusion features output by different adaptive bimodal feature fusion modules. The multi-scale aggregation decoding module is connected to the classification module.

[0070] In this embodiment, the preprocessed remote sensing image and DSM elevation image are input in parallel into the residual modules of the remote sensing image branch and the DSM elevation image branch for feature extraction, and then feature alignment and complementation are achieved through the cross-module feature correction module. The corrected remote sensing image features and DSM elevation image features are then input into the adaptive bimodal feature fusion module for adaptive fusion, and finally the fused feature information of different scales is input into the decoder for segmentation prediction to achieve classification.

[0071] Yanzhou District, located in the heart of the Huanghuaihai Plain, boasts a flat and open terrain and deep, fertile soils, primarily tidal and brown soils, making it ideal for winter wheat cultivation. Its warm temperate monsoon climate offers distinct four seasons, an average annual temperature of 13.5°C, simultaneous rainfall and heat, and abundant sunlight and heat, providing excellent natural conditions for winter wheat growth. Furthermore, Yanzhou District boasts abundant arable land resources and well-developed agricultural infrastructure, making it a key grain production base in Shandong Province and a representative region. To accurately extract the spatial distribution of winter wheat lodging, this example selected Beilijia Village in Yanzhou District, a typical example of lodging-affected areas, as the core research area. Aerial drones were used to capture the affected areas of Beilijia Village, acquiring high-resolution remote sensing images and DSM elevation images. The aerial photography covered the main farmland distribution area, ensuring the integrity of the sample's spatial distribution. To avoid cloud cover, data collection was conducted under clear, cloudless conditions with stable solar radiation intensity. Aerial photography parameter settings: Flight altitude set to 10m, flight speed set to 2.6m / s, GSD set to 0.5m / pixel, heading overlap set to 80%, and lateral overlap set to 70%. UAV imagery data includes remote sensing imagery and DSM elevation imagery. Remote sensing imagery contains three bands: red, green, and blue. DSM includes elevation data representing the height of features.

[0072] After the drone image data is acquired, the drone image is geometrically corrected and radiometrically corrected to reduce the effects of geometric distortion and lighting differences, ensure that the image corresponds to the real object in space, and eliminate errors caused by changes in flight altitude, angle, and lighting conditions. Subsequently, image stitching technology is used to seamlessly combine multiple aerial images into a complete winter wheat plot map. During the stitching process, feature matching algorithms are used to automatically identify overlapping areas for precise alignment. The vector data is marked using a manual outlining method to generate a marked map containing two categories of winter wheat: lodged and non-lodged winter wheat. The pre-processed image and the marked map are segmented using python slicing code to ensure that the original image and the marked map are consistent. Figure 1 One-to-one correspondence facilitates subsequent model training and verification. Based on this, lodging sample data, elevation sample data, and test lodging data are composed. Among them, lodging sample data and elevation sample data are used to train the model, and test lodging data is used to test the model.

[0073] In order to train DMFCFNet, the SGD algorithm is used as the training algorithm. The specific training steps include: determining the hyperparameters in the training process and , and initialize each parameter in the network, input the selected training image-DSM data sample-label pair as training sample data into DMFCFNet, use DMFCFNet to perform a forward calculation on the current training data, and calculate the loss according to the Loss loss function, use the SGD algorithm to update the parameters of DMFCFNet, complete a training process, repeat the forward calculation on the current training data, and calculate the loss according to the Loss loss function, until the loss function is less than the specified expected value or the loss value no longer changes.

[0074] This embodiment addresses the problem of uneven spatial distribution of lodging winter wheat areas in UAV remote sensing images. A loss function is designed based on cross entropy loss and Dice loss, and is defined as:

[0075]

[0076] in, For the overall loss, is the cross entropy loss, is the Dice loss, and is a hyperparameter, and the sum of the two is 1.

[0077] Cross-entropy loss emphasizes the consistency of class probabilities by calculating the logarithmic difference between the true label and the predicted probability. This effectively measures the model's prediction accuracy for each sample class, even when the classes are unevenly distributed. A smaller cross-entropy loss indicates a closer match between the predicted probability and the true label. Its calculation formula is:

[0078]

[0079] Where n is the number of samples, is the label value of the i-th sample, is the model's predicted value for the i-th sample.

[0080] Dice loss emphasizes the similarity of local regions by calculating the ratio of the intersection and union of the predicted region and the actual target region. It can effectively capture the morphological characteristics of smaller targets even when the target regions are unevenly distributed. The closer the Dice coefficient is to 1, the better the segmentation effect. Its calculation formula is:

[0081]

[0082] in, is the label value of the i-th sample, is the model's predicted value for the i-th sample, and N is the total number of pixels, which is equal to the number of pixels in a single image multiplied by batchsize.

[0083] In this embodiment, in order to select the appropriate and The strategy of gradual modification is adopted to As a starting point, and then gradually increase Value, each modification takes After organizing multiple rounds of training, we finally determined a group of suitable and value.

[0084] like Figure 3 As shown, in this embodiment, lodging sample data, elevation sample data and test lodging data are first prepared. The model is trained using the constructed training sample data set. During the training process, starting from initializing the model parameters, inputting samples, performing forward calculations, and calculating the loss according to the Loss loss function, the SGD algorithm is used to update the model parameters until the loss function is less than the specified expected value or the loss value no longer changes, and the optimal model is obtained. After the model is determined, field extraction is performed, and then spatial distribution extraction is performed for winter wheat, followed by accuracy evaluation. The model performance is analyzed based on the results of the accuracy evaluation, and the weight file is saved. After the training is completed, the model is tested using the test lodging data, and the spatial distribution information of lodging winter wheat is output.

[0085] S102: The pre-processed remote sensing image and the DSM elevation image are inputted into DMFCFNet in parallel to obtain semantic texture feature information of remote sensing images of different scales and spatial structure feature information of DSM elevation images.

[0086] After preprocessing remote sensing images and DSM elevation images are fed into the DMFCFNet in parallel, the remote sensing image branch input layer uses a large 7×7 kernel convolution to construct a wide receptive field, effectively capturing low-frequency texture information in the image and identifying features of fallen winter wheat. Downsampling reduces the image size and reduces subsequent computational effort. The first-stage residual module, Res1, preserves high-resolution details through skip connections and compresses the feature map with 3×3 max pooling at the end, preserving key information while effectively suppressing noise. The second to fourth-stage residual modules, Res2–Res4, employ a 1×1–3×3–1×1 compression-extraction-expansion residual unit to gradually mine deep semantic features. Dilated convolutions with dilation rates of 2 and 4 are applied in Enc3 and Enc4, respectively, to expand the receptive field while maintaining feature map resolution, thereby enhancing the ability to model the cross-regional context of scattered fallen patches.

[0087] The DSM elevation image branch adjusts the number of channels in the input convolution kernel to 1 for single-channel elevation data, revealing the spatial correlation between terrain undulation and crop lodging through elevation gradient features. The remaining architecture of this branch is consistent with the remote sensing image branch, including the same downsampling strategy and compression-extraction-expansion residual unit. Furthermore, a batch normalization layer (BN) and a Relu activation function are embedded at the end of each residual block in both branches to eliminate inter-channel distribution differences and enhance nonlinear representation capabilities.

[0088] S103 , performing feature correction and fusion on the semantic texture feature information of the remote sensing image and the spatial structure feature information of the DSM elevation image to obtain fused feature information.

[0089] In order to solve the nonlinear correlation problem of the lodging area in the multimodal feature space, this embodiment introduces a cross-modal feature correction module (CFCM) after each level of feature extraction. This module uses CPCA attention as the core building block for the remote sensing image feature map and the DSM elevation image feature map at the same level, and incorporates a learnable weight allocation strategy. While strengthening the single-modal feature discrimination ability, it also dynamically models the complementary relationship between modalities. The structure of the CFCM module is as follows: Figure 4 As shown, it includes: channel attention module, standard convolution module, depth-separable convolution layer, spatial attention module and cross-modal attention module.

[0090] First, input the remote sensing image feature map and DSM elevation image feature map Global average pooling and maximum pooling are performed separately to capture the mean information and extreme value response at the channel level. Then, two layers of 1×1 convolution are used for nonlinear transformation to generate channel attention weights and fuse the semantic clues of the two poolings by element-by-element addition. Finally, the sigmoid activation function is used to achieve adaptive weighting of the channel dimension and output the generated corrected features. and .

[0091] The rectified features are subjected to multi-scale depth-wise separable convolution using 5×5 standard convolution, (1, 7) / (7, 1), (1, 11) / (11, 1) and (1, 21) / (21, 1) asymmetric convolution kernels. and 5×5 convolution is used to capture the basic spatial relationships of the local neighborhood. Asymmetric convolution kernels decompose the spatial dimension, reducing computational effort while covering longer spatial dependencies. This effectively extracts structural features of varying aspect ratios, adapting to the diverse morphology of lodging winter wheat. All spatial convolution layers utilize grouped convolution to enhance local feature interactions. Finally, multi-scale features are fused through residual connections, and a spatial attention map is generated through 1×1 convolution and ReLU activation, optimizing the spatial structure of the input features.

[0092] In order to achieve collaborative optimization between modalities, the corrected features and Dynamic complementation is achieved through cross-modal cross-attention. First, the bimodal features of the remote sensing image feature map and the DSM elevation image feature map are spliced, and the cross-modal correlation is extracted through 3×3 convolution and RELU activation function. Finally, the spatial weight is generated by 1×1 convolution. and , which is used to identify the degree of dependence of each pixel on the features of another modality. Finally, two learnable parameters are used and The bimodal features are weighted and fused to generate output features. The calculation formula is:

[0093]

[0094]

[0095] in, 、 Represent remote sensing images and DSM elevation images respectively, and represents the learnable parameters, 、 Represents the corrected remote sensing image features and the corrected DSM elevation image features, and represents the spatial weight, Represents pixel-by-pixel multiplication.

[0096] In addition, to further reduce the offset problem caused by inter-modal differences that may occur during training, the module adds a Dropout layer after the output of each layer of feature maps, alleviating the overfitting problem and improving the robustness and generalization ability of the model.

[0097] The adaptive dual-modal feature fusion module AMFM realizes the adaptive fusion of remote sensing image and DSM elevation image features through structured cross-modal interaction and dynamic spatial fusion mechanism. The structure diagram is shown in the figure. Figure 5As shown in the figure, the AMFM module employs a dual-branch parallel architecture, processing rectified remote sensing image features and DSM elevation image features separately. Each branch uses an independent 1×1 convolution to expand the input channel to three hidden channels. The input is then split along the channel dimension into three feature maps: guidance features (which locate key areas for cross-modal information interaction), correlation features (which establish a cross-modal feature similarity metric), and content features (which carry modality-specific information to be transferred). Bidirectional interaction is then achieved through cross-modal feature concatenation. The guidance features of the remote sensing image are concatenated with the correlation features and content features of the DSM along the channel dimension to form a fused feature, allowing the spectral features to incorporate terrain structure information. Simultaneously, the DSM guidance features are concatenated with the correlation features and content features of the remote sensing image to incorporate their texture detail. Through cross-modal feature exchange, the complementary relationship between the two modalities is modeled. The fused high-dimensional features are then compressed to the hidden channel dimension using 1×1 convolutions to generate a compact representation that reduces computational complexity. To avoid the loss of small-scale lodging features during dimensionality reduction, the model maps the compact representation back to the original number of channels through a projection layer and adds it pixel by pixel to the original input features.

[0098] To accommodate the mixed distribution of scattered patches and large tracts of fallen winter wheat within the lodging area, a spatially adaptive fusion network generates a spatial dynamic weight map, enabling pixel-level adaptive adjustment of the modal fusion ratio. This network concatenates the residual-enhanced features along the channel dimension, sequentially applying a 3×3 depthwise separable convolution, a Reluctant Unit (RELU) activation function, and a 1×1 convolution to output a weight tensor. A softmax function is applied to this tensor along the channel dimension to generate a spatial weight map, and the final fused features are calculated through weighted summation. This mechanism dynamically adjusts the fusion ratio based on local context, enhancing the weights of remote sensing images in areas with insufficient depth information while relying on depth features in areas with weak texture.

[0099] S104, performing segmentation and prediction on the fused feature information of multiple scales to extract the spatial distribution information of the winter wheat lodging area.

[0100] In this embodiment, the decoder adopts a progressive feature pyramid architecture, which focuses on solving the problem of blurred boundaries and missed detection of small targets caused by scale differences in the lodging area through multi-stage feature interaction and spatial information reconstruction. Figure 6 The figure shows the structure of the multi-scale aggregation decoding module (MAE). The decoder input is the multi-scale feature maps of four different levels extracted by the backbone network and then fused by the adaptive bimodal feature fusion module, covering everything from low-level detail features to high-level semantic features.

[0101] First, the channel-wise alignment layer AlignConv aligns the multi-level input features. Parallel 1×1 convolutional layers are used to uniformly map the features across all layers. Group normalization and the GELU activation function are introduced to eliminate cross-modal feature distribution discrepancies and reduce computational complexity. For feature fusion, the MAE employs a top-down information fusion pathway. Starting with the highest-level semantic features, parameterized transposed convolutions achieve a factor of 2 upsampling. The output features are then concatenated with the adjacent lower-level features channel-wise, and then subjected to 3×3 convolutional kernels for cross-level feature fusion. This stacked process gradually infuses higher-level semantic information into lower-level features, forming a feature pyramid rich in contextual information. The transposed convolutions use a 3×3 kernel size with a stride of 2, along with padding parameters to ensure precise feature map scaling. Compared to traditional bilinear interpolation methods, the learnable parameter mechanism adaptively corrects feature distortions caused by upsampling. The multi-scale feature aggregation module utilizes a cascaded architecture to process the outputs of each pyramid layer. Bilinear interpolation is used to normalize all level features to the original input resolution, followed by dense concatenation along the channel dimension to form aggregated features. After being compressed by a 3×3 convolution, this feature is directly mapped to the target category space by a 1×1 convolution. Furthermore, a Softmax function is applied before the final output layer to normalize the channel dimensions and generate a pixel-level probability distribution map. By retaining the fully connected information of each layer's features, the model's ability to represent multi-scale objects is enhanced, effectively preserving edges when extracting areas of fallen winter wheat.

[0102] In order to verify the above method, the following experiments were designed for verification.

[0103] The U-Net, DeepLabv3+, RedNet, and MMFNet models were selected as comparison models for comparative experiments to analyze the impact of different network structures on the accuracy of generating winter wheat planting area DSMs. Table 1 shows the main characteristics of each model.

[0104]

[0105] Four indicators, Accuracy, Precision, Recall and F1 score, are selected to evaluate the accuracy of the experimental results.

[0106] In the comparative experiments of this example, a computer equipped with an NVIDIA GeForce RTX 4060 GPU was used as the test platform. The deep learning framework selected was PyTorch. The source code of the comparative model was obtained from public literature, while the rest of the code was written in Python. Furthermore, the image processing program required for the comparative experiments was developed using open source libraries such as GDAL and OpenCV.

[0107] In order to verify the effectiveness of different modules, this embodiment sets up four sets of ablation experiments, as shown in Table 2. Experiment 1 directly splices the feature maps of remote sensing images and DSM elevation image data in the channel dimension, and adopts a decoder architecture that only contains upsampling layers and convolution layers. Experiment 2 uses CFCM in the encoder part, Experiment 3 uses AMFM instead of simple splicing and fusion of remote sensing images and DSM elevation image information, and Experiment 4 uses a multi-scale aggregation decoder based on the existing architecture. This experimental design can effectively verify the impact of each module on the accuracy of extracting the spatial distribution of lodged winter wheat. The ablation experiment data show that only adding the CFCM module (Experiment 2) can make the accuracy, precision, recall rate and F1 score reach 92.3%, 88.5%, 89.2% and 88.6% respectively, verifying the improvement effect of feature correction on the problems of blurred boundaries and missed small targets. Experiment 3 performed worse than Experiment 4, which used a multi-scale aggregate decoder, in terms of evaluation metrics. This demonstrates the significant limitations of single-scale reconstruction mechanisms, particularly in small-scale lodging areas. Bilinear interpolation upsampling, due to the gradual attenuation of feature resolution, fails to preserve fine-grained textures, leading to misclassification of lodging winter wheat areas. Building on Experiment 3, the introduction of the MAE module achieved an 89.7% accuracy in extracting lodging patches. This advantage stems from the coordinated optimization of high-level semantic features and low-level detail features. The high-level features provide a prior understanding of the global distribution of lodging areas, while the low-level features reconstruct local geometric patterns through deconvolution. Furthermore, the progressive fusion strategy, by densely stitching multi-scale features, enhances the model's ability to perceive the heterogeneous distribution of lodging areas.

[0108]

[0109] After completing the extraction on the test data using all the comparison methods, 5 image blocks were randomly selected from the test results of Lijia Village to visually demonstrate the extraction effects of the DMFCFNet method and the comparison method. Figure 7 The results of five models generated in five regions are given. Figure 7 (a) is the original remote sensing image; Figure 7 (b) is the labeled image. Figure 7 (c) is the recognition result of the DMFCFNet model. Figure 7 (d) is the recognition result of the MMFNet model. Figure 7 (e) is the recognition result of REDNet model. Figure 7 (f) in the middle is the recognition result of DeepLabv3+ model. Figure 7Figure (g) shows the U-Net model's recognition results, where the green and black areas represent the extracted fallen and non-fallen areas, respectively. The results show that the U-Net model performs the worst, with blurred and broken edges in fallen areas and widespread misidentification and misidentification. The DeepLabv3+ model improves region integrity through multi-scale feature fusion, reducing the misidentification rate. However, boundary segmentation in transitional areas remains inaccurate, with significant underdetection of small, fragmented patches. Although the RedNet model integrates UAV imagery and DSM elevation data, the bimodal features fail to effectively synergize, resulting in increased edge segmentation roughness and susceptibility to shadow interference, limiting its ability to extract small, fallen areas. The MMFNet model, building on RedNet, introduces a 3D convolutional block attention module, improving the integrity and boundary clarity of fallen areas, but still performs poorly on small, fallen winter wheat areas. In contrast, the DMFCFNet model proposed in this example significantly improves the integrity and boundary clarity of fallen area extraction, effectively suppressing misidentification in complex scenes and accurately detecting even small, fragmented patches, achieving the best overall performance.

[0110] Table 3 gives the evaluation indicators of all comparison methods on the entire test set. It can be seen that in the test results of the Lijia Village dataset in Yanzhou District, the indicators of DMFCFNet are significantly better than those of other comparison methods.

[0111]

[0112] As shown in Table 3, the F1 score of the DMFCFNet model is approximately 5.3, 3.2, 2.3, and 1.2 percentage points higher than that of the U-Net, Deeplabv3+, RedNet, and MMFNet models, respectively. Based on these indicators, the DMFCFNet model is well suited for extracting the spatial distribution of fallen winter wheat from UAV remote sensing imagery.

[0113] Based on the constructed DMFCFNet model, this example used 0.5-meter high-resolution drone remote sensing imagery data and a data preprocessing process consistent with the Beilijiazhuang Village dataset to prepare drone and DSM data for the eastern experimental area of ​​Wangfuzhuang Village. Based on the model weights obtained from training in Lijiazhuang Village, a validation experiment was conducted on the winter wheat lodging area in Wangfuzhuang Village, systematically evaluating the model's generalization performance and extraction accuracy on different datasets. Figure 8 The DMFCFNet method effectively distinguishes the non-lost winter wheat and construction land in the eastern part of Wangfuzhuang Village from the lodged winter wheat area, and effectively removes the influence of infrastructure such as field roads and trees, accurately completing the extraction task of lodged winter wheat and verifying the effectiveness and reliability of the DMFCFNet model in different scenarios.

[0114] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0115] The above description is merely a specific embodiment of the present application. Any person skilled in the art may easily conceive of variations or substitutions within the technical scope disclosed in this application, and such variations or substitutions shall be within the scope of protection of this application. The scope of protection of this application shall be subject to the scope of protection of the claims.

Claims

1. A method for extracting the spatial distribution of lodging winter wheat by fusion of dual-modal features, characterized in that: include: A two-stream semantic segmentation network DMFCFNet is established and trained using a constructed training sample dataset. The optimal DMFCFNet is obtained after testing with a test sample dataset. The DMFCFNet comprises an encoder and a decoder. The encoder comprises a remote sensing image branch and a DSM elevation image branch. The remote sensing image branch and the DSM elevation image branch comprise multiple stage residual modules and multiple cross-module feature correction modules. Each stage residual module is connected to a corresponding cross-module feature correction module. The output of the cross-module feature correction module is connected to an adaptive dual-modal feature fusion module. The remote sensing image and the DSM elevation image are input in parallel to the residual modules of the remote sensing image branch and the DSM elevation image branch, and then feature alignment and complementation are achieved through a cross-module feature correction module; The adaptive dual-modal feature fusion module adaptively fuses the corrected remote sensing image features and the DSM elevation image features; The decoder includes a multi-scale aggregation decoding module and a classification module. The multi-scale aggregation decoding module is connected to each adaptive bimodal feature fusion module in the encoder and is used to receive fusion features output by different adaptive bimodal feature fusion modules. The multi-scale aggregation decoding module is connected to the classification module. The decoder is used to perform segmentation prediction and classification on the fused feature information of different scales. The pre-processed remote sensing images and DSM elevation images are input into DMFCFNet in parallel to obtain the semantic texture feature information of remote sensing images of different scales and the spatial structure feature information of DSM elevation images; Correcting and fusing the semantic texture feature information of the remote sensing image and the spatial structure feature information of the DSM elevation image to obtain fused feature information; The fused feature information of multiple scales is segmented and predicted to extract the spatial distribution information of the winter wheat lodging area.

2. The method for extracting spatial distribution of lodging winter wheat by dual-modal feature fusion according to claim 1, characterized in that: The method of establishing a dual-stream semantic segmentation network DMFCFNet and training DMFCFNet using the constructed training sample dataset includes: Determine the hyperparameters during training and initialize each parameter in the network; The selected training image-DSM data sample-label pair is input into DMFCFNet as training sample data; Use DMFCFNet to perform a forward calculation on the current training data and calculate the loss according to the Loss function; The SGD algorithm is used to update the parameters of DMFCFNet until the loss function is less than the specified expected value or the loss value no longer changes.

3. The method for extracting spatial distribution of lodging winter wheat by dual-modal feature fusion according to claim 2, characterized in that: The calculation formula of the Loss loss function is: Among them, CEloss is cross entropy loss, Diceloss is Dice loss, and ω are hyperparameters, n is the number of samples, y i is the label value of the i-th sample, is the model's predicted value for the i-th sample, and N is the total number of pixels.

4. The method for extracting the spatial distribution of lodging winter wheat by dual-modal feature fusion according to claim 1, characterized in that: The pre-processed remote sensing image and DSM elevation image are inputted into DMFCFNet in parallel to obtain semantic texture feature information of remote sensing images of different scales and spatial structure feature information of DSM elevation images, including: After the pre-processed remote sensing image and DSM elevation image are input into DMFCFNet in parallel, the remote sensing image branch input layer uses large kernel convolution to construct a wide receptive field, captures the low-frequency texture features in the input remote sensing image, and reduces the size of the remote sensing image by downsampling; The DSM elevation image branch adjusts the number of channels of the input layer convolution kernel for single-channel elevation data, reveals the spatial correlation between terrain undulation and crop lodging through elevation gradient features, and reduces the size of the DSM elevation image through downsampling; The first-stage residual module retains the high-resolution details of remote sensing images and DSM elevation images through skip connections, compresses the feature map with maximum pooling at the end, and extracts deep semantic features by expanding the residual unit.

5. The method for extracting spatial distribution of lodging winter wheat by dual-modal feature fusion according to claim 1, characterized in that: Correcting and fusing the semantic texture feature information of the remote sensing image and the spatial structure feature information of the DSM elevation image to obtain fused feature information includes: Global average pooling and maximum pooling are performed on the input remote sensing image and DSM elevation image respectively to capture the channel-level mean information and extreme value response, and then a nonlinear transformation is performed through the convolution module to generate channel attention weights; The semantic clues of the two pools are fused by element-by-element addition, and finally the adaptive weighting of the channel dimension is achieved through the sigmoid activation function, and the output generates the corrected feature F′ UAV and F′ DSM ; The rectified feature F′ is subjected to multi-scale depthwise separable convolution using standard convolution and asymmetric convolution kernels. UAV and F′ DSM to process; The corrected feature F′ UAV and F′ DSM Dynamic complementarity is achieved through cross-modal attention modules; The adaptive bimodal feature fusion module realizes the adaptive fusion of remote sensing image and DSM elevation image features through structured cross-modal interaction and dynamic spatial fusion mechanism.

6. The method for extracting the spatial distribution of lodging winter wheat by dual-modal feature fusion according to claim 5, characterized in that: The corrected feature F′ UAV and F′ DSM Dynamic complementarity is achieved through cross-modal attention modules, including: Stitching the dual-modal features of remote sensing images and DSM elevation images; The cross-modal association is extracted by 3×3 convolution and RELU activation function, and the spatial weight W is generated by 1×1 convolution. UAV and W DSM ; Using two learnable parameters and The bimodal features are weighted and fused to generate output features. The calculation formula is: Among them, F UAV 、F DSM Represent remote sensing images and DSM elevation images respectively, and represents the learnable parameter, F′ UAV , F′ DSM Represents the corrected remote sensing image features and the corrected DSM elevation image features, W UAV and W DSM represents the spatial weight, and ⊙ represents pixel-by-pixel multiplication.

7. The method for extracting spatial distribution of lodging winter wheat by dual-modal feature fusion according to claim 5, characterized in that: The adaptive dual-modal feature fusion module achieves adaptive fusion of remote sensing image and DSM elevation image features through structured cross-modal interaction and dynamic spatial fusion mechanism, including: The adaptive dual-modal feature fusion module adopts a dual-branch parallel architecture to process the corrected remote sensing image features and DSM elevation image features respectively; Each branch uses an independent convolution module to expand the input channel dimension and divide it into three groups of feature maps along the channel dimension: guidance feature, correlation feature and content feature; The guiding features of the remote sensing image are spliced ​​with the correlation features of the DSM elevation image and the content features of the DSM elevation image along the channel dimension to form the first fusion feature; At the same time, the guiding features of the DSM elevation image are reversely stitched with the correlation features and content features of the remote sensing image to form the second fusion feature; Compressing the first fused feature and the second fused feature to the hidden channel dimension respectively through convolution to generate a first compact representation and a second compact representation; The first compact representation and the second compact representation are mapped back to the original number of channels through the projection layer, and are added pixel by pixel with the input rectified remote sensing image features and DSM elevation image features respectively; A spatial dynamic weight map is generated through a spatial adaptive feature fusion network to achieve adaptive fusion of remote sensing image and DSM elevation image features.

8. The method for extracting spatial distribution of lodging winter wheat by dual-modal feature fusion according to claim 1, characterized in that: The method of segmenting and predicting the fused feature information of multiple scales to extract the spatial distribution information of the winter wheat lodging area includes: The fused feature information of multiple inputs at different scales is aligned in the channel dimension, and the features of each layer are uniformly mapped using parallel convolutional layers; After eliminating cross-modal feature distribution differences through group normalization and GELU activation function, a top-down information fusion path is adopted, starting from the highest-level semantic features, upsampling is achieved through parameterized transposed convolution, and the output features are channel-wise concatenated with the adjacent low-level features, and cross-level feature fusion is completed through the convolution kernel; A cascaded structure is used to process the output of each layer, unifying all layer features to the original input resolution through bilinear interpolation, and densely splicing along the channel dimension to form aggregated features; A Softmax function is set before the final output layer to normalize the channel dimension and generate a pixel-level probability distribution map; The fully connected information of each layer feature is retained according to the pixel-level probability distribution map, so as to realize the extraction of the spatial distribution information of the lodging winter wheat area.

Images