Crop remote sensing identification method based on multimodal deep learning
Through the multimodal deep learning method, the low-resolution multi-spectrum and high-resolution full-color remote sensing images are combined, which solves the problems of low calculation efficiency and poor timeliness of crop planting area in the prior art, and realizes the accuracy and detailed resolution of crop recognition The rate has increased to provide data support for smart agriculture.
Patent Information
- Application Number
- CN202411116688.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-15
AI Technical Summary
The calculation of existing crop planting area depends on ground surveys, with high time and labor intensity, low efficiency, and poor information timeliness, making it difficult to accurately reflect the current crop conditions.
Using a crop remote sensing recognition method based on multimodal deep learning, a deep network model is constructed through the LKA mechanism, and using a multi-scale attention fusion strategy to fuse information of low-resolution multispectral remote sensing images and high-resolution full-color band remote sensing images to improve the Identify accuracy and detail resolution.
It is possible to timely and accurately grasp crop categories, area and spatial distribution information in actual production, especially when "mutual species" and "mutual species" types of crops, it can accurately identify and calculate planting areas and provide data for agricultural production. Support and promote smart agriculture and refined management.
Smart Images

Figure CN118644783B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of smart agricultural technology, and in particular to a crop remote sensing identification method based on multimodal deep learning. Background Art
[0002] Agriculture is an important part of the global economy. It not only provides the main food source for humans, but also affects the global distribution of resources, environmental protection and economic development. With the continuous growth of population, it is particularly important to improve the yield and management efficiency of crops. The effective management of crops requires accurate calculation of planting area. At present, the existing calculation of crop planting area is mainly obtained through ground surveys and manual statistics. On the one hand, ground surveys require a lot of manpower and time, especially in vast farmland areas. The time and labor intensity are high, the efficiency is low, and it is difficult to deal with large-scale or inaccessible farmland; on the other hand, manual surveys and data processing take a long time, which easily leads to the collected information not reflecting the current crop status and poor timeliness. Based on this, in response to the above problems, we designed a crop remote sensing recognition method based on multimodal deep learning. Summary of the invention
[0003] The purpose of the present invention is to provide a crop remote sensing identification method based on multimodal deep learning, which adopts a deep network model based on the LKA mechanism, uses a multi-scale attention fusion strategy, and integrates the information of low-resolution multispectral remote sensing images and high-resolution panchromatic band remote sensing images, while reducing the number of parameters and calculations of the model in the identification process. At the same time, it improves the accuracy and detail resolution of crop identification, has high flexibility and scope of application. It can meet the needs of timely and accurate grasp of the category, area and spatial distribution information of crops in actual production, especially in the face of "same object different spectrum" and "same spectrum different object" type crops, it can more accurately identify and calculate the planting area of this type of crops, provide data support for agricultural production, and promote the development of smart agriculture and the refinement of crop management.
[0004] The embodiments of the present invention are implemented by the following technical solutions:
[0005] A crop remote sensing identification method based on multimodal deep learning, the method comprising the following steps:
[0006] Acquiring first remote sensing image data and second remote sensing image data;
[0007] Build a crop remote sensing image recognition model based on the LKA deep neural network, and build a fusion module based on the multi-scale attention strategy;
[0008] The crop remote sensing image recognition model is trained by using the first remote sensing image data and the second remote sensing image data, the features of the first remote sensing image data and the second remote sensing image data are extracted, and the features of the first remote sensing image data and the features of the second remote sensing image data are fused by a fusion module to obtain third remote sensing image data;
[0009] The third remote sensing image data is recognized based on the trained crop remote sensing image recognition model, and segmented based on the crop type recognition result to complete the accurate recognition of crop type and planting area.
[0010] Optionally, after acquiring the first remote sensing image data and the second remote sensing image data, the method further includes image enhancement and registration steps, wherein the image registration step is:
[0011] Performing radiometric calibration on the first remote sensing image data and the second remote sensing image data;
[0012] Performing atmospheric correction on the first remote sensing image data and the second remote sensing image data after radiation calibration;
[0013] The first remote sensing image data and the second remote sensing image data that have been subjected to radiation calibration and atmospheric correction are resampled to complete image registration of the first remote sensing image data and the second remote sensing image data.
[0014] Optionally, the image enhancement step is:
[0015] Extracting a near infrared band, a red band, and a green band from the first remote sensing image data for which image registration has been completed;
[0016] Performing enhancement processing on the near infrared band, the red band and the green band in the first remote sensing image data;
[0017] The various bands in the first remote sensing image data are fused, and the second remote sensing image data is enhanced to complete the image enhancement of the first remote sensing image data and the second remote sensing image data.
[0018] Optionally, the crop remote sensing image recognition model is constructed based on the LKA deep neural network, which is specifically: an encoder model is constructed based on the visual attention block in the LKA deep neural network, and a decoder model is constructed based on the depth separable convolution block in the LKA deep neural network, wherein the calculation formula of the encoder model is:
[0019]
[0020]
[0021]
[0022] in, Represents the convolution operation; , represents the point-to-point convolution operation, represents the depth-expanded convolution operation, represents the depthwise convolution operation, represents the input feature map, represents the obtained large core attention, represents batch normalization, represents a feed-forward network, represents the intermediate feature map, The feature map representing the output;
[0023] The calculation formula of the decoder model is:
[0024]
[0025] in, Represents a depthwise convolution operation.
[0026] Optionally, the fusion module is constructed based on the multi-scale attention strategy, which is specifically:
[0027] Construct spatial normalization module and channel normalization module;
[0028] A spatial fusion module is constructed based on the spatial normalization module, and a channel fusion module is constructed based on the channel normalization module;
[0029] A multi-scale attention fusion module is constructed based on the spatial fusion module and the channel fusion module.
[0030] Optionally, the calculation formula of the spatial fusion module is:
[0031]
[0032]
[0033]
[0034] in, Represents exponential operation; and Respectively represent the corresponding pixel positions of the first remote sensing image data and the second remote sensing image data in the channel direction The initial spatial attention map of and Respectively represent the spatially normalized features of the first remote sensing image data and the second remote sensing image data, and They respectively represent the features of the first remote sensing image data and the second remote sensing image data extracted by the decoder model in the spatial direction, Represents the spatial feature map after weighted fusion.
[0035] Optionally, the calculation formula of the channel fusion module is:
[0036]
[0037]
[0038]
[0039] in, and Represents the initial channel attention vector of the first remote sensing image data and the second remote sensing image data, and Respectively represent the normalized features of the first remote sensing image data and the second remote sensing image data channels, and They respectively represent the characteristics of the first remote sensing image data and the second remote sensing image data extracted by the decoder in the channel direction, Represents the channel feature map after weighted fusion.
[0040] Optionally, the calculation formula of the multi-scale attention fusion module is:
[0041]
[0042] in, Represents the spatial feature map after weighted fusion of the spatial fusion module, Represents the channel feature map after weighted fusion of the channel fusion module, Represents the feature map after weighted fusion.
[0043] Optionally, the specific process of training the crop remote sensing image recognition model by using the first remote sensing image data and the second remote sensing image data is as follows:
[0044] The first remote sensing image data and the second remote sensing image data after image enhancement are input into a crop remote sensing image recognition model;
[0045] Determining the comprehensive loss function of crop remote sensing image recognition model ;
[0046] Obtain the third remote sensing image data output by the crop remote sensing image recognition model in the comprehensive loss function The overall loss value under ;
[0047] Update the parameters of the crop remote sensing image recognition model based on the overall loss value;
[0048] Complete the training of crop remote sensing image recognition model.
[0049] Optionally, the comprehensive loss function The calculation formula is:
[0050]
[0051] in, and represents the average value of the input image, and represents the variance of the input image, represents the covariance of the input image, and is a constant, represents the fused image; represents first remote sensing image data; represents second remote sensing image data; Represents the height and width of the image, Represents the results of the model output; Indicates the corresponding label, , and Represent weights respectively.
[0052] The technical solution of the embodiment of the present invention has at least the following advantages and beneficial effects:
[0053] The embodiment of the present invention adopts a deep network model based on the LKA mechanism, and utilizes a multi-scale attention fusion strategy to fuse the information of low-resolution multispectral remote sensing images and high-resolution panchromatic band remote sensing images, thereby reducing the number of parameters and calculations of the model in the recognition process, while improving the accuracy and detail resolution of crop recognition, and having high flexibility and scope of application. It can meet the needs of timely and accurate grasp of the category, area and spatial distribution information of crops in actual production, especially when facing crops of the "same object, different spectrum" and "same spectrum, different object" types, it can more accurately identify and calculate the planting area of this type of crops, provide data support for agricultural production, and promote the development of smart agriculture and the refinement of crop management. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 A schematic diagram of a process flow of a crop remote sensing identification method based on multimodal deep learning provided by an embodiment of the present invention;
[0055] Figure 2 A schematic diagram of the construction principle of the fusion part provided in an embodiment of the present invention;
[0056] Figure 3 A schematic diagram of the principle of the fusion module provided in an embodiment of the present invention;
[0057] Figure 4A schematic diagram of the principle of a crop remote sensing identification system based on multimodal deep learning provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0059] like Figure 1 As shown, Figure 1 A schematic flow chart of a method for remote sensing crop identification based on multimodal deep learning provided in an embodiment of the present invention.
[0060] In one embodiment, a crop remote sensing identification method based on multimodal deep learning comprises the following steps:
[0061] Acquiring first remote sensing image data and second remote sensing image data;
[0062] Build a crop remote sensing image recognition model based on the LKA deep neural network, and build a fusion module based on the multi-scale attention strategy;
[0063] The crop remote sensing image recognition model is trained by using the first remote sensing image data and the second remote sensing image data, the features of the first remote sensing image data and the second remote sensing image data are extracted, and the features of the first remote sensing image data and the features of the second remote sensing image data are fused by a fusion module to obtain third remote sensing image data;
[0064] The third remote sensing image data is recognized based on the trained crop remote sensing image recognition model, and segmented based on the crop type recognition result to complete the accurate recognition of crop type and planting area.
[0065] Specifically, the first remote sensing image data described in this embodiment specifically represents a low-resolution multispectral remote sensing image; the second remote sensing image data specifically represents a high-resolution panchromatic band remote sensing image; and the third remote sensing image data specifically represents a high-resolution multispectral remote sensing image. Low-resolution multispectral remote sensing images refer to image data that can capture multiple single bands of radiation from ground objects. These data contain spectral information of multiple bands. These images can provide color information of ground objects, but the spatial resolution is relatively low. High-resolution panchromatic band remote sensing images capture image data of a single band. These images generally have high spatial resolution, but cannot display the color of ground objects, that is, the image has little spectral information. Therefore, it is possible to consider fusing the two to obtain a high-resolution multispectral remote sensing image.
[0066] Based on the above, the specific application process of this embodiment is: obtain low-resolution multispectral remote sensing images and high-resolution panchromatic band remote sensing images taken by drones and satellites, enhance and align the acquired remote sensing image data, and ensure that multispectral remote sensing images and high-resolution panchromatic band remote sensing images of the same resolution size are obtained; construct a crop remote sensing image recognition model based on the LKA deep neural network, and construct a fusion module based on the multi-scale attention strategy; train the recognition model based on the enhanced and aligned remote sensing image data; extract the features of the remote sensing image based on the trained recognition model, and use the fusion module to perform feature fusion using local and global contextual image features to obtain a high-resolution multispectral remote sensing image; based on the recognition model, identify the crop type in the fused high-resolution multispectral remote sensing image to achieve accurate recognition and calculation of the crop planting area.
[0067] In this embodiment, the enhancement and registration processing of the original low-resolution multispectral remote sensing image and the high-resolution panchromatic band remote sensing image specifically includes:
[0068] Perform registration processing on the captured original multispectral remote sensing image and the high-resolution panchromatic band remote sensing image;
[0069] Image enhancement is performed on the multispectral remote sensing images after registration and the high-resolution panchromatic band remote sensing images respectively.
[0070] Furthermore, the registration processing of the original multispectral remote sensing image and the high-resolution panchromatic band remote sensing image includes:
[0071] Carry out radiometric calibration on the original multispectral remote sensing image and the high-resolution panchromatic band remote sensing image respectively;
[0072] Perform atmospheric correction on the radiometrically calibrated multispectral remote sensing images and high-resolution panchromatic remote sensing images respectively;
[0073] The multispectral remote sensing images and the high-resolution panchromatic band remote sensing images that have been calibrated for radiation and corrected for atmosphere are resampled to obtain the multispectral remote sensing images and the high-resolution panchromatic band remote sensing images with the same resolution size.
[0074] Furthermore, image enhancement is performed on the registered original multispectral remote sensing image and the high-resolution panchromatic band remote sensing image, including:
[0075] Extracting a near infrared band, a red band, and a green band from the first remote sensing image data for which image registration has been completed;
[0076] Performing enhancement processing on the near infrared band, the red band and the green band in the first remote sensing image data;
[0077] The various bands in the first remote sensing image data are fused, and the second remote sensing image data is enhanced to complete the image enhancement of the first remote sensing image data and the second remote sensing image data.
[0078] In implementation, the CIR remote sensing image is formed by the fusion of near-infrared, red, and green bands. However, the panchromatic band remote sensing image only has a grayscale channel and no other channels, so this embodiment performs corresponding image enhancement processing on it respectively. The reflectivity of the near-infrared band is affected by the structure inside the plant leaves. Different types of crops have different reflective characteristics in this band, which can be used to distinguish different types of crops. In addition, healthy vegetation has a higher reflectivity in the near-infrared band, so the health of the vegetation can be evaluated by analyzing the data in the near-infrared band. The vegetation index and leaf area index can be calculated using the data from the infrared band and the near-infrared band. These data are crucial for accurately identifying crop types, monitoring vegetation health, and evaluating vegetation coverage and biomass. Then, the images of the near-infrared, red, and green bands are fused to obtain the CIR remote sensing image. CIR remote sensing images use near-infrared bands to replace the blue bands in RGB remote sensing images, making healthy vegetation appear red in the image. The healthier the vegetation, the brighter the red. Compared with RGB remote sensing images, CIR remote sensing images can better distinguish vegetation types, detect crop diseases, and implement precision agriculture technology.
[0079] In this embodiment, the construction process of the crop remote sensing image recognition model is as follows:
[0080] Select the LKA-based visual attention block to build the encoder model;
[0081] Choose to build the decoder model based on depthwise separable convolutional blocks.
[0082] The crop remote sensing image recognition model consists of a fusion part and a segmentation part. The fusion part consists of an encoder, a decoder and a fusion module, and the segmentation part consists of a pre-trained segmentation model. The fusion part in the recognition model specifically refers to a deep network model that fuses the features of low-resolution multispectral remote sensing images and high-resolution panchromatic band remote sensing images to obtain high-resolution multispectral remote sensing images. The segmentation part in the recognition model specifically refers to a semantic segmentation model that has been pre-trained on a high-resolution multispectral remote sensing image dataset in a specific crop area. In this embodiment, the fusion part in the recognition model is mainly constructed. The fusion part is as follows: Figure 2 shown.
[0083] Specifically, the encoder model constructed based on the visual attention block of LKA includes:
[0084] The encoder part of the optimized UNet++ network is selected as the encoder model. The optimized UNet++ network encoder model includes:
[0085] Delete the convolutional blocks of the UNet++ network encoder;
[0086] An LKA-based visual attention block is introduced to replace the deleted convolutional blocks.
[0087] This embodiment selects the UNet++ model as the crop remote sensing image recognition model, and optimizes the encoder part in the UNet++ model based on the visual attention of LKA. The UNet++ model is an existing convolutional neural network model with nested and dense jump connections, which can better integrate features from different network depths, thereby improving the quality and accuracy of the fused image; adding a fusion module at the bottleneck allows the network to better fuse the high-level features extracted by the encoder; the visual attention based on the LKA mechanism decomposes the large kernel convolution, allowing it to pass through a large receptive field while reducing the amount of computation. By optimizing the encoder part of the UNet++ model based on the visual attention of the LKA mechanism, the encoder model can better capture the contextual information and feature relationships in the image, improve the accuracy and robustness of the model in identifying crops, and reduce computational overhead.
[0088] Among them, the expression of the LKA large core attention mechanism is:
[0089]
[0090] The expression of the visual attention block based on the LKA mechanism is:
[0091]
[0092]
[0093] in, Represents the convolution operation; , represents the point-to-point convolution operation, represents the depth-expanded convolution operation, represents the depthwise convolution operation, represents the input feature map, represents the obtained large core attention, represents batch normalization, represents a feed-forward network, represents the intermediate feature map, Represents the output feature map.
[0094] Specifically, the decoder model constructed based on the depth-separable convolutional block includes:
[0095] The decoder part of the optimized UNet++ network is selected as the decoder model. The optimized UNet++ network decoder model includes:
[0096] Delete the convolutional blocks in the decoder part of the UNet++ network;
[0097] Depthwise separable convolution is introduced to replace the deleted convolution blocks. Depthwise separable convolution is an efficient convolution operation that divides the standard convolution operation into two parts: depthwise convolution and pointwise convolution. This convolution operation can significantly reduce the number of parameters and computational complexity of the model while maintaining the original performance of the model. The decoder part of the UNet++ model is optimized through depthwise separable convolution, thereby further reducing the computational cost.
[0098] Among them, the expression of depth-wise separable convolution is:
[0099]
[0100] in, Represents a depthwise convolution operation.
[0101] In this embodiment, based on the multi-scale attention strategy, a fusion module is constructed, including:
[0102] Construct spatial normalization module and channel normalization module;
[0103] Construct spatial fusion module and channel fusion module;
[0104] Construct a multi-scale attention fusion module based on the spatial fusion module and the channel fusion module;
[0105] Add a multi-scale attention fusion module to the bottleneck of the UNet++ network.
[0106] In the implementation, the fusion module consists of a spatial normalization module, a channel normalization module, a spatial fusion module, and a channel fusion module. The fusion module uses a multi-scale attention strategy to weight the input feature map at different scales to extract multi-level information. By normalizing and weighting the information in the feature map channel and space, it enhances the important information in the feature map space and channel direction and suppresses irrelevant information. Figure 3 As shown in the figure, the fusion module first obtains the feature maps of the two remote sensing images normalized in space and channel through the spatial normalization module and the channel normalization module respectively, and then weightedly fuses the normalized spatial feature maps and channel feature maps of the two remote sensing images through the spatial fusion module and the channel fusion module respectively, and finally weightedly fuses the feature maps passed through the spatial fusion module and the channel fusion module to obtain the fused feature map.
[0107] Specifically, constructing the spatial normalization module and the channel normalization module includes:
[0108] The calculation formula of the spatial normalization module is:
[0109]
[0110]
[0111] in, Represents exponential operation; and Respectively represent the corresponding pixel positions of the first remote sensing image data and the second remote sensing image data in the channel direction The initial spatial attention map of and They respectively represent the spatially normalized features of the first remote sensing image data and the second remote sensing image data.
[0112] The calculation formula of the channel fusion module is:
[0113]
[0114]
[0115] in, and Represents the initial channel attention vector of the first remote sensing image data and the second remote sensing image data, and They respectively represent the normalized features of the first remote sensing image data and the second remote sensing image data channels.
[0116] Furthermore, a spatial fusion module and a channel fusion module are constructed according to the above-mentioned spatial normalization module and channel normalization module, wherein the calculation formula of the spatial fusion module is:
[0117]
[0118] in, and They respectively represent the features of the first remote sensing image data and the second remote sensing image data extracted by the decoder model in the spatial direction, Represents the spatial feature map after weighted fusion.
[0119] The calculation formula of the channel fusion module is:
[0120]
[0121] in, and They respectively represent the characteristics of the first remote sensing image data and the second remote sensing image data extracted by the decoder in the channel direction, Represents the channel feature map after weighted fusion.
[0122] Based on the above, the calculation formula of the multi-scale attention fusion module is:
[0123]
[0124] in, Represents the spatial feature map after weighted fusion of the spatial fusion module, Represents the channel feature map after weighted fusion of the channel fusion module, Represents the feature map after weighted fusion.
[0125] In this embodiment, training the recognition model based on the enhanced and registered remote sensing image data specifically includes:
[0126] Input the enhanced and registered low-resolution multispectral remote sensing image and high-resolution panchromatic band remote sensing image into the recognition model;
[0127] Determine the comprehensive loss function for image fusion ;
[0128] Obtain high-resolution multispectral remote sensing images output by the model in the loss function The overall loss value under ;
[0129] The recognition model parameters are updated based on the overall loss value.
[0130] In the reference of this embodiment, the training of the recognition model should be trained using pre-processed low-resolution multispectral remote sensing images and high-resolution panchromatic band remote sensing images to ensure that the two images have the same resolution, the same geometric position, and the same shooting time; for example: if it is necessary to identify the wheat crop area, it is necessary to use a drone or satellite to shoot a low-resolution multispectral remote sensing image and a high-resolution panchromatic band remote sensing image above this area at the same time and place, and pre-process these two images before inputting the model for training to ensure the accuracy of the model output results. It should be noted that the setting of the weight of the loss function during training should be set according to the characteristics of the crops to be identified; for example: if it is necessary to identify the cotton crop area, it is necessary to adjust the structural similarity loss weight, texture loss weight and L1 loss weight in the comprehensive loss function according to the characteristics of the cotton crop, so that the model can better capture the geometric features, texture features and sparse features of the cotton crop area, so as to achieve the purpose of more accurate identification of specific crop areas.
[0131] Specifically, the comprehensive loss function It is obtained by determining the texture loss, the structural similarity loss, and the L1 loss, and its calculation formula is:
[0132]
[0133] in, and represents the average value of the input image, and represents the variance of the input image, represents the covariance of the input image, and is a constant, represents the fused image; represents first remote sensing image data; represents second remote sensing image data; Represents the height and width of the image, Represents the results of the model output; Indicates the corresponding label, , and Represent weights respectively.
[0134] The features of the remote sensing image are extracted based on the trained recognition model, and the local and global context image features are used to perform feature fusion through the fusion module to obtain a high-resolution multispectral remote sensing image; wherein the trained recognition model is a fusion model in the recognition model that is fully trained on the registered low-resolution multispectral remote sensing image and high-resolution panchromatic band remote sensing image data set. The enhanced and registered low-resolution multispectral remote sensing image and the high-resolution panchromatic band remote sensing image are input into the fusion model in the recognition model to obtain a fused high-resolution multispectral remote sensing image.
[0135] In this embodiment, based on the recognition model, the crop types in the fused high-resolution multispectral remote sensing image are identified to achieve accurate recognition and calculation of the crop planting area, including:
[0136] The fused high-resolution multispectral remote sensing image is input into the pre-trained segmentation model in the recognition model.
[0137] Based on the recognition model, the crop types in the fused high-resolution multispectral remote sensing images are identified to achieve accurate recognition and calculation of the crop planting area. Among them, the high-resolution multispectral remote sensing image output by the fusion part of the recognition model is input into the pre-trained segmentation part to obtain the recognition result of the specific crop planting area.
[0138] In summary, the recognition model constructed based on the LKA mechanism learns the registered and enhanced low-resolution multispectral remote sensing images and high-resolution panchromatic band remote sensing images, and uses the multi-scale attention strategy to fuse the features of the two modal remote sensing images to obtain high-resolution multispectral remote sensing images. Then, the crop area is identified through the pre-trained recognition model to achieve accurate identification and calculation of the crop planting area with high accuracy. The LKA mechanism and deep separable convolution are used to further reduce the network parameters, and the parameters and resource consumption are reduced on the basis of improving the recognition accuracy, which is convenient for agricultural producers and technicians to grasp the category, area and spatial distribution information of crops on site conveniently, quickly and accurately, and monitor the growth of crops and predict the yield.
[0139] like Figure 4 As shown, Figure 4 A schematic diagram of the principle of a crop remote sensing identification system based on multimodal deep learning provided in an embodiment of the present invention.
[0140] In another embodiment, a crop remote sensing identification system based on multimodal deep learning includes:
[0141] A data acquisition module configured to acquire low-resolution multispectral remote sensing images and high-resolution panchromatic band remote sensing images;
[0142] An image processing module is configured to enhance and register the acquired low-resolution multispectral remote sensing image and the high-resolution panchromatic band remote sensing image to obtain the enhanced and registered low-resolution multispectral remote sensing image and the high-resolution panchromatic band remote sensing image;
[0143] A network training module is configured to learn the enhanced low-resolution multispectral remote sensing image and the high-resolution panchromatic band remote sensing image, calculate the overall loss value through the loss function, and update the network parameters through the loss value;
[0144] An image fusion module is configured to fuse the registered low-resolution multispectral remote sensing image and the high-resolution panchromatic band remote sensing image to obtain a fused high-resolution multispectral remote sensing image;
[0145] The region recognition module is configured to recognize the crop part of the high-resolution multispectral remote sensing image and obtain the recognition result of the crop planting area.
[0146] It can be understood that the crop remote sensing identification system based on multimodal deep learning provided in this embodiment and the crop remote sensing identification method based on multimodal deep learning provided in the above embodiment are based on the same inventive concept. For more specific working principles of each module in the embodiment of the present invention, please refer to the above embodiment and will not be repeated in the embodiment of the present invention.
[0147] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A crop remote sensing identification method based on multimodal deep learning, characterized in that: The steps of the method include: Acquiring first remote sensing image data and second remote sensing image data; Build a crop remote sensing image recognition model based on the LKA deep neural network, and build a fusion module based on the multi-scale attention strategy; The crop remote sensing image recognition model is trained by using the first remote sensing image data and the second remote sensing image data, the features of the first remote sensing image data and the second remote sensing image data are extracted, and the features of the first remote sensing image data and the features of the second remote sensing image data are fused by a fusion module to obtain third remote sensing image data; The third remote sensing image data is identified based on the trained crop remote sensing image recognition model, and segmented based on the crop type recognition result to complete the accurate recognition of crop type and planting area; The crop remote sensing image recognition model based on the LKA deep neural network is specifically constructed by: constructing an encoder model based on the visual attention block in the LKA deep neural network, and constructing a decoder model based on the deep separable convolution block in the LKA deep neural network, wherein the calculation formula of the encoder model is: in, Represents the convolution operation; , represents the point-to-point convolution operation, represents the depth-expanded convolution operation, represents the depthwise convolution operation, represents the input feature map, represents the obtained large core attention, represents batch normalization, represents a feed-forward network, represents the intermediate feature map, The feature map representing the output; The calculation formula of the decoder model is: in, Represents a depthwise convolution operation; The fusion module is constructed based on the multi-scale attention strategy, which is specifically: Construct spatial normalization module and channel normalization module; A spatial fusion module is constructed based on the spatial normalization module, and a channel fusion module is constructed based on the channel normalization module; Construct a multi-scale attention fusion module based on the spatial fusion module and the channel fusion module; The calculation formula of the spatial fusion module is: in, Represents exponential operation; and Respectively represent the corresponding pixel positions of the first remote sensing image data and the second remote sensing image data in the channel direction The initial spatial attention map of and Respectively represent the spatially normalized features of the first remote sensing image data and the second remote sensing image data, and They respectively represent the features of the first remote sensing image data and the second remote sensing image data extracted by the decoder model in the spatial direction, Represents the spatial feature map after weighted fusion; The calculation formula of the channel fusion module is: in, and Represents the initial channel attention vector of the first remote sensing image data and the second remote sensing image data, and Respectively represent the normalized features of the first remote sensing image data and the second remote sensing image data channels, and They respectively represent the characteristics of the first remote sensing image data and the second remote sensing image data extracted by the decoder in the channel direction, Represents the channel feature map after weighted fusion; The calculation formula of the multi-scale attention fusion module is: in, Represents the spatial feature map after weighted fusion of the spatial fusion module, Represents the channel feature map after weighted fusion of the channel fusion module, Represents the feature map after weighted fusion.
2. The crop remote sensing identification method based on multimodal deep learning according to claim 1, characterized in that: After the first remote sensing image data and the second remote sensing image data are obtained, the method further includes image enhancement and registration steps, wherein the image registration step is: Performing radiometric calibration on the first remote sensing image data and the second remote sensing image data; Performing atmospheric correction on the first remote sensing image data and the second remote sensing image data after radiation calibration; The first remote sensing image data and the second remote sensing image data that have been subjected to radiation calibration and atmospheric correction are resampled to complete image registration of the first remote sensing image data and the second remote sensing image data.
3. The crop remote sensing identification method based on multimodal deep learning according to claim 2 is characterized in that: The image enhancement steps are: Extracting a near infrared band, a red band, and a green band from the first remote sensing image data for which image registration has been completed; Performing enhancement processing on the near infrared band, the red band and the green band in the first remote sensing image data; The various bands in the first remote sensing image data are fused, and the second remote sensing image data is enhanced to complete the image enhancement of the first remote sensing image data and the second remote sensing image data.
4. The crop remote sensing identification method based on multimodal deep learning according to claim 3 is characterized in that: The specific process of training the crop remote sensing image recognition model by using the first remote sensing image data and the second remote sensing image data is as follows: The first remote sensing image data and the second remote sensing image data after image enhancement are input into a crop remote sensing image recognition model; Determining the comprehensive loss function of crop remote sensing image recognition model ; Obtain the third remote sensing image data output by the crop remote sensing image recognition model in the comprehensive loss function The overall loss value under ; Update the parameters of the crop remote sensing image recognition model based on the overall loss value; Complete the training of crop remote sensing image recognition model.
5. The crop remote sensing identification method based on multimodal deep learning according to claim 4 is characterized in that: The comprehensive loss function The calculation formula is: in, and represents the average value of the input image, and represents the variance of the input image, represents the covariance of the input image, and is a constant, represents the fused image; represents first remote sensing image data; represents second remote sensing image data; Represents the height and width of the image, Represents the results of the model output; Indicates the corresponding label, , and Represent weights respectively.
