A lodging area identification method and system based on crop remote sensing images

Abstract

The application discloses a crop lodging area recognition method and system based on remote sensing images, and belongs to the technical field of image processing, comprising the following steps: collecting remote sensing images of crop fields by using a UAV; performing image splicing, radiation calibration and image cropping preprocessing on the collected remote sensing images; extracting features from the processed remote sensing images to obtain color features, texture features and vegetation index features; and identifying the crop lodging area in the images based on a convolutional neural network semantic segmentation algorithm. In view of the low recognition accuracy of the crop lodging area in the prior art, the application collects high-definition remote sensing images of the target area by using a UAV, and realizes image semantic segmentation by constructing a convolutional neural network model, so that the crop lodging area can be automatically identified. The use of the attention mechanism and multi-scale feature fusion improves the learning ability of the model to subtle features and improves the recognition accuracy of the crop lodging area.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a method and system for identifying lodged areas based on remote sensing images of crops. Background Technology

[0002] Agriculture is an indispensable part of human life, and crops, as one of the main food crops, are crucial for food security through improved yield and quality. However, crop lodging is a common problem, especially when subjected to extreme weather events or abnormal conditions during the growing season. Lodging prevents crops from growing normally, affecting yield and quality. Therefore, timely and accurate identification of lodged areas is essential for taking effective countermeasures. Traditional methods for identifying crop lodging are often limited to manual visual inspection or simple image processing techniques, resulting in low accuracy that is difficult to meet the needs of agricultural production. With the rapid development of remote sensing technology and deep learning, applying these advanced technologies to the agricultural field has become an effective way to improve the accuracy of crop lodging identification.

[0003] Current technologies face significant challenges in accurately identifying lodged crop areas. Traditional image processing methods often fail to adequately account for the impact of complex terrain, vegetation, and lighting conditions on images, resulting in inaccurate identification results. Therefore, a more advanced and precise method is urgently needed to address this issue.

[0004] Chinese patent application CN202210658126.2, published on September 13, 2022, discloses a method and apparatus for extracting lodged area of ​​crops based on UAV images. The method includes: 1) acquiring RGB image data of crops in a field, and performing stitching, cropping, and labeling to obtain a study area map and label data; 2) using the RGB images of the crop study area to obtain a digital surface model image of the crop, and fusing the crop RGB images and the digital surface model image to obtain a crop RGB-digital surface model image; 3) constructing a crop lodged segmentation model based on deep learning, including a spatial path, a contextual path, and a feature fusion module, wherein the contextual path includes a ResNet18 network and a pyramid segmentation attention; 4) training the crop lodged segmentation model using the crop RGB-digital surface model image data, and segmenting the lodged area of ​​the crop image, thereby estimating the lodged area area using the segmentation results. However, in this scheme, the ResNet18 network and pyramid segmentation attention are used as the contextual path. The choice of network structure affects the extraction of complex features in lodged areas. If the chosen network structure is not suitable for the scenario of crop lodging, it may lead to low recognition accuracy. Summary of the Invention

[0005] 1. Technical problems to be solved

[0006] To address the problem of low accuracy in identifying lodged areas of crops in existing technologies, this application provides a method and system for identifying lodged areas based on remote sensing images of crops. By using remote sensing images acquired by UAVs and semantic segmentation performed by convolutional neural networks, the accuracy of identifying lodged areas of crops is improved.

[0007] 2. Technical Solution

[0008] The purpose of this application is achieved through the following technical solution.

[0009] One aspect of the embodiments of this specification provides a method for identifying lodged areas of crops based on remote sensing images, comprising: acquiring remote sensing images of crops; performing preprocessing on the acquired remote sensing images, including image stitching, radiometric calibration, and image cropping; extracting features from the processed remote sensing images to obtain color features, texture features, and vegetation index features; and identifying lodged areas of crops in the images based on a semantic segmentation algorithm using a convolutional neural network.

[0010] Furthermore, the convolutional neural network adopts the PSPNet network; the encoder uses a pre-trained ResNet as the encoder to extract features from the input image; the fusion module contains convolutional layers with pooling kernels of different sizes to perform multi-scale pooling on the features extracted by the encoder to obtain feature maps of different scales; the decoder contains n-level upsampling layers and n-level convolutional layers, where the i-th level upsampling layer enlarges the i-th level feature map to the size of the (i+1)-th level feature map, and the i-th level convolutional layer fuses the feature map output by the i-th level upsampling layer with the feature map output by the i-th level upsampling layer.

[0011] Further steps include: extracting features from the input image through convolutional layers as the first feature map; performing multi-scale pooling on the first feature map using pooling kernels of different sizes to obtain pooled feature maps of different scales as the second feature map; fusing features from the second feature maps of different scales, wherein the smallest scale feature map is convolved, and other feature maps besides the smallest scale feature map are upsampled or downsampled and concatenated to obtain a concatenated feature map; fusing the concatenated feature maps using an attention mechanism to obtain a fused feature map; upsampling the fused feature map through deconvolution to obtain an upsampled feature map; and performing dot product, concatenation, and convolution operations on the upsampled feature map and the smallest scale feature map to output an image containing the predicted results of the fallen area.

[0012] Furthermore, feature fusion of second feature maps at different scales includes the following steps: inputting second feature maps at different scales into the fusion module; performing first feature extraction on the second feature maps through a first convolutional layer to extract feature maps at different scales as the first feature maps; performing bidirectional spatial separation convolution on the first feature maps using a second convolutional layer to extract feature maps at different scales as the second feature maps, wherein the bidirectional spatial separation convolution includes horizontal one-dimensional convolution and vertical one-dimensional convolution; concatenating the second feature maps extracted by the bidirectional spatial separation convolution to obtain a concatenated feature map; and performing a convolution operation on the concatenated feature map using a third convolutional layer to output the fused feature map.

[0013] Furthermore, when performing horizontal one-dimensional convolution, the length of the convolution kernel is set to be greater than its width to extract the horizontal features of the image; when performing vertical one-dimensional convolution, the width of the convolution kernel is set to be greater than its length to extract the vertical features of the image.

[0014] Furthermore, remote sensing image acquisition of crop fields using drones includes the following steps: the drone acquires remote sensing images of the crop fields using a multispectral camera; the multispectral camera splits the acquired remote sensing images into RGB three channels to obtain three single-channel images of red, green, and blue; the acquired three single-channel images of red, green, and blue are stored separately to reduce storage requirements.

[0015] Furthermore, the acquired remote sensing images are stitched together, including the following steps: arranging and combining the acquired single-channel images in chronological order; performing image fusion processing on the overlapping areas of adjacent images, where the overlapping areas represent the common surface areas covered by images taken at the same geographical location; and stitching together to generate a complete remote sensing image covering the target crop field.

[0016] Furthermore, radiometric calibration of the acquired remote sensing images is performed, including the following steps: acquiring illumination parameters when the UAV acquires the remote sensing images; constructing an image radiometric calibration model using the acquired illumination parameters; and correcting the illumination information of each pixel in the stitched overall remote sensing image using the constructed image radiometric calibration model to eliminate the influence of different illumination conditions.

[0017] Furthermore, the acquired remote sensing images are cropped, including the following steps: using the radiometrically calibrated overall remote sensing image as the input image; obtaining the target crop field area and background area in the input image through an image segmentation algorithm; calculating the boundary of the target crop field area through a boundary detection algorithm to obtain the polygonal region coordinates of the target crop field area; and cropping and saving the input image based on the obtained polygonal region coordinates as the target area.

[0018] Another aspect of the embodiments of this specification provides a lodging area identification system based on remote sensing images of crops, used to execute the lodging area identification method based on remote sensing images of crops of this application, including: an image acquisition module for acquiring remote sensing images of crop fields; a preprocessing module for stitching, radiometric calibration, and image cropping of the acquired remote sensing images; a feature extraction module for extracting features from the preprocessed remote sensing images, obtaining color features, texture features, and vegetation index features respectively; and a recognition module for semantic segmentation of the feature-extracted remote sensing images based on a pre-trained convolutional neural network model to identify lodging areas of crops in the remote sensing images, wherein the convolutional neural network model adopts a PSPNet structure; the PSPNet structure includes: an encoder for feature extraction; a fusion module for multi-scale feature fusion; and a decoder for pixel prediction and upsampling to the original resolution.

[0019] 3. Beneficial effects

[0020] Compared to existing technologies, the advantages of this application are:

[0021] By using UAVs to acquire multispectral remote sensing images of crops and combining them with semantic segmentation algorithms from convolutional neural networks, high-precision identification of lodged areas of crops was achieved. The use of multi-scale pooling and feature fusion modules helps to capture feature information at different scales in crop images, improving the sensitivity to lodged areas. The use of an attention mechanism to fuse feature maps helps the model focus more on the key parts of the lodged areas, improving the model's accuracy and robustness.

[0022] Arranging and fusing multiple acquired images chronologically helps eliminate stitching gaps caused by drone flight path changes or other factors, reducing inconsistencies and thus minimizing potential noise. For overlapping areas in adjacent images, image fusion processing ensures a smoother transition of information in these areas, reducing potential noise and discontinuities. Identifying the target crop field area using image segmentation algorithms, followed by cropping using boundary detection algorithms, yields a subset of images containing the target region. This cropping operation helps remove irrelevant areas, focusing analysis on the target region and improving the efficiency and stability of subsequent algorithms.

[0023] By introducing illumination parameters and a radiometric calibration model, the overall remote sensing image is corrected to eliminate the influence of different illumination conditions on the recognition results, thus improving the robustness of the system. Collecting illumination parameters and constructing an image radiometric calibration model can correct the illumination information of individual pixels in the overall remote sensing image. This helps to obtain more consistent images under different illumination conditions, improving the stability and consistency of subsequent algorithms for image features.

[0024] By introducing bidirectional spatially separable convolution, feature extraction can be performed simultaneously in both the horizontal and vertical directions. This parallel feature extraction mechanism helps the model to more comprehensively understand the structure and details in the image, enhancing its ability to identify fine-grained lodged areas of crops. The combination of horizontal and vertical convolutions enables the model to more sensitively capture subtle horizontal and vertical features in the image, including the texture, shape, and orientation information of lodged areas of crops. The use of bidirectional spatially separable convolution enhances the model's perception of the spatial structure of the image. By performing convolutions in the horizontal and vertical directions respectively, the model can better maintain the spatial resolution of the image, improve its sensitivity to fine structures, and help to more accurately segment lodged areas of crops.

[0025] Splitting an image into its RGB channels and storing each channel separately means that information from each channel can be stored independently. This helps reduce storage requirements while preserving important color information. Storing each channel of an image, rather than the entire RGB image, significantly reduces storage needs. This is particularly important for remote sensing image data of large-scale farmland, alleviating the burden of data management and transmission; separate channel storage allows for independent analysis of each color channel, making subsequent image processing and analysis more flexible. This has a positive impact on tasks such as vegetation index calculation and vegetation health assessment.

[0026] In summary, using UAVs to acquire high-resolution remote sensing images of target areas and constructing a convolutional neural network model for semantic image segmentation can automatically identify areas of crop lodging. Employing attention mechanisms and multi-scale feature fusion enhances the model's ability to learn subtle features, thereby improving the accuracy of crop lodging area identification. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of a method for identifying lodged areas based on remote sensing images of crops, according to an embodiment of this application.

[0028] Figure 2 This is a diagram illustrating the architecture of an unmanned aerial vehicle (UAV) system for automatic acquisition of remote sensing images of wheat, according to an embodiment of this application.

[0029] Figure 3 This is a diagram of a PSPNet model according to an embodiment of this application;

[0030] Figure 4 This is a diagram showing the wheat lodging area identification result according to an embodiment of this application. Detailed Implementation

[0031] The present application will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0032] Figure 1This diagram illustrates a method for identifying lodging areas based on remote sensing images of crops, according to an embodiment of this application. The method includes the following steps: data acquisition; data preprocessing; dataset construction; and feature extraction. Specifically, a multispectral camera from a drone is used to conduct low-altitude flight photography of the crop field, acquiring remote sensing image data at different time points. The acquired image data includes visible light images and near-infrared images. The acquired raw remote sensing image data undergoes preprocessing such as image stitching, image cropping, and image registration to remove invalid areas and construct an orthophoto that accurately covers the target field. Atmospheric correction and noise reduction are then performed to obtain high-quality preprocessed images. Representative samples are selected from the preprocessed images, and image patches are extracted to construct a dataset. The constructed dataset includes normally growing samples and lodged samples, with the lodged samples covering different degrees of mild and severe lodging. Based on the constructed sample dataset, feature information expressing the crop growth state is extracted, including features such as color, texture, and morphology. The extracted features become the key criteria for identifying crop lodging.

[0033] Figure 2 This is a schematic diagram of an automatic wheat remote sensing image acquisition UAV system according to an embodiment of this application, including a quadcopter UAV, its onboard multispectral camera, a ground base station, a server, and a client. Specifically: the quadcopter UAV is used to automatically cruise and perform field patrol tasks according to a set route and parameters; the multispectral camera onboard the quadcopter UAV is used to capture remote sensing images of the field; the ground base station is used to park the quadcopter UAV, provide a charging interface for the UAV, and receive remote sensing data acquired by the multispectral camera through an internal wireless transceiver, then transmits the remote sensing data to the server via a wireless network; the server is used to collect, store, and process the remote sensing images acquired by the multispectral camera; the client is used to send control commands to the UAV and the ground base station, and display the server's processing results of the remote sensing images to the user.

[0034] Specifically, the crop lodging monitoring system of this invention includes a quadcopter drone, an onboard multispectral camera, a ground base station, a server, and a client. Specifically: The quadcopter drone automatically cruises over the target crop field according to a preset flight path and parameters to collect remote sensing images. The drone automatically tracks its cruise path via an autonomous flight control system. The multispectral camera, mounted on the quadcopter drone's onboard gimbal, includes both visible light and near-infrared imaging spectral sensors. During its cruise, the drone captures images of the crop field from multiple angles, acquiring raw image data of different spectra. The ground base station, located on open ground next to the crop field, sends control commands and positioning signals to remotely and intelligently control the quadcopter drone. It also receives image data transmitted from the multispectral camera. The server, with high-speed processing capabilities, stores and preprocesses the received raw images, and constructs a deep learning model for crop lodging monitoring, enabling intelligent identification and location of lodged areas in the images. Client-side: The operation and management software installed with the monitoring system can be used to plan flight routes, set parameters, and monitor the drone's working status and image recognition results in real time.

[0035] Specifically, the ground base station is used to park quadcopter drones and provide them with charging interfaces to meet their charging requirements during mission rotation. The ground base station is also equipped with a highly sensitive wireless transceiver to receive raw remote sensing image data transmitted in real time by multispectral cameras during flight. The ground base station transmits the received image data to the monitoring system's server via a wireless network transmission system. The server has high-speed storage and data processing capabilities. It can receive and store large amounts of raw remote sensing image data transmitted from the ground base station and perform a series of preprocessing operations such as image format conversion, image cropping, and image stitching. The server will also perform feature extraction and training sample selection on the preprocessed image data to build a deep learning model for monitoring and identifying crop lodging. Finally, using the trained model, intelligent identification and location of lodged areas in crop field images are achieved. The server's processing results are fed back to the monitoring system's client, providing users with intuitive monitoring results such as the location of crop lodging and the degree of damage.

[0036] Specifically, the client device is equipped with human-machine interface software for the crop lodging monitoring system, providing a user-friendly interface. The client can send remote control commands to the drone for takeoff, landing, and flight path modifications, enabling precise control of the drone mission. The client can also send control commands related to image data transmission to the ground base station. Additionally, the client can receive processed remote sensing images of crops from the server, along with lodging identification results derived from these images. The client software visualizes the processing results, clearly displaying them to the user through images, curves, and reports, such as using different colors to mark the distribution of lodging areas in the image. Through the client, users can intuitively understand the growth status of crops in the field and the occurrence of lodging, assess the severity of losses, and take timely preventative measures.

[0037] The unmanned aerial vehicle (UAV) system for automatically acquiring remote sensing images of crops includes the following steps: The user sets the cruise route, flight altitude, flight time, and data acquisition area on the client device; the ground base station receives the command and automatically opens the cabin door via an internal transmission mechanism. The quadcopter UAV then takes off from the cabin to perform its flight mission; during flight, the ground base station interacts with the quadcopter UAV in real time. The onboard infrared spectral camera adjusts its servo angle at predetermined coordinates according to preset commands to capture remote sensing images of the wheat field; after completing the cruise mission, the quadcopter UAV automatically returns to the cabin for automatic charging, preparing for the next cruise mission. Simultaneously, the quadcopter UAV transmits the acquired remote sensing data to the ground base station; the base station transmits the data to a server, and the server's data processing results are fed back to the client platform, providing data support for agricultural production.

[0038] Specifically, the workflow of an unmanned aerial vehicle (UAV) system for automatic remote sensing image acquisition of crops includes the following steps: The user plans and sets key mission parameters such as the UAV's cruise route, flight altitude, flight time, and the area for remote sensing data acquisition through client software. After receiving the flight mission command from the client, the ground base station automatically opens the maintenance hatch of its internal UAV launcher to prepare for launch. Once the user confirms the parameters are correct, they send a flight mission start command to the ground base station and the UAV on the payload via the client. The transmission device inside the ground base station pushes the UAV, fixed to the launcher, out of the maintenance hatch. After the UAV's self-check program is initiated, its flight control system automatically unlocks the propellers, and the UAV takes off according to the preset parameters. After reaching the target area, the UAV automatically cruises according to the set route and altitude, and its onboard multispectral imaging system initiates the remote sensing image acquisition mission. After acquisition is complete, the UAV automatically returns and lands in the maintenance hatch of the ground base station. The ground base station closes the maintenance hatch, completing one flight mission.

[0039] Specifically, after completing system self-checks and unlocking the propellers, the quadcopter drone takes off from the maintenance bay of the ground base station and flies to the preset target area to begin its remote sensing image acquisition mission. Throughout the flight mission, the communication system in the ground base station maintains radio communication with the quadcopter drone in the air. On one hand, the ground base station receives flight telemetry data transmitted back by the drone in real time, including parameters such as the drone's GPS position, altitude, speed, and battery level, monitoring the drone's flight status. On the other hand, if it is necessary to modify the flight trajectory or issue a return command, the ground base station can also transmit commands in real time to control the drone's flight. After the drone completes data acquisition and returns, the ground base station's communication system switches to video link communication with the drone, allowing the ground operator to monitor the drone's landing process in real time and ensure its safe and smooth landing.

[0040] Specifically, during flight, the onboard infrared spectral camera, according to preset shooting instructions, adjusts its angle by controlling the rotation of servos when it reaches the designated coordinates, capturing multi-angle images of the wheat field below to obtain remote sensing image data. Infrared spectral imaging can compensate for the influence of lighting conditions, ensuring clear information about the wheat field features. After successfully completing a full cruise shooting mission, the quadcopter drone will automatically fly back to the ground base station and land accurately in the maintenance cabin. At this time, the charging port of the ground base station will automatically charge the drone's battery. Once the battery level is restored to the set value, the quadcopter drone will be ready for the next cruise shooting mission.

[0041] Specifically, after the quadcopter drone safely lands in the maintenance bay of the ground base station, it transmits the raw remote sensing image data collected during the mission to the ground base station's storage device via a data transmission interface. Upon receiving the image data from the drone, the ground base station immediately transmits this data to the backend server for storage via network equipment. The server, based on the raw data collected by the sensors, uses a preset algorithm model to intelligently analyze or identify crop growth status, pest and disease conditions, and other data, generating the resulting data. Finally, the server feeds back these processed results to the monitoring system's client platform and displays them to the user through the software interface. This provides data support for subsequent agricultural production management, crop condition assessment, and the formulation of prevention and control strategies.

[0042] Data preprocessing includes the following steps: preprocessing the acquired remote sensing images, including image stitching, radiometric calibration, and image cropping; performing feature value analysis and filtering on the preprocessed image data, which involves calculating separability using Mahalanobis distance and obtaining parameters such as color features, texture features, and vegetation index through random forest feature filtering; and finally, extracting the lodged areas using a modified PSPNet.

[0043] Specifically, the data preprocessing module processes the acquired raw remote sensing images, mainly including the following steps: Image stitching, stitching together images captured segment by segment by the UAV to construct a complete orthophoto, ensuring full coverage of the target area. The SIFT feature matching algorithm is used to achieve accurate stitching of overlapping areas; Radiometric calibration, based on the lighting conditions and sensor parameters at the time of capture, constructs a radiometric calibration model to compensate for image pixel values ​​and eliminate the influence of lighting, ensuring the reliability of subsequent processing and analysis; Image cropping, based on latitude and longitude coordinates, automatically detects and crops the entire image, extracting precise sub-images containing the target wheat field, reducing the computational load of subsequent processing and improving efficiency.

[0044] Specifically, the preprocessed remote sensing image data enters the feature extraction module for further processing, mainly obtaining the following features: color features, using pixel statistical features in color spaces such as RGB and HSV to describe the overall color information of the wheat field; texture features, extracting texture features such as variance, entropy, and second moment of angle to characterize the texture information of the wheat field surface; and vegetation indices, calculating indices such as NDVI and RDVI to reflect the growth status of wheat stalks, leaves, etc. During the feature extraction process, separability analysis and feature selection are performed, specifically including: calculating the separability of each feature using Mahalanobis distance and removing redundant and related features; using a random forest algorithm to filter features, selecting those that significantly contribute to target recognition. The feature set obtained after filtering can serve as a key criterion for subsequent modeling training.

[0045] Specifically, based on feature extraction, the system uses an improved PSPNet network model to extract lodging areas of crops. This model extracts multi-scale features through a pyramid pooling module, fully utilizing global information under different receptive fields. An attention mechanism is introduced into the network to focus on key local features, improving the model's ability to identify lodging areas. Based on training sample annotation and network parameter tuning, the PSPNet model can perform semantic segmentation on the input remote sensing image of crops, outputting pixel-level classifications of the original image and clearly identifying the lodging areas. By applying the improved PSPNet model, the system achieves automated and accurate extraction of crop lodging, providing support for subsequent damage assessment and control measures.

[0046] When a specific area is designated as the object of image acquisition, it is difficult to obtain high-resolution images when using a single UAV image to cover all ground features. Furthermore, when crops are affected by lodging, the damage is often widespread; therefore, it is necessary to stitch together multiple UAV images to obtain high-resolution crop image data. In addition, GPS information acquired from UAV images may be inaccurate due to wind and atmospheric conditions, necessitating calibration of the stitched images using ground control points. Radiometric calibration is required to obtain relative spectral reflectance, and considering the distortion that occurs in images acquired by optical sensors, geometric correction is necessary. Images acquired at the edges of UAV flight paths may exhibit discontinuities and misalignments after stitching, requiring cropping to retain the complete study area.

[0047] Specifically, when a specific local area is designated as the target for remote sensing image acquisition, directly using single-view images taken by drones at high altitudes is insufficient due to the distance, making it difficult to obtain high-resolution images and clearly capture the details of the terrain features in that area. Furthermore, crop lodging damage often occurs over large, contiguous areas. Therefore, to obtain high-quality images for crop lodging monitoring, a low-altitude, multi-view flight approach is necessary to acquire images of the target area. Specifically, the drone's low-altitude flight path can be planned in advance, repeatedly capturing images of the target area from different angles and altitudes. This yields multiple partial images from different angles, which are then stitched together using robust image matching algorithms that utilize common feature information to construct a high-resolution overall image. The stitched image contains richer and more complete details of the terrain features in the target area, clearly reflecting different degrees of crop lodging, and is used for subsequent monitoring and identification.

[0048] Specifically, after acquiring multi-view UAV images, the following further preprocessing is required: Image calibration based on ground control points. Because UAV flight is affected by atmospheric conditions such as wind direction and airflow, the GPS information recorded in the images will have certain deviations. To eliminate this deviation, clearly marked ground control points need to be added to the images, and their latitude and longitude coordinates are used to perform geometric correction of the images through two-dimensional transformation, ensuring the spatial accuracy of subsequent analysis. Radiometric calibration of relative spectral reflectance. To eliminate the influence of different lighting conditions and sensor parameters, a radiometric calibration model needs to be established to convert the image's digital values ​​to standard spectral reflectance, serving as the basis for subsequent analysis and interpretation. Correction of geometric distortion. Since optical images can exhibit geometric distortions such as prism distortion and sleeper distortion, distortion coefficient estimation and correction are needed to ensure image authenticity, obtaining orthophotos with accurate scale.

[0049] Specifically, when stitching multiple images, attention must be paid to the issues of images acquired along the edges of the UAV flight path. These edge images often exhibit discontinuities and misalignment with adjacent images during stitching, failing to achieve a tight fit with the overall image. After stitching, necessary image cropping is required, removing edge images captured on both sides and in the forward and backward directions of the flight path, retaining only the complete and continuous image region at the center. After cropping, we can obtain a remote sensing image that accurately contains information about the land cover throughout the entire study area. The cropping range needs to be determined based on the actual flight trajectory and the coverage area, ensuring that the entire study area is included while avoiding the introduction of invalid edge regions. The cropped image provides a reliable data foundation for subsequent accurate identification of crop growth status.

[0050] Image stitching determines transformation parameters between images by calculating similarity metrics, transforming multiple acquired images to the same coordinate system and achieving optimal matching at the pixel level. Gray-level-based matching uses image gray-level information as the processing object, performing matching by calculating and optimizing extreme values. Gray-level-based matching methods do not involve the feature value extraction process of image segmentation, thus possessing characteristics such as high accuracy and strong robustness.

[0051] Specifically, the image stitching module needs to integrate multiple partial images into a single panoramic image. First, feature points between images are detected, and feature point descriptors are established. The similarity metric between feature point descriptors is calculated, and the geometric transformation model parameters between images, including rotation, translation, and scaling, are determined. All images are then transformed to a unified target coordinate system based on the obtained transformation parameters. At the pixel level, a weighted fusion method is used to achieve a smooth transition between images. After the above processing, accurate stitching of multiple images can be achieved, outputting a seamless, high-resolution panoramic remote sensing image that meets the needs of subsequent refined monitoring of crop growth.

[0052] Specifically, in multi-scene image stitching, a common matching method is grayscale-based matching. This involves converting multiple images into grayscale images and transforming color information into individual grayscale channels. Image patches are extracted from overlapping regions, and the grayscale correlation between these patches is calculated to construct a grayscale similarity matrix. An optimization algorithm is then used to find the maximum value in the similarity matrix, determining the best-matching image patch. Based on the matching point, a geometric transformation model between the two images is determined, achieving image alignment. This method matches the grayscale texture information of the images without considering complex color variations, resulting in high computational efficiency. Furthermore, grayscale reflects the structural contours of the scene, making it more accurate in determining the geometric relationships between images.

[0053] Specifically, image matching based on grayscale information is directly built upon image grayscale values, avoiding the error accumulation caused by intermediate steps such as image segmentation and feature extraction, thus improving matching accuracy. Image grayscale is insensitive to changes in illumination and noise interference, making the matching results robust. Calculating only grayscale information results in low computational load and fast processing speed. In summary, the image stitching method based on grayscale matching is highly accurate and reliable, suitable for remote sensing image stitching tasks with high accuracy requirements, and can effectively serve the monitoring of crop growth.

[0054] Radiometric calibration can only compare images within the same scene using pixel values. To enhance the distinction between images of lodged and unlodged crops, calibration is used. Calibration is the process of transforming remote sensing data measurements into relative values ​​related to physical quantities such as absolute brightness, surface reflectance, or surface temperature. Radiometric calibration provides more comparable remote sensing data. Radiometric and reflectance calibration of remote sensing data can be performed using ENVI 5.3 software.

[0055] Specifically, the pixel values ​​(DN values) in raw digital images are significantly affected by shooting conditions and can only be used for comparative analysis within the same scene. To enhance the spectral reflectance differences between lodged and unlodged areas of crops captured at different times and under different lighting conditions, radiometric calibration is necessary. The purpose of radiometric calibration is to convert DN values ​​into standardized reflectance, eliminating the influence of variations in conditions such as solar illumination angle and sensor gain. This ensures comparability between images acquired at different times and with other regions of interest. After radiometric calibration, the pixel values ​​of the remote sensing image directly reflect the essential spectral characteristics of the land surface at that location. This provides a foundation for subsequent comparative analysis of lodged and unlodged crop areas using quantitative indicators.

[0056] Specifically, radiometric calibration is the process of converting the raw digital representation (DN) values ​​of remote sensing images into quantitative reflectance or temperature values ​​related to surface physical quantities. It converts the DN values ​​of each band into the absolute radiance or irradiance of the scene. Using the observation geometry and atmospheric parameters at the time of image acquisition, it further converts them into the directional reflectance or emissivity of the surface. Alternatively, it can be converted into standardized band brightness temperature or surface component temperature. Calibration eliminates the influence of variations in shooting conditions, allowing the quantitative reflectance or temperature values ​​of the image to reflect the essential spectral characteristics or thermal properties of the surface, providing a foundation for subsequent information extraction and analysis.

[0057] Specifically, ENVI 5.3, a remote sensing image processing software, can be used to perform radiometric calibration on raw images acquired by drones. Collect drone shooting parameters, such as lighting conditions and detector parameters, as auxiliary data required for calibration. Import the raw image into ENVI and select the "Radiometric Calibration" module. Follow the on-screen prompts to select the radiometric calibration type (reflectance or irradiance) and input the calibration parameters. Click "OK" to obtain the calibrated image. You can further select the "Convert Reflectance" module to obtain the standardized reflectance result. Using this software can effectively improve the automation of radiometric calibration, output quantitative result images, and provide a data foundation for subsequent analysis of crop growth.

[0058] The dataset construction includes the following steps: image annotation of the preprocessed data; block cropping of the annotated image data; and augmentation of the image data to expand the training samples. The expanded dataset is obtained by performing image flipping, geometric transformations, translation, rotation, cropping, adding noise, and color space transformations.

[0059] Specifically, on the preprocessed remote sensing images, areas of lodged and unlodged crops in the fields are manually labeled using the experience and knowledge of professionals, serving as samples for different categories. Based on the labels, the images are segmented and cropped to extract smaller image patches containing different categories, generating an initial sample set. The sample set is then augmented through operations such as rotation, flipping, and adding noise to increase the sample size and improve model robustness. Through these steps, a rich dataset of lodged crop images containing both positive and negative samples is constructed, which can be used for subsequent model training and validation, improving the intelligence level of crop lodging monitoring.

[0060] Specifically, to expand the dataset, the original images underwent various transformations, including: Flipping: Flipping the image horizontally or vertically. Geometric transformations: Scaling, stretching, etc. Translation: Moving the image in different directions. Rotation: Rotating the image at different angles. Cropping: Extracting local regions of the image. Adding noise: Adding Gaussian noise, etc. Color space transformation: Converting to HSV, YIQ, etc. The number of image samples after expansion can reach several times that of the original images, which helps improve the robustness of the model. Figure 3 This image displays pre-processed, manually labeled remote sensing images of crops. Red areas indicate lodged crops, while green areas represent normally growing crops.

[0061] Figure 3Here is a diagram of the PSPNet model according to an embodiment of this application: a convolution operation is performed on the input image to obtain a feature map; the feature map is subjected to four pooling operations with pooling kernels of different sizes to obtain multi-scale feature maps, wherein the larger the pooling kernel, the larger the feature scale of the obtained feature map; the feature maps of different scales are input into the feature integration module respectively; a convolution operation is performed on the feature map with the smallest scale; and a concatenation operation is performed on the feature maps of the other three scales.

[0062] Specifically, the structure and flow of the improved PSPNet network are as follows: The input image undergoes convolution to extract primary feature maps. These primary feature maps are then subjected to four-dimensional pyramid pooling using pooling kernels of different sizes (e.g., 1×1, 2×2, 3×3, 6×6) to obtain multi-scale pooled feature maps. These pooled feature maps of different scales are upsampled to a certain size and concatenated to fuse global information from different receptive fields. The concatenated feature maps are then compressed using 1×1 convolution and fused with the feature maps obtained in step 1. The fused features are then processed by a convolution classifier to perform classification prediction for each pixel.

[0063] Specifically, in multi-scale spatial pooling, the size of the pooling kernel corresponds to the scale of information contained in the feature map: the larger the pooling kernel, the more global and macroscopic the semantic information contained in the resulting feature map; the smaller the pooling kernel, the more local and detailed the information contained in the feature map. Pooled feature maps of different scales are input into the feature integration module for processing: convolution is performed on the feature map with the smallest pooling kernel (e.g., 1×1) to enhance its ability to extract detailed features. Other feature maps with larger kernel sizes are interpolated and concatenated to integrate semantic information at different scales. Finally, the feature maps at each scale are fused to form a feature representation that integrates both global and local information.

[0064] In the feature maps of the three scales, before the stitching operation, the feature map with the largest scale needs to be upsampled and the feature map with the smallest scale needs to be downsampled; attention fusion is performed on the stitched result; deconvolution is performed on the attention fusion result so that the feature map size of the deconvolution result is the same as that of the convolution result; dot product is performed on the convolution result and the deconvolution result; deconvolution is performed on the attention fusion result so that the feature map size of the deconvolution result is the same as that of the dot product result; stitching is performed on the dot product result and the deconvolution result; the stitched result is then subjected to convolution and sigmoid activation function operations in sequence to obtain the output image.

[0065] Specifically, before concatenating feature maps of different scales, they need to be sampled and adjusted to the same size: the feature map with the largest scale (e.g., 6×6 pooling) is upsampled to increase its spatial resolution. The feature map with the smallest scale (e.g., 1×1 pooling) is downsampled to reduce its resolution. The feature maps of intermediate scales (e.g., 2×2, 3×3 pooling) retain their original resolution. Then, the three scale feature maps after sampling and adjustment are concatenated along the channel dimension. To enhance feature representation, an attention mechanism is further applied to the concatenated features for weighted fusion of multi-scale features. Finally, a deconvolution operation is used to upsample the feature maps to the same spatial size as the original convolution result.

[0066] Specifically, the convolutional result of the primary feature extraction is multiplied by the deconvolutioned feature map from the feature integration module to achieve feature complementarity. The feature map obtained through attention fusion is also deconvolved, mapping it to the same spatial dimension as the multiplication result. The two feature maps are then concatenated along the channel dimension. A convolution operation is performed on the concatenated feature map to further extract features. The convolutional result is then activated by the Sigmoid activation function, ultimately outputting the classification result for each pixel. Through this process, the model effectively fuses features at different levels, enabling classification decisions to utilize both global and local information.

[0067] The feature integration module includes the following steps: The input image is sequentially subjected to convolution, ReLU activation, and batch normalization; the output result is subjected to two spatially separated dilated convolutions in parallel, followed by ReLU and batch normalization operations sequentially. The spatially separated dilated convolution consists of a horizontal one-dimensional convolution kernel and a vertical one-dimensional convolution kernel. The difference between the two spatially separated dilated convolutions is that in one case, the horizontal one-dimensional convolution is performed first, followed by the vertical one-dimensional convolution; in the other case, the vertical one-dimensional convolution is performed first, followed by the horizontal one-dimensional convolution. The batch normalized results of the two spatially separated dilated convolutions are concatenated; the concatenated result is then sequentially subjected to convolution, ReLU, and batch normalization operations.

[0068] Specifically, the feature integration module's processing flow is as follows: The input image undergoes convolution to extract primary features. The convolution result is then subjected to a non-linear transformation using the ReLU activation function. The activation result is then batch-normalized. The output of the batch-normalized function is input in parallel into two atrous convolutions with dilatations of 6 and 12, respectively. The results from each atrous convolution are then processed by the ReLU activation function and batch-normalized, respectively. The two normalized feature maps are concatenated across channels to obtain the module's final output. This multi-scale feature extraction and fusion provided by atrous convolution enhances the robustness of feature representation, providing information support for subsequent recognition.

[0069] Specifically, the spatial separation dilated convolution in the feature integration module employs two one-dimensional convolutions in different orders: the first dilated convolution performs a horizontal one-dimensional convolution with a kernel size of 1xk, sliding across the input feature map to extract horizontal information. This is followed by a vertical one-dimensional convolution with a kernel size of kx1 to extract vertical information. The second dilated convolution operates in the same order: first vertical one-dimensional convolution, then horizontal one-dimensional convolution. This two-order spatial separation operation allows for a more comprehensive and balanced extraction of both horizontal and vertical information from the feature map, enhancing feature representation capabilities. Using the same dilation rate in both operations ensures consistent spatial scale extraction, which is beneficial for subsequent feature fusion.

[0070] Specifically, the feature maps obtained from two separate spatially separated dilated convolutions, each batch-normalized, are concatenated along the channel dimension. The concatenated feature maps are then subjected to convolution, ReLU activation, and batch normalization. Convolution extracts a new representation of the concatenated features, ReLU introduces non-linearity, and batch normalization improves numerical stability. Through this convolution-activation-normalization process, two sets of dilated features are extracted and fused. Finally, the batch-normalized result of the convolution is output as the final feature representation of the module.

[0071] To evaluate the model's performance, this embodiment employs two metrics: pixel accuracy (PA) and mean intersection over union (MIoU). Pixel accuracy directly reflects the model's accuracy in extracting lodged areas, while MIoU directly reflects the model's segmentation precision. Regions correctly classified as lodged crops are denoted as true positives (TP); regions correctly classified as non-lodged crops are denoted as true negatives (TN); regions incorrectly classified as lodged crops are denoted as false positives (FP); and regions incorrectly classified as non-lodged crops are denoted as false negatives (FN).

[0072]

[0073]

[0074] Specifically, to evaluate the performance of the crop lodging recognition model, two evaluation metrics were used: Pixel Accuracy (PA), which represents the proportion of pixels correctly predicted as positive out of all pixels predicted as positive. It directly reflects the accuracy of the model in extracting lodged areas. Mean Intersection over Union (mIoU) is calculated as the average of the intersection and union ratios of the predicted and ground truth regions for each category. It reflects the model's ability to accurately segment different categories. Combining these two metrics provides a comprehensive assessment of the model's effectiveness in identifying and locating lodged areas. A higher PA value indicates more accurate extraction, while a higher mIoU value represents more refined segmentation. These metrics will serve as important criteria for evaluating the model's performance.

[0075] Specifically, when calculating pixel accuracy and mean intersection-over-union (MIU), the model's predicted class for each pixel is compared with the true class, which can be categorized into four cases: True Positives (TP): The number of samples where the model correctly predicted the pixel as lodged crop. True Negatives (TN): The number of samples where the model correctly predicted the pixel as non-lodged crop. False Positives (FP): The number of samples where the model incorrectly predicted the pixel as lodged crop. False Negatives (FN): The number of samples where the model incorrectly predicted the pixel as non-lodged crop. Based on these four cases, evaluation metrics can be calculated to assess the model's recognition and segmentation performance from different perspectives.

[0076] Figure 4 This is a wheat lodging area identification result image from one embodiment of this application. The UAV imagery is characterized by its large scale and high resolution, resulting in a massive data volume, which undoubtedly consumes a huge amount of storage space and computing power. Therefore, this embodiment decomposes the original RGB imagery along its color channels, splitting the original 3-channel data into single-channel data based on red, green, and blue, reducing the data volume to one-third of the original. These single-channel data are then input into the network for testing. The accuracy decreased by an average of 0.53%, and the cross-validation ratio (CVR) decreased by an average of 0.74%. It is evident that this model maintains excellent performance despite a significant reduction in data volume.

[0077] Specifically, remote sensing images of crop fields acquired by UAVs are characterized by their large scale and high resolution, resulting in a massive amount of data. This large-scale data places extremely high demands on storage space and computing power in image processing. To reduce data redundancy and improve processing efficiency, this solution adopts the following measures: Channel decomposition of the original RGB image is performed to extract individual red, green, and blue channels, reducing the data volume by two-thirds. Processing and analysis are performed on single-channel images, significantly reducing storage and computational load. The processing results of each channel are fused to ensure extraction accuracy. More efficient model structure designs are used, such as dilated convolution modules that save computational load. Distributed processing methods such as cloud computing are utilized to improve computing power. Through data volume control and computing power expansion, this solution can support intelligent analysis of large-scale UAV images, achieving efficient monitoring of crop lodging.

[0078] Specifically, the original RGB image was decomposed into three color channels: red, green, and blue, with each channel containing one-third the data volume of the original image. The single-channel image data for each of the three channels were then input into a segmentation network for testing. The network test results for each channel were then fused to obtain the final segmentation result of the lodged area in the crop field. Compared to directly using a 3-channel RGB image, channel decomposition testing reduced the data volume by two-thirds. While maintaining accuracy, it significantly reduced storage and computational load. Single-channel networks are more suitable for large-scale remote sensing image processing. Through channel decomposition, data volume control in UAV image processing was achieved, reducing storage and computational burden and improving processing efficiency.

[0079] Specifically, compared to using the original 3-channel RGB image, the results of testing with single-channel input are as follows: pixel accuracy decreased by an average of 0.53%, and the average intersection-over-union (IoU) ratio decreased by an average of 0.74%. The combined changes in these two metrics show that even with a three-fold reduction in data volume, the model's segmentation performance only decreased slightly. This demonstrates that the model maintains excellent recognition and segmentation performance while significantly reducing data redundancy. Channel decomposition reduces storage and computational requirements, making it more suitable for large-scale remote sensing image processing and possessing significant engineering application value.

[0080] The foregoing illustrative description of the invention and its embodiments is not restrictive and can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. The accompanying drawings are only one embodiment of the invention, and the actual structure is not limited thereto. No reference numerals in the claims should limit the scope of the claims. Therefore, if a person skilled in the art, inspired by this description, designs a similar structure and embodiment without departing from the spirit of the invention, such design should fall within the scope of protection of this patent. Furthermore, the word "comprising" does not exclude other elements or steps, and the word "a" preceding an element does not exclude the inclusion of "a plurality" of that element. Multiple elements stated in the product claims can also be implemented by a single element through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A method for identifying lodged areas based on remote sensing images of crops, comprising: Collect remote sensing images of crops; Preprocessing of the acquired remote sensing images includes image stitching, radiometric calibration, and image cropping; Feature extraction is performed on the processed remote sensing images to obtain color features, texture features, and vegetation index features; A semantic segmentation algorithm based on convolutional neural networks is used to identify lodged areas of crops in images. The convolutional neural network uses the PSPNet network; The encoder uses a pre-trained ResNet as the encoder to extract features from the input image; The fusion module contains convolutional layers with pooling kernels of different sizes, which perform multi-scale pooling on the features extracted by the encoder to obtain feature maps of different scales. The decoder contains n upsampling layers and n convolutional layers. The i-th upsampling layer enlarges the i-th feature map to the size of the (i+1)-th feature map, and the i-th convolutional layer fuses the feature map output by the i-th upsampling layer with the feature map output by the i-th upsampling layer. Includes the following steps: Features of the input image are extracted through convolutional layers and used as the first feature map. The first feature map is multi-scale pooled using pooling kernels of different sizes to obtain pooled feature maps of different scales, which are then used as the second feature map. Feature fusion is performed on the second feature maps of different scales. Specifically, the smallest scale feature map is convolved, and the other feature maps are upsampled or downsampled and concatenated to obtain a concatenated feature map. An attention mechanism is used to fuse the spliced ​​feature maps to obtain a fused feature map; The fused feature map is upsampled by deconvolution to obtain an upsampled feature map; Perform dot product, concatenation and convolution operations on the upsampled feature map and the minimum scale feature map to output an image containing the prediction results of the fallen area; Feature fusion of second feature maps at different scales includes the following steps: Second feature maps of different scales are used as inputs to the fusion module; The first feature is extracted from the second feature map by the first convolutional layer, and feature maps of different scales are extracted as the first feature map. The second convolutional layer is used to perform bidirectional spatial separation convolution on the first feature map to extract feature maps of different scales as the second feature map. The bidirectional spatial separation convolution includes horizontal one-dimensional convolution and vertical one-dimensional convolution. The second feature map extracted through bidirectional spatial separation convolution is concatenated to obtain the concatenated feature map; A third convolutional layer is used to perform convolution operations on the concatenated feature map, and the fused feature map is output. When performing horizontal one-dimensional convolution, the length of the convolution kernel is set to be greater than its width in order to extract the horizontal features of the image. When performing vertical one-dimensional convolution, the width of the convolution kernel is set to be greater than its length to extract the vertical features of the image.

2. The method for identifying lodged areas based on remote sensing images of crops according to claim 1, characterized in that: Remote sensing image acquisition of crop fields using drones includes the following steps: Drones use multispectral cameras to collect remote sensing images of crop fields; The multispectral camera splits the acquired remote sensing images into RGB three channels to obtain three single-channel images of red, green and blue respectively; The acquired red, green, and blue single-channel images are stored separately for each channel to reduce storage requirements.

3. The method for identifying lodged areas based on remote sensing images of crops according to claim 2, characterized in that: Image stitching of acquired remote sensing images includes the following steps: The acquired single-channel images are arranged and combined in chronological order; Image fusion processing is performed on the overlapping areas of adjacent images. The overlapping areas represent the common surface areas covered by images taken from the same geographical location. The images are stitched together to generate a complete remote sensing image covering the target crop field.

4. The method for identifying lodging areas based on remote sensing images of crops according to claim 3, characterized in that: Radiometric calibration of acquired remote sensing images includes the following steps: Illumination parameters when acquiring remote sensing images using drones; An image radiometric calibration model is constructed using the collected illumination parameters; The illumination information of each pixel in the stitched overall remote sensing image is corrected by constructing an image radiometric calibration model to eliminate the influence of different illumination conditions.

5. The method for identifying lodging areas based on remote sensing images of crops according to claim 4, characterized in that: Cropping the acquired remote sensing images includes the following steps: The radiometrically calibrated overall remote sensing image is used as the input image. The target crop field region and background region in the input image are obtained through an image segmentation algorithm; The boundary of the target crop field area is calculated using a boundary detection algorithm, and the coordinates of the polygonal region of the target crop field area are obtained. Based on the obtained polygonal region coordinates, the input image is cropped and saved as the target region.

6. A lodging area identification system based on remote sensing images of crops, used to execute the lodging area identification method based on remote sensing images of crops as described in any one of claims 1 to 5, comprising: The image acquisition module acquires remote sensing images of crop fields; The preprocessing module performs image stitching, radiometric calibration, and image cropping on the acquired remote sensing images; The feature extraction module extracts features from the preprocessed remote sensing image, obtaining color features, texture features, and vegetation index features respectively. The recognition module, based on a pre-trained convolutional neural network model, performs semantic segmentation on the remote sensing images after feature extraction to identify lodged areas of crops in the remote sensing images. The convolutional neural network model adopts the PSPNet structure. The PSPNet structure includes: Encoder, used for feature extraction; The fusion module is used for multi-scale feature fusion. The decoder is used for pixel prediction and upsampling to the original resolution.

Images