A soybean early missing plant position extraction method and system based on multi-source remote sensing and sowing row topological constraints

Abstract

The application discloses a soybean early seedling-lacking position extraction method and system based on multi-source remote sensing and sowing row topological constraint, relates to the technical field of agricultural remote sensing monitoring, and comprises the following steps: data acquisition, data joint preprocessing, spatial registration and fusion; constructing a multi-dimensional feature dataset and optimizing the same to obtain an optimized feature set; identifying soybean seedlings based on the optimized feature set and a preset model, extracting seedling centroids, and fitting sowing rows based on the seedling centroid point set; generating a theoretical position sequence according to a preset plant distance or an adaptive plant distance; matching the actual seedling centroid with the theoretical position; and determining the theoretical position that is not matched with the actual seedling as a seedling-lacking position. The application fuses unmanned aerial vehicle LiDAR three-dimensional structure, multispectral information and image texture features, optimizes single remote sensing data, solves the problems of low seedling-lacking recognition accuracy and inaccurate position positioning caused by shadow and soil color difference interference at the soybean seedling stage, and improves the soybean early seedling-lacking discrimination accuracy and position extraction efficiency.

Description

Technical Field

[0001] This invention relates to the field of agricultural remote sensing monitoring technology, and more specifically to a method and system for extracting early missing seedling positions in soybeans based on multi-source remote sensing and sowing row topological constraints. Background Technology

[0002] Soybeans are an important dual-purpose crop for grain and oil, and the seedling emergence directly determines the final yield. After emergence, soybeans are highly susceptible to gaps in the rows due to factors such as uneven sowing depth, soil moisture, pests and diseases, bird damage, and land preparation quality. The early stage of soybean growth (5-12 days after emergence) is the golden period for replanting and reseeding. Quickly and accurately identifying the location and number of missing seedlings in the field is a key step in achieving refined field management and ensuring stable yields and increased income.

[0003] Traditional soybean seedling loss detection relies primarily on manual field inspections, which suffers from drawbacks such as high labor intensity, low efficiency, time-consuming large-scale field inspections, and significant errors in manual counting, making it unsuitable for the monitoring needs of large-scale, contiguous soybean planting areas. With the widespread adoption of UAV remote sensing technology, existing techniques mostly utilize single visible light UAV images or single multispectral images for crop seedling loss identification. However, current UAV remote sensing seedling loss detection methods largely focus on pixel-level or area-level seedling / bare soil identification, making it difficult to distinguish between normal bare soil between rows and between plants, and the actual location of missing seedlings. Furthermore, they struggle to directly output plant location coordinates suitable for replanting operations.

[0004] LiDAR can accurately acquire structural information such as crop three-dimensional height and point cloud density, and is not affected by light or shadow. Multispectral images can capture differences in crop spectral vegetation indices, and texture features can characterize the spatial distribution patterns of field surfaces.

[0005] Therefore, how to propose a method and system for extracting early missing soybean plant positions based on multi-source remote sensing and sowing row topological constraints, which can transform remote sensing identification results into theoretical plant position matching results within the sowing row and overcome the shortcomings of existing technologies, is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] In view of this, the present invention provides a method and system for extracting early-stage missing seedling locations in soybeans based on multi-source remote sensing and sowing row topological constraints. This method integrates multi-dimensional features such as three-dimensional structure, spectrum, and texture, combines a supervised learning model to identify candidate regions for soybean seedlings, and further utilizes sowing row topological constraints and theoretical plant position sequences to locate missing seedling locations, thereby improving the efficiency and accuracy of seedling missing seedling detection. To achieve the above objectives, the present invention adopts the following technical solution: A method for extracting early missing soybean plant locations based on multi-source remote sensing and sowing row topological constraints includes: A drone equipped with a LiDAR and a multispectral camera was used to conduct aerial surveys of the target soybean field, simultaneously collecting LiDAR point cloud data and multispectral raw image data of soybean seedlings. The LiDAR point cloud data and multispectral raw image data were preprocessed separately, and then the two types of preprocessed data were jointly registered as pixels and point clouds. The registered data is fused at the pixel level to generate a multi-channel fused feature stack; Based on a multi-channel fusion feature stack, spectral features, LiDAR 3D structural features, and texture features based on the gray-level co-occurrence matrix (GLCM) are extracted and integrated to generate the original multi-dimensional feature set. Calculate the mutual information value between each feature in the original multidimensional feature set and the category label of the training sample. The category label of the training sample includes at least the soybean seedling label and the background label. Select effective features with mutual information values ​​greater than a preset threshold to obtain an optimized feature set. Based on the optimized feature set and combined with the preset model, soybean seedlings are identified to obtain candidate regions for soybean seedlings and extract the centroid set of actual seedlings. Sowing rows are fitted based on the actual seedling centroid set, and the starting and ending points of each sowing row segment are determined according to the intersection of the sowing row and the target field boundary, the sowing machine's operating trajectory, manually set row endpoints, or extrapolation results of the first and last seedlings within the sowing row. A theoretical plant position sequence is generated along the sowing row direction according to the preset plant spacing or locally adaptive plant spacing. The actual seedling centroids are matched with the theoretical plant positions according to a preset matching distance threshold. Theoretical plant positions that do not match the actual seedling centroids are identified as missing seedling positions.

[0007] Optionally, the preprocessing of LiDAR point cloud data and multispectral raw image data includes: The original point cloud was sequentially denoised, ground point filtered, and point cloud normalized to remove flying points, noise points, and interference from farmland topographic undulations, resulting in a normalized LiDAR point cloud. The normalized point cloud was then converted into raster data to generate digital elevation raster and plant height raster. The original multispectral image is sequentially subjected to radiometric correction and geometric correction. The image DN value is converted into ground reflectance to eliminate errors caused by atmospheric, illumination and attitude deviations, resulting in a corrected multispectral image.

[0008] Optionally, the step of performing pixel-point cloud joint registration on the two types of preprocessed data includes: Initial spatial registration is performed based on UAV POS data, camera intrinsic and extrinsic parameters, and LiDAR extrinsic parameters. Fine registration is then performed by combining ground control points, obvious ground feature control points, or feature matching methods. The LiDAR raster data is then resampled to the multispectral image coordinate system.

[0009] Optionally, the step of performing pixel-level fusion on the registered data to generate a multi-channel fusion feature stack includes: The reflectance data of each band of the registered multispectral image and the LiDAR plant height raster data are channel-by-channel stitching and fusion. Each pixel contains both multispectral reflectance information and LiDAR height information, generating a multi-channel fusion feature stack. The fusion rule is that the pixels are superimposed one-to-one without changing the spatial resolution and geometric position of the original image.

[0010] Optionally, the step of extracting spectral features, LiDAR 3D structural features, and texture features based on the gray-level co-occurrence matrix (GLCM) based on a multi-channel fusion feature stack, and integrating them to generate an original multi-dimensional feature set, includes: Based on the multispectral band reflectance of the multichannel fusion feature stack, the Normalized Difference Vegetation Index (NDVI), Red Edge Vegetation Index (NDRE), and Soil-Regulated Vegetation Index (OSAVI) are calculated as spectral features. The average plant height, height variance, and point cloud density (calculated from the local neighborhood of the original normalized point cloud) of the corresponding pixels in the fused image are extracted as LiDAR 3D structural features. Using the Gray-Level Co-occurrence Matrix (GLCM), a sliding window is selected to perform gray-level quantization on NDVI raster, OSAVI raster, and LiDAR normalized height raster, and GLCM texture features are calculated. The spectral features, LiDAR 3D structural features, and texture features are concatenated along the pixel dimension to construct the original multidimensional feature set.

[0011] Optionally, the Normalized Difference Vegetation Index (NDVI), the Red Edge Vegetation Index (NDRE), and the Soil-Regulated Vegetation Index (OSAVI) are calculated as spectral features, including: ; ; ; in, For near-infrared reflectivity, For red light band reflectivity, This refers to the reflectivity of the red-edge band.

[0012] Optionally, calculating the mutual information values ​​between each feature in the original multidimensional feature set and the class labels of the training samples includes: ; in, The mutual information value between feature variable X and training sample class labels Y, where Y includes at least soybean seedling labels and background labels. The joint probability distribution of features and labels. For the characteristic marginal probability distribution, The label edge probability distribution is defined; the mutual information threshold is set to the median of all mutual information values, and features with mutual information values ​​greater than the threshold are retained to obtain the optimized feature set.

[0013] Optionally, the preset model includes: random forest, gradient boosting tree, and support vector machine.

[0014] Optionally, the step of identifying soybean seedlings based on an optimized feature set and a preset model further includes: For sections where the distance between the centroids of adjacent seedlings is greater than the preset plant spacing threshold, the number of missing seedlings is calculated based on the ratio of this distance to the standard plant spacing or the locally adaptive plant spacing, and the coordinates of the missing seedlings are corrected.

[0015] Optionally, a soybean early missing seedling position extraction system based on multi-source remote sensing and sowing row topology constraints includes: an UAV aerial survey module, used to conduct aerial surveys of the target soybean field using a UAV equipped with a LiDAR and a multispectral camera, and simultaneously collect LiDAR point cloud data and multispectral raw image data of soybean seedlings. The multi-source data preprocessing module is used to preprocess LiDAR point cloud data and multispectral raw image data respectively, and then perform pixel-point cloud joint registration on the two types of preprocessed data. The data fusion module is used to perform pixel-level fusion on the registered data to generate a multi-channel fusion feature stack. The multi-dimensional feature extraction module is used to extract spectral features, LiDAR three-dimensional structural features, and texture features based on the gray-level co-occurrence matrix (GLCM) based on a multi-channel fusion feature stack, and integrate them to generate the original multi-dimensional feature set. The feature optimization module is used to calculate the mutual information value between each feature in the original multidimensional feature set and the category label of the training sample. The category label of the training sample includes at least a soybean seedling label and a background label. Valid features with mutual information values ​​greater than a preset threshold are selected to obtain an optimized feature set. The module for extracting and outputting missing seedling positions is used to identify soybean seedlings based on an optimized feature set and a preset model, obtain candidate regions for soybean seedlings, and extract the centroid set of actual seedlings. It then fits the planting rows based on the actual seedling centroid set and determines the start and end points of each planting row segment according to the intersection of the planting row and the target field boundary, the planter's operating trajectory, manually set row endpoints, or extrapolation results of the first and last seedlings within the planting row. It generates a theoretical seedling position sequence along the planting row direction based on a preset plant spacing or a locally adaptive plant spacing. Finally, it matches the actual seedling centroids with the theoretical seedling positions according to a preset matching distance threshold, and identifies theoretical seedling positions that do not match actual seedling centroids as missing seedling positions.

[0016] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a method and system for extracting early missing seedling positions in soybeans based on multi-source remote sensing and sowing row topological constraints, which has the following beneficial effects: This invention improves the stability of soybean seedling candidate region identification by using multi-source remote sensing features, and converts pixel-level seedling identification results into plant position-level missing seedling determination results by utilizing the topological constraints of sowing rows and theoretical plant position sequences. This reduces misjudgments caused by bare soil and soil background between rows, and outputs the coordinates of missing seedlings, row number, and number of missing seedlings, which facilitates manual replanting or fixed-point replanting by agricultural machinery. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a method for extracting early missing soybean plant positions based on multi-source remote sensing and sowing row topological constraints, provided by the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] This invention discloses a method for extracting early missing seedling locations in soybeans based on multi-source remote sensing and sowing row topological constraints, such as... Figure 1 As shown, it includes: A drone equipped with a LiDAR and a multispectral camera was used to conduct aerial surveys of the target soybean field, simultaneously collecting LiDAR point cloud data and multispectral raw image data of soybean seedlings. LiDAR point cloud data and multispectral raw image data are preprocessed separately, and then pixel-point cloud joint registration is performed on the two types of preprocessed data to achieve the unification of spatial coordinates of multi-source data. The registered data is fused at the pixel level to generate a multi-channel fused feature stack; Based on a multi-channel fusion feature stack, spectral features, LiDAR 3D structural features, and texture features based on the gray-level co-occurrence matrix (GLCM) are extracted and integrated to generate the original multi-dimensional feature set. Calculate the mutual information value between each feature in the original multidimensional feature set and the category label of the training sample. The category label of the training sample includes at least the soybean seedling label and the background label. Select effective features with mutual information values ​​greater than a preset threshold to obtain an optimized feature set. Based on the optimized feature set and combined with the preset model, soybean seedlings are identified to obtain candidate regions for soybean seedlings and extract the centroid set of actual seedlings. Sowing rows are fitted based on the actual seedling centroid set, and the starting and ending points of each sowing row segment are determined according to the intersection of the sowing row and the target field boundary, the sowing machine's operating trajectory, manually set row endpoints, or extrapolation results of the first and last seedlings within the sowing row. A theoretical plant position sequence is generated along the sowing row direction according to the preset plant spacing or locally adaptive plant spacing. The actual seedling centroids are matched with the theoretical plant positions according to a preset matching distance threshold. Theoretical plant positions that do not match the actual seedling centroids are identified as missing seedling positions.

[0021] Furthermore, the drone equipped with LiDAR and a multispectral camera specifically includes: Planned UAV aerial survey parameters: set flight altitude to 8m, flight speed to 1~3m / s, forward overlap to 75%~85%, and lateral overlap to 60%~70%, to ensure full coverage of the target soybean field. Equipment parameter configuration: LiDAR equipment sampling frequency ≥200kHz, multispectral camera includes four standard bands: green light, red light, red edge, and near-infrared; LiDAR data is not used as the sole basis for determining missing seedlings, but is used to extract the three-dimensional structural features of seedling candidate areas, including local relative height, canopy height difference, point cloud density, and height continuity, to confirm the structure of seedling candidate areas when the spectrum is interfered with by shadows or soil background; Synchronous aerial survey data acquisition: The UAV flies along the preset route, and the LiDAR collects three-dimensional point cloud data of the field in real time. The multispectral camera simultaneously captures multispectral images. The acquisition time is the seedling stage 5 to 12 days after soybean emergence, preferably 6 to 10 days after emergence. The growth stage is from the cotyledon unfolding to the early stage of the first compound leaf unfolding. At this time, there is no obvious canopy overlap between adjacent single plants, and the sowing row structure is clear, which is suitable for extracting the centroid of single plants and inferring the position of missing seedlings, thus completing the original multi-source data acquisition.

[0022] Furthermore, the preprocessing of LiDAR point cloud data and multispectral raw image data includes: The original point cloud was sequentially denoised, ground point filtered, and point cloud normalized to remove flying points, noise points, and interference from farmland topographic undulations, resulting in a normalized LiDAR point cloud. The normalized point cloud was then converted into raster data to generate digital elevation raster and plant height raster. The original multispectral image is sequentially subjected to radiometric correction and geometric correction. The image DN value is converted into ground reflectance to eliminate errors caused by atmospheric, illumination and attitude deviations, resulting in a corrected multispectral image.

[0023] Furthermore, the step of performing pixel-point cloud joint registration on the two types of preprocessed data includes: Using the corrected multispectral image as a reference, uniformly distributed ground control points are selected. Feature points are extracted using a scale-invariant feature transformation algorithm, and mismatched points are eliminated using a random sample consensus algorithm. The transformation matrix is ​​then solved, and the LiDAR raster data and multispectral image are registered at the pixel level to align the coordinates of the same ground feature in both types of data. Specifically: first, initial spatial registration is performed using UAV POS data, camera intrinsic and extrinsic parameters, and LiDAR extrinsic parameters; then, fine registration is performed using ground control points or obvious ground feature control points; finally, the LiDAR height raster is resampled to the spatial resolution of the multispectral image. Registration error control requirements are clearly defined, such as a registration error of less than 0.5 pixels or less than a certain centimeter-level threshold.

[0024] Furthermore, the step of performing pixel-level fusion on the registered data to generate a multi-channel fusion feature stack includes: The reflectance data of each band of the registered multispectral image and the LiDAR plant height raster data are channel-by-channel stitching and fusion. Each pixel contains both multispectral reflectance information and LiDAR height information, generating a multi-channel fusion feature stack. The fusion rule is that the pixels are superimposed one-to-one without changing the spatial resolution and geometric position of the original image.

[0025] Furthermore, the process of extracting spectral features, LiDAR 3D structural features, and texture features based on the gray-level co-occurrence matrix (GLCM) from a multi-channel fusion feature stack, and integrating them to generate an original multi-dimensional feature set, includes: Based on the multispectral band reflectance of the multichannel fusion feature stack, the Normalized Difference Vegetation Index (NDVI), Red Edge Vegetation Index (NDRE), and Soil-Regulated Vegetation Index (OSAVI) are calculated as spectral features. The average plant height, height variance, and point cloud density of corresponding pixels in the fused image are extracted as LiDAR 3D structural features. The gray-level co-occurrence matrix (GLCM) is used to quantify the gray levels of NDVI, OSAVI, and LiDAR normalized height rasters. GLCM texture features are calculated in the 0°, 45°, 90°, and 135° directions using 3×3, 5×5, and 7×7 sliding windows. The texture scale and direction combination participating in the classification model are determined by the importance of training set features or cross-validation.

[0026] The spectral features, LiDAR 3D structural features, and multi-scale texture features are stitched together according to the pixel dimension to construct the original multi-dimensional feature set.

[0027] Furthermore, the Normalized Difference Vegetation Index (NDVI), the Red Edge Vegetation Index (NDRE), and the Soil-Regulated Vegetation Index (OSAVI) were calculated as spectral features, including: ; ; ; in, For near-infrared reflectivity, For red light band reflectivity, This refers to the reflectivity of the red-edge band.

[0028] Furthermore, the calculation of the mutual information values ​​between each feature in the original multidimensional feature set and the class labels of the training samples includes: ; in, The mutual information value between feature variable X and training sample class labels Y, where Y includes at least soybean seedling labels and background labels. The joint probability distribution of features and labels. For the characteristic marginal probability distribution, Define the probability distribution of the label margins; set the mutual information threshold to the median of all mutual information values, and retain features with mutual information values ​​greater than the threshold.

[0029] Furthermore, the variance inflation factor is calculated for the features after mutual information filtering, and features with a VIF greater than 10 are iteratively removed until the VIF of the remaining features is no greater than 10, resulting in an optimized feature set with low redundancy and low collinearity.

[0030] Furthermore, the preset models include: random forest, gradient boosting tree, and support vector machine.

[0031] Furthermore, the soybean seedling identification based on the optimized feature set and combined with the preset model includes: Input the optimized feature set of the field to be detected into the pre-trained model, and output the candidate regions of soybean seedlings; Connectivity analysis was performed on the candidate regions of soybean seedlings to extract the centroid coordinates of individual soybean seedlings; The seeding row is fitted based on the centroid coordinates of the seedlings, and a theoretical plant position sequence is generated; The actual seedling centroid is matched with the theoretical plant position sequence using a preset matching distance threshold; The theoretical plant positions that do not match the actual seedling centroid are identified as missing plant positions; For sections where the distance between the centroids of adjacent seedlings is greater than the preset plant spacing threshold, the number of missing seedlings is calculated based on the ratio of this distance to the standard plant spacing or the locally adaptive plant spacing, and the coordinates of the missing seedlings are corrected.

[0032] Furthermore, based on the seedling centroid fitting the standard sowing row in the field, and according to the preset standard sowing plant spacing L in the field, a seedling missing threshold is set; the seedling centroids are traversed sequentially along each fitted sowing row, and when the row distance between two adjacent seedling centroids is greater than the preset plant spacing threshold, the segment is marked as a suspected seedling missing segment; the number of missing plants is determined according to the ratio of the row distance to the standard plant spacing or the local adaptive plant spacing, and the coordinates of the missing plant position are determined based on the theoretical plant position sequence; Extract the geographic coordinates, number of missing seedlings, and row number of all missing seedlings to generate a statistical table and a visual distribution map of early soybean missing seedlings.

[0033] In a specific embodiment, for a fixed standard plant spacing threshold of 1.5L, the row spacing in the field headland, turning areas, and manual reseeding areas is disordered, and the fixed threshold misses seedling gaps. DBSCAN clustering is used to adaptively divide the planting rows, dynamically generating a specific seedling gap detection threshold for each row, and adaptively determining seedling gaps in the rows by segmentation. Specifically: Estimate the main direction of the planting rows across the field; Based on the distance along the vertical direction of the sowing rows, the centroid of the seedlings is divided into different sowing rows; Fit a straight line or curve to the centroid of each sowing row; Connect rows that are interrupted due to continuous missing seedlings; Obtain the complete seed row.

[0034] Extract the seedling centroid, and DBSCAN spatial clustering automatically divides each sowing row; The row-wise distance between the centroids of adjacent seedlings within the same sowing row was statistically analyzed. After removing abnormally large spacing, the median of the remaining row-wise spacing was taken as the local plant spacing. The threshold for judging missing seedlings in that row was set to 1.5L. i The location of missing seedlings is determined independently row by row.

[0035] Furthermore, the theoretical plant positions are interpolated at equal intervals along the sowing row direction. When the distance between the centroids of two consecutive actual seedlings is greater than the preset plant spacing threshold, the number of missing plants is calculated based on the ratio of this distance to the standard plant spacing, and the coordinates of the missing plant positions are generated between the centroids of the two actual seedlings according to the standard plant spacing. For example: if the row distance between two adjacent actual seedling centroids is d and the standard plant spacing is L, then the number of missing plants m can be determined by m=round(d / L)-1; when m≥1, m theoretical missing plant positions are generated at intervals of L along the line connecting the two seedling centroids.

[0036] The one-to-one matching result between the theoretical plant position and the actual seedling centroid is used as the main determination result of the missing plant position; the integer multiple relationship between the row distance of adjacent seedlings and the local plant distance is used to verify and correct the number of missing plants in continuous missing plant sections, and the missing plant positions obtained by matching the theoretical plant position are not regenerated.

[0037] In a specific embodiment, a method for extracting early-stage missing soybean seedling positions based on multi-source remote sensing and sowing row topological constraints is proposed. To distinguish between permanently missing seedlings in bare soil and temporarily late-emerging weak seedlings and avoid waste from replanting, a two-stage temporal differential feature is used. The first aerial survey is conducted 5-7 days after soybean emergence, and the second aerial survey is conducted 9-12 days after emergence. Temporal differential spectrum, differential height, and differential texture features are constructed. Weak seedlings continuously increase in height / NDVI over time, while truly missing seedlings do not change, thus achieving the distinction between the two.

[0038] LiDAR+ multispectral data were collected twice, at T1 (5 days after seedling emergence) and T2 (10 days after seedling emergence). The two sets of data were uniformly registered, and the difference features were calculated pixel by pixel: ; ; ; The temporal difference features are incorporated into the feature set, and the preset model adds three types of temporal discrimination dimensions.

[0039] In a specific implementation, a soybean early missing seedling position extraction system based on multi-source remote sensing and sowing row topology constraints includes: an UAV aerial survey module, used to conduct aerial surveys of the target soybean field using a UAV equipped with a LiDAR and a multispectral camera, and simultaneously collect LiDAR point cloud data and multispectral raw image data of soybean seedlings. The multi-source data preprocessing module is used to preprocess LiDAR point cloud data and multispectral raw image data respectively, and then perform pixel-point cloud joint registration on the two types of preprocessed data. The data fusion module is used to perform pixel-level fusion on the registered data to generate a multi-channel fusion feature stack. The multi-dimensional feature extraction module is used to extract spectral features, LiDAR three-dimensional structural features, and texture features based on the gray-level co-occurrence matrix (GLCM) based on a multi-channel fusion feature stack, and integrate them to generate the original multi-dimensional feature set. The feature optimization module is used to calculate the mutual information value between each feature in the original multidimensional feature set and the category label of the training sample. The category label of the training sample includes at least a soybean seedling label and a background label. Valid features with mutual information values ​​greater than a preset threshold are selected to obtain an optimized feature set. The module for extracting and outputting missing seedling positions is used to identify soybean seedlings based on an optimized feature set and a preset model, obtain candidate regions for soybean seedlings, and extract the centroid set of actual seedlings. It then fits the planting rows based on the actual seedling centroid set and determines the start and end points of each planting row segment according to the intersection of the planting row and the target field boundary, the planter's operating trajectory, manually set row endpoints, or extrapolation results of the first and last seedlings within the planting row. It generates a theoretical seedling position sequence along the planting row direction based on a preset plant spacing or a locally adaptive plant spacing. Finally, it matches the actual seedling centroids with the theoretical seedling positions according to a preset matching distance threshold, and identifies theoretical seedling positions that do not match actual seedling centroids as missing seedling positions.

[0040] Specifically, matching the actual seedling centroid with the theoretical plant position according to the preset matching distance threshold includes: sequentially calculating the Euclidean distance between the actual seedling centroid and the theoretical plant position, comparing the Euclidean distance with the matching threshold, and marking targets that exceed the threshold. Candidate matching relationships between actual seedling centroids and theoretical plant positions are established within the same sowing row; when the distance between the actual seedling centroid and the theoretical plant position along the row direction is not greater than the first matching threshold, and the distance perpendicular to the sowing row direction is not greater than the second matching threshold, the two are determined as a candidate matching pair; one-to-one matching is performed according to the principle of minimum distance, so that each actual seedling centroid matches at most one theoretical plant position, and each theoretical plant position matches at most one actual seedling centroid.

[0041] Furthermore, the starting and ending points of the theoretical plant position sequence are determined by the plot boundary, the sowing row boundary, the operating trajectory of the seeder, manually set row endpoints, or extrapolation of the first and last seedlings within the sowing row.

[0042] When generating the theoretical plant position sequence, the start and end positions of the row segment are determined based on the intersection of the sowing row and the boundary of the target field. The theoretical plant position sequence is generated along the direction of the sowing row using the starting point of the row segment, the preset plant spacing or the locally adaptive plant spacing, and the theoretical plant position sequence is allowed to be translated as a whole.

[0043] Specifically, within a local adaptive plant spacing range, multiple candidate phase offsets are set for the theoretical plant position sequence; the effective matching number and the total matching distance between the theoretical plant position and the actual seedling centroid are calculated for each candidate phase offset; the candidate phase offset corresponding to the largest effective matching number and the smallest total matching distance is determined as the plant position phase offset of the target sowing row; and the theoretical plant position sequence is corrected according to the plant position phase offset.

[0044] Furthermore, it also includes a data storage module and a human-computer interaction module; the data storage module is used to store raw aerial survey data, preprocessed data, feature datasets, model parameters, and seedling shortage identification results; the human-computer interaction module is used to receive aerial survey parameters and plant spacing thresholds input by the user, and display images, features, classification results, and seedling shortage distribution maps.

[0045] In a specific embodiment, a method for extracting early missing soybean plant locations based on multi-source remote sensing and sowing row topological constraints includes the following steps: I. Preliminary Preparations 1. Experimental field and crop conditions A contiguous soybean planting field was selected, and mechanical row sowing was used. The standard design plant spacing L=10cm and row spacing 40cm. The monitoring window period was 5-12 days after emergence (suitable for replanting), divided into two aerial surveys: T1 on the 5th day after emergence and T2 on the 10th day after emergence, to distinguish between late-emerging, weak seedlings and permanently missing seedlings in bare soil. The experimental field included various interference scenarios: low-lying shaded areas, weed belts, areas with soil color differences, and areas with slight topographic undulations, to verify the algorithm's anti-interference ability. That is, the background labels included: shadow, weeds, and color difference.

[0046] Non-target vegetation was eliminated based on features such as the sowing row buffer zone, object area, object shape, distance from the center line of the sowing row, and LiDAR height. Green vegetation objects that deviated from the center line of the sowing row and did not meet the morphological characteristics of soybean seedlings were marked as weeds or non-target vegetation and were not included in the theoretical plant position matching.

[0047] 2. UAV hardware and aerial survey parameter configuration

[0048] A hexacopter drone equipped with a LiDAR sensor and a 4-band multispectral camera: LiDAR parameters: sampling frequency 220kHz, ranging accuracy ±2cm, can output 3D point cloud coordinates and point cloud density; Multispectral camera bands: green light, red light, red-edge RE, near-infrared NIR; Aerial survey operation parameters: In small-scale trials or high-precision verification scenarios, the flight altitude can be 5~10m; in large-scale field applications, the flight altitude can be adjusted according to the target resolution, plant spacing, and sensor parameters, for example, 10~30m, and the plant positioning accuracy requirements can be met by using RTK / PPK and a small number of ground control points. Example: Flight altitude 8m, flight speed 2m / s, lateral overlap 80%, lateral overlap 65%; Ground auxiliary equipment: Evenly distributed ground control points (15 per hectare) are set up for later registration and correction.

[0049] 3. Software operating environment

[0050] The data processing end is equipped with a Windows server, configured with Python 3.9, GDAL, PCL point cloud processing library, Scikit-learn, OpenCV, and Matplotlib; modular processing programs are built to correspond to the seven functional modules of the system.

[0051] II. Complete Implementation Steps

[0052] Step 1: Collect multi-source raw data simultaneously using UAV aerial surveying.

[0053] Plan a rectangular flight path and fly back and forth along the direction perpendicular to the soybean planting rows; Aerial surveys were completed at two time points, T1 and T2, simultaneously collecting two types of data: LiDAR raw 3D point cloud (including 3D coordinates, echo intensity, and point cloud density); Multispectral raw imagery (DN value format, with GPS geographic coordinates); The raw data is categorized by plot number and collection time and stored in the data storage module.

[0054] Step 2: Multi-source data preprocessing by channel + pixel-point cloud joint registration

[0055] (1) LiDAR point cloud preprocessing

[0056] Noise reduction: Remove discrete flying points and ultra-low / ultra-high noise points; Ground point filtering: The fabric simulation filter CSF is used to separate ground points from plant points; Point cloud normalization: eliminates farmland topographic undulations and calculates the relative height of plants; Raster conversion: Generate digital elevation raster (DEM) and plant height raster (H), with a spatial resolution of 0.05m.

[0057] (2) Multispectral image preprocessing

[0058] Perform three levels of calibration sequentially: Radiometric correction: Converting image DN values ​​to radiance; Geometric correction: Corrects image distortion caused by drone attitude tilt.

[0059] (3) Pixel-point cloud joint registration

[0060] Initial spatial registration is performed based on UAV POS data, camera intrinsic and extrinsic parameters, and LiDAR extrinsic parameters. Fine registration is then performed by combining ground control points, obvious ground feature control points, or feature matching methods. The LiDAR raster data is resampled to the multispectral image coordinate system to achieve spatial registration that meets the plant location accuracy requirements for the same ground feature pixel coordinates.

[0061] Step 3: Pixel-level multi-source data fusion to generate a multi-channel fused feature stack.

[0062] Perform channel splicing and fusion on the registered data: Fusion channels: 4 spectral reflectance channels (green light, red light, red edge, and near-infrared) + LiDAR plant height channel; Blending rule: Pixels are superimposed one-to-one without changing the original resolution and geographic coordinates; Output a multi-channel fused feature stack as the basic data for feature extraction.

[0063] Step 4: Multi-scale and multi-dimensional feature extraction to construct the original feature set.

[0064] Simultaneous extraction of four main categories of features: spectral features, LiDAR 3D structural features, multi-scale GLCM texture features, and temporal difference features. 1. Spectral characteristics (single-period + time-series difference) Three vegetation indices were calculated based on reflectance, and the time-series differences ΔNDVI, ΔNDRE, and ΔOSAVI between T2 and T1 were also calculated. ; ; ; 2. Three-dimensional structural features of LiDAR Extract the following within the pixel window: average plant height H, height variance, and point cloud density; calculate the difference ΔH between the two periods.

[0065] 3. Texture feature verification output

[0066] The sliding window is set with three layers: 3×3, 5×5, and 7×7. The second moment of the angle, contrast, entropy, uniformity, and correlation are extracted for each layer. The grayscale of NDVI, OSAVI, and LiDAR height grids are quantified respectively. GLCM texture features are calculated in the 0°, 45°, 90°, and 135° directions using 3×3, 5×5, and 7×7 windows. Whether the texture features at each scale are included in the model is determined by the importance of the training set features or the results of cross-validation.

[0067] 4. Concatenation of original multidimensional feature sets

[0068] The spectral, 3D structure, texture, and temporal difference features are concatenated along the pixel dimension to form a high-dimensional original feature dataset.

[0069] Step 5: Mutual information + VIF dual screening to construct a low-redundancy optimized feature set

[0070] Initial screening using mutual information: Calculate the mutual information between each feature and the "soybean seedling / background" label. The median of all mutual information is used as the threshold. Features with mutual information greater than the threshold are retained, while weakly correlated redundant features are removed. ; VIF Collinearity Screening: Calculate the variance inflation factor of the remaining features and iteratively remove strongly collinear features with VIF > 10; Output a low-redundancy, low-collinearity optimized feature set and input it into a classification model. The classification model can employ supervised learning models such as random forests, gradient boosting trees, and support vector machines. A random forest model with class weights can be used to identify candidate regions for seedlings, and cross-validation can be used to determine the number of trees, maximum depth, minimum number of leaf node samples, and class weight parameters.

[0071] Step 6: Adaptive sowing row fitting + dynamic plant spacing threshold identification of missing seedling positions

[0072] The whole-field pixel input model outputs soybean seedling candidate regions or seedling probability maps; after threshold segmentation and connected component analysis, soybean seedling objects and their centroid coordinates are extracted. Connectivity analysis was used to extract the centroid geographic coordinates of each seedling. DBSCAN spatial clustering automatically divides each seeding row into independent rows. Single-row adaptive dynamic threshold: The threshold for determining missing seedlings is set to 1.5. ; Traverse the centroid of the seedling row by row, with an adjacent spacing > 1.5L. i Then, pre-mark the locations of missing seedlings; The theoretical plant position is interpolated at equal intervals along the direction of the sowing row. When the distance between the centroids of two consecutive actual seedlings is greater than the preset plant spacing threshold, the number of missing plants is calculated based on the ratio of this distance to the standard plant spacing, and the final coordinates of the missing plant position are generated by interpolating the centroids of two actual seedlings according to the standard plant spacing. Record the geographical coordinates of the plant location and its row number.

[0073] Step 7: Result Output and Visualization Storage

[0074] Statistical indicators: total number of missing seedlings in the field, number of missing seedlings per row, latitude and longitude of the location of the missing seedlings; Outputs: Excel table of missing seedlings, GIS visualization of missing seedling distribution; All raw data, features, models, and results are stored in the system's data storage module, and can be queried, exported, and edited via the human-computer interaction interface.

[0075] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0076] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extracting early missing seedling locations in soybean based on multi-source remote sensing and sowing row topological constraints, characterized in that, include: A drone equipped with a LiDAR and a multispectral camera was used to conduct aerial surveys of the target soybean field, simultaneously collecting LiDAR point cloud data and multispectral raw image data of soybean seedlings. The LiDAR point cloud data and multispectral raw image data were preprocessed separately, and then the two types of preprocessed data were jointly registered as pixels and point clouds. The registered data is fused at the pixel level to generate a multi-channel fused feature stack; Based on a multi-channel fusion feature stack, spectral features, LiDAR 3D structural features, and texture features based on the gray-level co-occurrence matrix (GLCM) are extracted and integrated to generate the original multi-dimensional feature set. Calculate the mutual information value between each feature in the original multidimensional feature set and the category label of the training sample. The category label of the training sample includes at least the soybean seedling label and the background label. Select effective features with mutual information values ​​greater than a preset threshold to obtain an optimized feature set. Based on the optimized feature set, combined with the preset model, soybean seedlings are identified to obtain candidate regions for soybean seedlings and extract the actual seedling centroid point set. The sowing rows are fitted based on the actual seedling centroid set, and the starting and ending points of each sowing row segment are determined according to the intersection of the sowing row and the boundary of the target field, the operation trajectory of the seeder, the manually set row endpoints, or the extrapolation results of the first and last seedlings in the sowing row. A theoretical plant position sequence is generated along the sowing row direction according to the preset plant spacing or the locally adaptive plant spacing. The actual seedling centroids are matched with the theoretical plant positions according to the preset matching distance threshold. The theoretical plant positions that do not match the actual seedling centroids are determined as missing seedling positions.

2. The method for extracting early missing soybean plant positions based on multi-source remote sensing and sowing row topological constraints according to claim 1, characterized in that, The preprocessing of LiDAR point cloud data and multispectral raw image data includes: The original point cloud was sequentially denoised, ground point filtered, and point cloud normalized to remove flying points, noise points, and interference from farmland topographic undulations, resulting in a normalized LiDAR point cloud. The normalized point cloud was then converted into raster data to generate digital elevation raster and plant height raster. The original multispectral image is sequentially subjected to radiometric correction and geometric correction. The image DN value is converted into ground reflectance to eliminate errors caused by atmospheric, illumination and attitude deviations, resulting in a corrected multispectral image.

3. The method for extracting early missing soybean plant positions based on multi-source remote sensing and sowing row topological constraints according to claim 2, characterized in that, The next step involves performing pixel-point cloud joint registration on the two types of preprocessed data, including: Initial spatial registration is performed based on UAV POS data, camera intrinsic and extrinsic parameters, and LiDAR extrinsic parameters. Fine registration is then performed by combining ground control points, obvious ground feature control points, or feature matching methods. The LiDAR raster data is then resampled to the multispectral image coordinate system.

4. The method for extracting early missing soybean plant positions based on multi-source remote sensing and sowing row topological constraints according to claim 1, characterized in that, The step of performing pixel-level fusion on the registered data to generate a multi-channel fused feature stack includes: The reflectance data of each band of the registered multispectral image and the LiDAR plant height raster data are channel-by-channel stitching and fusion. Each pixel contains both multispectral reflectance information and LiDAR height information, generating a multi-channel fusion feature stack. The fusion rule is that the pixels are superimposed one-to-one without changing the spatial resolution and geometric position of the original image.

5. The method for extracting early missing soybean plant positions based on multi-source remote sensing and sowing row topological constraints according to claim 1, characterized in that, The multi-channel fusion feature stack extracts spectral features, LiDAR 3D structural features, and texture features based on the gray-level co-occurrence matrix (GLCM), and integrates them to generate an original multi-dimensional feature set, including: Based on the multispectral band reflectance of the multichannel fusion feature stack, the Normalized Difference Vegetation Index (NDVI), Red Edge Vegetation Index (NDRE), and Soil-Regulated Vegetation Index (OSAVI) are calculated as spectral features. The average plant height, height variance, and point cloud density of corresponding pixels in the fused image are extracted as LiDAR 3D structural features. Using the Gray-Level Co-occurrence Matrix (GLCM), a sliding window is selected to perform gray-level quantization on NDVI raster, OSAVI raster, and LiDAR normalized height raster, and GLCM texture features are calculated. The spectral features, LiDAR 3D structural features, and texture features are concatenated along the pixel dimension to construct the original multidimensional feature set.

6. The method for extracting early missing soybean plant positions based on multi-source remote sensing and sowing row topological constraints according to claim 5, characterized in that, The Normalized Difference Vegetation Index (NDVI), Red-edged Vegetation Index (NDRE), and Soil-Regulated Vegetation Index (OSAVI) were calculated as spectral features, including: ； ； ； in, For near-infrared reflectivity, For red light band reflectivity, This refers to the reflectivity of the red-edge band.

7. The method for extracting early missing soybean plant positions based on multi-source remote sensing and sowing row topological constraints according to claim 1, characterized in that, The calculation of the mutual information values ​​between each feature in the original multidimensional feature set and the class labels of the training samples includes: ； in, The mutual information value between feature variable X and training sample class labels Y, where Y includes at least soybean seedling labels and background labels. The joint probability distribution of features and labels. For the characteristic marginal probability distribution, The label edge probability distribution is defined; the mutual information threshold is set to the median of all mutual information values, and features with mutual information values ​​greater than the threshold are retained to obtain the optimized feature set.

8. The method for extracting early missing soybean plant positions based on multi-source remote sensing and sowing row topological constraints according to claim 1, characterized in that, The preset models include: random forest, gradient boosting tree, and support vector machine.

9. The method for extracting early missing soybean plant positions based on multi-source remote sensing and sowing row topological constraints according to claim 1, characterized in that, The soybean seedling identification based on the optimized feature set and the preset model also includes: For sections where the distance between the centroids of adjacent seedlings is greater than the preset plant spacing threshold, the number of missing seedlings is calculated based on the ratio of this distance to the standard plant spacing or the locally adaptive plant spacing, and the coordinates of the missing seedlings are corrected.

10. A system for extracting early missing soybean plant locations based on multi-source remote sensing and sowing row topological constraints, characterized in that, include: The UAV aerial survey module is used to conduct aerial surveys of target soybean fields using UAVs equipped with LiDAR and multispectral cameras, and simultaneously collect LiDAR point cloud data and multispectral raw image data of soybean seedlings. The multi-source data preprocessing module is used to preprocess LiDAR point cloud data and multispectral raw image data respectively, and then perform pixel-point cloud joint registration on the two types of preprocessed data. The data fusion module is used to perform pixel-level fusion on the registered data to generate a multi-channel fusion feature stack. The multi-dimensional feature extraction module is used to extract spectral features, LiDAR three-dimensional structural features, and texture features based on the gray-level co-occurrence matrix (GLCM) based on a multi-channel fusion feature stack, and integrate them to generate the original multi-dimensional feature set. The feature optimization module is used to calculate the mutual information value between each feature in the original multidimensional feature set and the category label of the training sample. The category label of the training sample includes at least a soybean seedling label and a background label. Valid features with mutual information values ​​greater than a preset threshold are selected to obtain an optimized feature set. The missing seedling location extraction and output module is used to identify soybean seedlings based on an optimized feature set and a preset model, obtain candidate regions for soybean seedlings and extract the actual seedling centroid point set; The sowing rows are fitted based on the actual seedling centroid set, and the starting and ending points of each sowing row segment are determined according to the intersection of the sowing row and the boundary of the target field, the operation trajectory of the seeder, the manually set row endpoints, or the extrapolation results of the first and last seedlings in the sowing row. A theoretical plant position sequence is generated along the sowing row direction according to the preset plant spacing or the locally adaptive plant spacing. The actual seedling centroids are matched with the theoretical plant positions according to the preset matching distance threshold. The theoretical plant positions that do not match the actual seedling centroids are determined as missing seedling positions.

Images