Millimeter-level crop image deep learning analysis method and system

Abstract

The invention discloses a millimeter-level crop image deep learning analysis method and system, and belongs to the technical field of crop image processing, and the method specifically comprises the steps: collecting crop image data, a digital surface model and GNSS positioning data of a target farmland through employing an unmanned plane carrying a millimeter-level resolution camera, generating a standardized orthographic image after geometric and radiation correction; vegetation health indexes and texture features are calculated based on the standardized orthographic images, and pest candidate areas are marked after screening; performing morphological opening or closing operation on the pest candidate region to generate a preliminary pest region vector boundary, and determining a final pest boundary in combination with a pest region output by a pre-training deep learning model; in the final insect pest boundary, the insect pest grades are divided into light, medium and heavy grades according to the area, density and damage degree indexes, and a statistical report is output; according to the method, high-precision positioning and grading of insect pests are realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of crop image processing, and specifically provides a method and system for deep learning analysis of millimeter-level crop images. Background Art

[0002] In modern agricultural production, field crops such as rice, wheat, and corn are vulnerable to pests such as rice planthoppers, wheat aphids, and corn borers, resulting in reduced yield and quality. Traditional pest detection relies primarily on manual field inspections or conventional optical image analysis, making it difficult to accurately identify millimeter-level pest characteristics such as eggs, larval morphology, and leaf damage texture in real time. On the one hand, single-spectrum or low-resolution imagery has limited accuracy in processing complex pest symptoms; on the other hand, existing methods lack the comprehensive utilization of historical pest data, sensor characteristics, and crop growth stages, resulting in high rates of missed detections and false positives. Therefore, it is necessary to leverage high-resolution imagery, multi-source data fusion, and deep learning algorithms to improve the accuracy and robustness of pest localization. Summary of the Invention

[0003] In response to the shortcomings of the existing technology, the present invention proposes a millimeter-level crop image deep learning analysis method and system. The method uses an unmanned aerial vehicle equipped with a millimeter-level resolution camera to collect crop image data, digital surface models and GNSS positioning data of the target farmland, and generates a standardized orthophoto image after geometric and radiometric correction. The vegetation health index and texture features are calculated based on the standardized orthophoto image, and candidate pest areas are marked after screening. Morphological opening or closing operations are performed on the candidate pest areas to generate preliminary pest area vector boundaries, and the final pest boundaries are determined based on the pest areas output by the pre-trained deep learning model. Within the final pest boundary, the pest level is divided into light, medium and heavy levels based on the comprehensive area, density and damage degree index, and a statistical report is output. This method achieves high-precision positioning and classification of pests.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] A deep learning analysis method for millimeter-level crop images, comprising:

[0006] S1: Use a drone to acquire data from a target farmland, synchronously acquire crop image data, a digital surface model, and GNSS positioning data, and obtain a standardized orthophoto image after geometric and radiometric correction of the acquired crop image data; the drone is equipped with a millimeter-level resolution camera;

[0007] S2: Based on the standardized orthophoto image, the vegetation health index and texture characteristics are calculated, and pixels with abnormally low vegetation health index or sudden changes in texture characteristics are screened out and marked as candidate areas for pests;

[0008] S3: Perform opening and closing operations on the candidate pest area to generate a preliminary vector boundary of the pest area;

[0009] S4: Load the pre-trained deep learning model, perform a logical AND operation on the pest area output by the deep learning model and the generated preliminary pest area vector boundary to generate the final pest boundary.

[0010] Specifically, the process of acquiring the standardized orthophoto image in S1 includes:

[0011] S1.1: Acquire crop image data and digital surface model data;

[0012] S1.2: Use the camera calibration tool to obtain radial distortion parameters and tangential distortion parameters , the original coordinates of the pixels of the crop image data Perform distortion correction to obtain N distortion-corrected crop images; the pixel coordinates in the distortion-corrected crop images are ;

[0013] S1.3: Using the structure-from-motion algorithm, N distortion-corrected crop images are aligned to a unified coordinate system through feature point matching and bundle adjustment to generate a stitched orthoimage.

[0014] S1.4: Before shooting, obtain the camera response function and convert the pixel DN values ​​of the stitched orthoimage into actual radiance values ​​L to obtain a radiance image.

[0015] S1.5: Based on the radiance value L, combined with the distance between the sun and the earth d, the solar irradiance at the top of the atmosphere , solar zenith angle , and obtain the apparent reflectance of the radiance image ;

[0016] S1.6: Use the FLAASH model to convert the apparent reflectance of the radiance image to the true reflectance of the surface , after normalization, the standardized orthophoto image is obtained, where Indicates atmospheric transmittance.

[0017] Specifically, the specific steps of S1.3 include:

[0018] S1.3.1: Obtain N distortion-corrected crop images, and use the Gaussian difference function to detect local extreme points in each distortion-corrected crop image, and use them as feature points;

[0019] S1.3.2: Generate a 128-dimensional vector for each feature point by calculating the gradient direction histogram in the neighborhood of the feature point, and obtain the feature descriptor corresponding to the feature point;

[0020] S1.3.3: Compute the nearest neighbor distances between feature descriptors of crop images with different distortion corrections and next nearest neighbor distance , combined with the preset distance threshold , will satisfy The feature points corresponding to the feature descriptors are taken as feature matching point pairs;

[0021] S1.3.4: Use the random sampling consensus algorithm to calculate the basic matrix F from the feature matching point pairs, combine it with the camera intrinsic parameter matrix, and obtain the essential matrix. Then, after the singular value decomposition of the essential matrix, the relative rotation matrix and translation vector are obtained.

[0022] Specifically, the specific steps of S1.3 also include:

[0023] S1.3.5: Based on the relative rotation matrix and translation vector, combined with the known camera extrinsic parameters of each distortion-corrected crop image, construct a function to minimize the reprojection error. Obtain the initial 3D point cloud by solving the function to minimize the reprojection error.

[0024] S1.3.6: Based on the initial 3D point cloud, the camera intrinsic parameter matrix, and the camera extrinsic parameter parameters, construct a total reprojection error function, and obtain the 3D point cloud and camera pose by minimizing the total reprojection error function.

[0025] S1.3.7: Use the chain rule to compute the partial derivatives of the total reprojection error function with respect to the 3D point cloud, the camera intrinsic matrix, and the camera extrinsic parameters, constructing a sparse linear system to obtain the optimized camera pose.

[0026] S1.3.8: Based on the optimized camera pose, apply a semi-global matching algorithm to all distortion-corrected crop images and generate a dense point cloud by calculating the similarity between pixels.

[0027] S1.3.9: Filter and interpolate the dense point cloud to generate a digital surface model in the form of a regular grid and obtain elevation data for each grid point;

[0028] S1.3.10: Reproject each distortion-corrected crop image onto the digital surface model and fuse the overlapping areas using a weighted average algorithm to generate a seamless orthoimage by setting a distance-based weight function.

[0029] Specifically, the specific steps of S2 include:

[0030] S2.1: Obtain a standardized orthophoto image, extract the red light band and the near-infrared band from the standardized orthophoto image, and calculate the Normalized Difference Vegetation Health Index (NDVI) based on the red light band and the near-infrared band;

[0031] If the Normalized Difference Vegetation Health Index (NDVI) is greater than or equal to the preset NDVI health threshold, the vegetation mask is set to 1;

[0032] If the normalized difference vegetation health index NDVI is less than the preset NDVI health threshold, the vegetation mask is set to 0;

[0033] S2.2: Calculate the gray-level co-occurrence matrix of the standardized orthoimage, count the occurrence frequencies of pixel pairs at preset directions and distances, and calculate texture feature parameters based on the occurrence frequencies; the texture feature parameters refer to contrast;

[0034] If the contrast change amplitude is greater than the preset amplitude fluctuation threshold, it is marked as a candidate pixel for pests and the mutation pixel mask is set to 1;

[0035] If the contrast change amplitude is less than or equal to the preset amplitude fluctuation threshold, it is marked as a non-pest candidate pixel and the mutation pixel mask is set to 0;

[0036] S2.3: Calculate the product of the vegetation mask and the mutation pixel mask to obtain the pest candidate area mask;

[0037] S2.4: Output the pest candidate area mask as an image file to obtain the pest candidate area.

[0038] Specifically, the specific steps of S3 include:

[0039] S3.1: Obtain a pest candidate region mask; the pest candidate region mask is a binary image; pest candidate pixels in the binary image are marked as 1, and non-candidate pixels are marked as 0;

[0040] S3.2: Set diamond-shaped structural elements based on the minimum expected size of the infestation patch;

[0041] S3.3: Based on the diamond structuring element, perform an opening operation and a closing operation on the binary image to obtain a binary image after the opening and closing operations; the opening operation is first erosion followed by dilation; the closing operation is first dilation followed by erosion;

[0042] S3.4: Extract the contours of all connected regions in the binary image after the opening and closing operations;

[0043] S3.5: Combined with the GNSS positioning data, the pixel coordinate sequence of each contour is converted into a geographic coordinate sequence to construct a closed vector polygon, i.e., the vector boundary of the preliminary pest area; each polygon in the closed vector polygon corresponds to a preliminary pest area.

[0044] Specifically, the specific steps of S4 include:

[0045] S4.1: Load the pre-trained deep learning model, input the standardized orthophoto image into the pre-trained deep learning model, and output the predicted pest binary mask;

[0046] S4.2: Convert the preliminary pest area vector boundary generated in S3.5 into a binary mask with the same resolution as the standardized orthophoto through rasterization operation to obtain a preliminary vector boundary raster mask;

[0047] S4.3: Perform a logical AND operation on the predicted insect pest binary mask and the preliminary vector boundary raster mask to obtain the final insect pest area mask;

[0048] S4.4: Perform connected region analysis on the final pest area mask, extract the contours and convert them into vector polygons, which are the final pest boundaries.

[0049] A millimeter-level crop image deep learning analysis system, comprising: a data processing module, a candidate region generation module, a boundary generation module, a boundary fusion module, and a pest classification module;

[0050] The data processing module is used to collect data on the target farmland and perform geometric and radiation correction to generate a standardized orthophoto image;

[0051] The candidate area generation module is used to screen candidate pest areas based on vegetation health index and texture features;

[0052] The boundary generation module is used to optimize the candidate area through mathematical morphological operations and convert it into a preliminary pest area vector boundary;

[0053] The boundary fusion module is used to determine the pest area using the pre-trained model and fuse it with the preliminary vector boundary to generate the final pest boundary.

[0054] The boundary generation module includes: a morphological operation unit and a vector conversion unit;

[0055] The morphological operation unit is used to perform opening and closing operations on the binary candidate region using a diamond structure element;

[0056] The vector conversion unit is used to extract the connected area contour of the image after the opening and closing operations, convert the pixel coordinates into geographic coordinates in combination with the GNSS positioning data, and generate the vector polygon boundary of the preliminary insect infestation area.

[0057] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps of a deep learning analysis method for millimeter-level crop images.

[0058] Compared with the prior art, the present invention has the following beneficial effects:

[0059] 1. The present invention proposes a millimeter-level crop image deep learning analysis system, and optimizes and improves the architecture, operation steps and processes. The system has the advantages of simple process, low investment and operation costs, and low production work costs.

[0060] 2. This paper proposes a deep learning analysis method for millimeter-level crop images. In terms of data processing and analysis accuracy, high-definition crop images are acquired by using a millimeter-level resolution camera mounted on an unmanned aerial vehicle (UAV). Combined with digital surface model data, these images undergo geometric and radiometric corrections to ensure the spatial accuracy and spectral authenticity of the images. Based on standardized orthophotos, the method integrates vegetation health index and texture features to screen candidate pest areas. Morphological operations and deep learning model inference are then used to generate precise pest boundaries. This method effectively combines the advantages of traditional image processing and deep learning to improve the accuracy and robustness of pest management and reduce the false detection and missed detection problems caused by single features or algorithms.

[0061] 3. The present invention proposes a deep learning analysis method for millimeter-level crop images. At the level of pest quantification and decision support, based on the final pest boundary and positioning data, it can accurately calculate the pest density, damage index and pest area, and classify the pests into three levels: light, medium and heavy according to multi-dimensional indicators, and output a graded statistical report. This process realizes the automated and quantitative analysis from image data to pest information, facilitates the timely formulation of targeted prevention and control strategies, optimizes resource allocation, and improves the level of intelligent management of agricultural production. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A schematic diagram of a deep learning analysis method for millimeter-level crop images according to the present invention;

[0063] Figure 2 This is a flow chart showing the principle of a deep learning analysis method for millimeter-level crop images according to the present invention;

[0064] Figure 3 This is a flowchart of the standardized orthophoto acquisition method for the millimeter-level crop image deep learning analysis method of the present invention;

[0065] Figure 4 This is an architecture diagram of the millimeter-level crop image deep learning analysis system of the present invention. DETAILED DESCRIPTION

[0066] Example 1

[0067] See also Figure 1-Figure 3 , an embodiment of the present invention provides: a millimeter-level crop image deep learning analysis method, comprising the following steps:

[0068] S1: Use a drone to acquire data from a target farmland, synchronously acquire crop image data, a digital surface model, and GNSS positioning data, and obtain a standardized orthophoto image after geometric and radiometric correction of the acquired crop image data; the drone is equipped with a millimeter-level resolution camera;

[0069] Furthermore, in addition to being equipped with a millimeter-level resolution camera, the drone is also equipped with a digital surface model acquisition module and a GNSS positioning module. The digital surface model acquisition module integrates a LiDAR sensor, and the GNSS positioning module is equipped with a real-time dynamic receiver to achieve centimeter-level positioning accuracy.

[0070] It should be noted that when using drones for data collection, route planning and flight parameter settings are required first. Route planning is to use Pix4Dcapture software to design a grid route to ensure full coverage of farmland, including an overlap rate of ≥80% for heading overlap and ≥70% for lateral overlap; flight parameter settings include setting the flight altitude and shooting mode. The flight altitude is calculated based on the ground resolution requirements, mainly by multiplying the camera pixel size by the ground resolution and then dividing it by the lens focal length. The shooting mode uses equidistant trigger shooting. In the present invention, one picture is taken every 1 second to ensure the consistency of the image time series.

[0071] S2: Based on the standardized orthophoto image, the vegetation health index and texture characteristics are calculated, and pixels with abnormally low vegetation health index or sudden changes in texture characteristics are screened out and marked as candidate areas for pests;

[0072] S3: Perform opening and closing operations on the candidate pest area to generate a preliminary vector boundary of the pest area;

[0073] S4: Load the pre-trained deep learning model, perform a logical AND operation on the pest area output by the deep learning model and the generated preliminary pest area vector boundary to generate the final pest boundary;

[0074] Furthermore, a millimeter-level crop image deep learning analysis method also includes:

[0075] S5: Calculate pest density, damage index, and infested area based on the final pest boundary and GNSS positioning data;

[0076] Among them, GNSS positioning data refers to Global Navigation Satellite System (GNSS) positioning data.

[0077] Furthermore, the specific steps of S5 include:

[0078] (1) Obtain the final pest boundary and GNSS positioning data;

[0079] (2) Use the deep learning model to detect pest targets in the image area within the final pest boundary, count the number of individual pests, determine the geographical range of the pest area in combination with GNSS positioning data, and calculate the number of pests per unit area through the conversion relationship between the number of pixels and the actual area;

[0080] (3) Obtain the normalized vegetation health index (NDVI) and contrast change amplitude, and obtain the damage degree index by weighted fusion of the normalized vegetation health index (NDVI) and contrast change amplitude indicators;

[0081] (4) Calculate the area by combining the digital surface model and GNSS positioning data, including:

[0082] Plane area: The projected area is calculated directly through the plane coordinates of the vector boundary. This is achieved by using the area calculation tool in GIS.

[0083] Terrain undulating areas: Based on the elevation data of the digital surface model, the vector boundary is converted into a three-dimensional surface, and the surface area is calculated by the triangulation method. The triangulation method is the existing technology in this field and is not the creative solution of this application, so it will not be elaborated here.

[0084] S6: Classify the pest area into three levels: light, medium, and severe based on the pest area, pest density, and crop damage index, and output a graded statistical report. The threshold setting method is used to classify the pest area.

[0085] If at least two of the infestation area, insect pest density, and crop damage index are mild, and no index reaches moderate or severe, the infestation area is judged to be at a mild level;

[0086] If at least two of the infested area, insect pest density, and crop damage index are moderate, or one is severe and one is mild, the infested area is judged to be at a moderate level;

[0087] If at least two of the infested area, pest density and crop damage index are severe, or three are moderate, the infested area is judged to be at a severe level.

[0088] The process of obtaining the standardized orthophoto image in S1 includes:

[0089] S1.1: Acquire crop image data and digital surface model data;

[0090] S1.2: Use the camera calibration tool to obtain radial distortion parameters and tangential distortion parameters , using the formula The original coordinates of the pixels of the crop image data Perform distortion correction to obtain N distortion-corrected crop images, where: represents the pixel coordinates in the crop image after distortion correction, r represents the radial distance from the original coordinate point to the image center, and satisfies ;

[0091] In the present invention, the camera calibration tool adopts MATLAB Camera Calibrator. Moreover, it should be emphasized that the crop image after distortion correction is a single image.

[0092] S1.3: Using the structure-from-motion algorithm, N distortion-corrected crop images are aligned to a unified coordinate system through feature point matching and bundle adjustment to generate a stitched orthoimage.

[0093] Among them, feature point matching is used to find corresponding points between different images, and bundle adjustment optimizes camera parameters and three-dimensional point coordinates to achieve accurate stitching of multiple images.

[0094] S1.4: By formula The pixel DN value of the stitched orthoimage is converted into the actual radiation brightness value to obtain the radiation brightness image, where L represents the radiation brightness value and B represents the black level offset. and Respectively represent the minimum and maximum values ​​of the radiant brightness corresponding to the dynamic range of the camera, and They represent the minimum and maximum pixel DN values ​​of the stitched orthoimages respectively;

[0095] The DN value is a quantized value for each pixel in a digital image. It represents the sensor's sampling of the radiated or reflected energy of the target object, reflecting the relative radiation intensity of the area corresponding to the pixel. It is a dimensionless digital signal value. For example, in an 8-bit grayscale image, the DN value ranges from 0 to 255, with 0 representing the darkest value and 255 representing the brightest. In multispectral or hyperspectral images, each pixel in each band has a separate DN value.

[0096] It should be noted that the pixel DN value of the stitched orthoimage is derived from the original record during the data acquisition phase. The specific process includes:

[0097] (1) Data collection of the target farmland is performed using a drone equipped with a millimeter-level resolution camera. During the shooting process, the camera sensor converts the received light signal into an electrical signal, quantizes it into a digital signal, and stores it in the form of a matrix. Each value in the matrix is ​​the DN value of the corresponding pixel in the original crop image, thereby obtaining crop image data containing the DN value;

[0098] (2) When the DN value is still retained after lens distortion correction, it can be seen from S1.2 that the radial distortion parameters and tangential distortion parameters are obtained using the camera calibration tool, and distortion correction is performed on each crop image. In this process, only the geometric position of the crop image is adjusted, and the DN value of its pixel is not changed, so that the crop image that still contains the DN value after distortion correction is obtained;

[0099] (3) Finally, when the DN values ​​are integrated through image registration and stitching, the structure-from-motion algorithm is used to align the N crop images that have been corrected for distortion into a unified coordinate system through feature point matching and bundle adjustment to generate a stitched orthoimage. This process integrates the pixels and corresponding DN values ​​of each crop image into the same image to form a stitched orthoimage containing complete DN value information. Therefore, when calculating the radiance value, the pixel DN value can be directly extracted.

[0100] S1.5: Calculate the apparent reflectance of the radiance image based on the radiance value L ,in, represents pi, d represents the distance between the earth and the sun, represents the solar irradiance at the top of the atmosphere, cos represents the cosine function, represents the solar zenith angle;

[0101] It should be explained that solar irradiance is obtained through sensors or real-time observations; the solar zenith angle represents the angle between the sun's rays and the surface normal, which can be calculated through imaging time, geographical location and solar position model to correct the influence of the sun's incidence angle; apparent reflectance is also called atmospheric apparent reflectance or headspace reflectance, which refers to the reflectance of the surface at the sensor entrance pupil when multiple scattering of the atmosphere is ignored.

[0102] S1.6: Use the FLAASH model to convert the apparent reflectance of the radiance image to the true reflectance of the surface , after normalization processing, the standardized orthophoto image is obtained, where It represents the atmospheric transmittance, and the FLAASH model is the existing technical content in this field, which is not the creative solution of this application and will not be described here.

[0103] Among them, FLAASH (Fast Line-of-sight Atmospheric Analysis of SpectralHypercubes) is an atmospheric correction model based on radiation transfer theory. It is used to eliminate the effects of atmospheric scattering and absorption on satellite or aerial images and convert apparent reflectance into true surface reflectance. Its core principle is to solve the radiation transfer equation and consider the effects of atmospheric molecules, aerosols, water vapor and other factors on radiation.

[0104] The specific steps of S1.3 include:

[0105] S1.3.1: Obtain N distortion-corrected crop images, and use the Gaussian difference function to detect local extreme points in each distortion-corrected crop image, and use them as feature points;

[0106] Furthermore, the specific steps of S1.3.1 include:

[0107] (1) Obtain N distortion-corrected crop images, and perform grayscale processing on each distortion-corrected crop image to obtain a crop grayscale image;

[0108] (2) The crop grayscale image is used as the bottom layer of the pyramid. Gaussian kernels with different standard deviations are used to convolve each layer of the crop grayscale image to generate b Gaussian blurred images of the layer. The Gaussian blurred image of the current layer is downsampled and used as the input of the next layer of the pyramid. This process is repeated to generate D layers of the pyramid. The use of Gaussian kernels for convolution is a prior art in this field and is not an inventive solution of the present application, so it will not be described in detail here.

[0109] (3) Subtract the two adjacent Gaussian blurred images at each level of the pyramid to obtain the Gaussian difference response image;

[0110] (4) In the Gaussian difference response image, check each pixel to see if it is a local maximum or minimum point, that is, the response value of the point is greater than or less than the 18 neighboring points in the 8 surrounding neighborhoods and the corresponding positions of the upper and lower layers, a total of 26 neighboring points;

[0111] If any point is an extreme point in the neighborhood, it is marked as a candidate feature point;

[0112] (6) Set the response threshold. If the absolute value of the response value of the point is less than the response threshold, the point is eliminated and the feature point is finally obtained.

[0113] S1.3.2: Generate a 128-dimensional vector for each feature point by calculating the gradient direction histogram in the neighborhood of the feature point, and obtain the feature descriptor corresponding to the feature point;

[0114] Furthermore, the specific process of calculating the gradient direction histogram in the neighborhood of the feature point includes:

[0115] (1) Take a square neighborhood with the feature point as the center;

[0116] (2) For each pixel in the neighborhood, the difference between the pixel and the feature point is calculated to obtain its coordinate offset relative to the feature point;

[0117] (3) After rotating the coordinate offset by an angle, the coordinates of the rotated pixel are obtained;

[0118] (4) Using the Sobel operator to calculate the gradient magnitude and direction of pixels in the neighborhood, wherein the Sobel operator is a prior art in this field and is not an inventive solution of the present application, and is not described in detail here;

[0119] (5) For the coordinates of the rotated pixels in each sub-block, a gradient direction histogram is constructed using statistical methods based on their gradient magnitude and direction.

[0120] S1.3.3: Compute the nearest neighbor distances between feature descriptors of crop images with different distortion corrections and next nearest neighbor distance , combined with the preset distance threshold , will satisfy The feature points corresponding to the feature descriptors are used as feature matching point pairs, wherein the Euclidean distance formula is used to calculate the distance. The Euclidean distance formula is a prior art in this field and is not an inventive solution of the present application, and is not described in detail here;

[0121] S1.3.4: Use the random sampling consensus algorithm to calculate the fundamental matrix F from the feature matching point pairs, combined with the camera intrinsic parameter matrix , and get the essential matrix , and then the relative rotation matrix is ​​obtained after singular value decomposition and translation vectors , where T represents transpose, and the random sampling consensus algorithm and singular value decomposition are prior art contents in this field and are not the inventive solution of this application, and are not described in detail here;

[0122] Furthermore, the relative rotation matrix and translation vectors The solution process includes:

[0123] (1) Perform singular value decomposition on the essential matrix, that is , get the orthogonal matrices U and V, and the diagonal matrix S;

[0124] (2) Obtain the rotation matrix by combining U and V or , and the translation vector or , where W represents the preset antisymmetric matrix, Represents the third column of U;

[0125] (3) Based on each set of relative rotation matrices and translation vectors, the initial 3D point cloud is calculated using triangulation. If the vertical coordinates of each 3D point in all camera coordinate systems are greater than 0, then the set of relative rotation matrices and translation vectors is the correct solution, that is, the final relative rotation matrix and translation vectors Among them, the triangulation method is the existing technical content in this field and is not the creative solution of this application, so it will not be described here.

[0126] S1.3.5: According to the relative rotation matrix and translation vectors , combined with the known camera extrinsic parameters A and C of each distortion-corrected crop image, the triangulation method is used to solve the problem of minimizing the reprojection error: , get the initial 3D point cloud P, where represents the observed position of the i-th feature point in the crop image after distortion correction, Represents the projection function, which is to transform the 3D point Mapped to the image plane, represents the coordinates of the i-th feature point in three-dimensional space, and represents the camera extrinsic parameters of the i-th feature point, represents the camera intrinsic parameter matrix of the i-th feature point, n represents the number of feature points, represents the Euclidean distance, and min represents the minimum function;

[0127] S1.3.6: Based on the initial 3D point cloud P and the camera intrinsic parameter matrix and camera extrinsic parameters , construct the total reprojection error function , the total reprojection error function is minimized by the Levenberg-Marquardt optimization algorithm to obtain the 3D point cloud and camera pose, where m represents the number of camera views; The number of feature points in the j-th camera view, represents the pixel coordinates of the i-th feature point observed in the j-th camera view, and represents the camera extrinsic parameters in the j-th camera view, and the Levenberg-Marquardt optimization algorithm is the prior art content in this field, which is not the inventive solution of the present application and is not described in detail here;

[0128] S1.3.7: Use the chain rule to compute the partial derivatives of the total reprojection error function with respect to the 3D point cloud, the camera intrinsic matrix, and the camera extrinsic parameters to construct a sparse linear system. , get the optimized camera pose, where J represents the Jacobian matrix, e represents the error vector, represents the parameter increment, wherein the chain rule is the prior art content in this field and is not the inventive solution of this application, and is not described here in detail;

[0129] S1.3.8: Based on the optimized camera pose, apply a semi-global matching algorithm to all distortion-corrected crop images and generate a dense point cloud by calculating the similarity between pixels.

[0130] Furthermore, the specific steps of S1.3.8 include:

[0131] (1) Normalize the brightness of all distortion-corrected crop images to reduce illumination differences;

[0132] (2) Using the optimized camera pose, the image pairs are corrected to the same plane through epipolar correction, so that corresponding points only need to be searched in the horizontal direction. Epipolar correction is a prior art in this field and is not an inventive solution of this application, so it will not be described in detail here;

[0133] (3) For each pixel and each possible disparity value, the matching cost is calculated using the normalized cross-correlation method. The normalized cross-correlation method is a prior art in this field and is not an inventive solution of the present application, so it will not be described in detail here.

[0134] (4) Calculate the cumulative matching cost along 8 directions;

[0135] (5) Select the disparity value with the minimum cumulative cost for each pixel;

[0136] (6) According to the disparity value with the minimum cumulative cost and the camera intrinsic parameter matrix and camera extrinsic parameter, the 3D coordinates corresponding to each pixel are calculated by triangulation, and the obtained 3D coordinates are the dense point cloud.

[0137] S1.3.9: Filter and interpolate the dense point cloud to generate a digital surface model in the form of a regular grid, and obtain elevation data for each grid point. The filtering method utilizes a statistical filtering method, and the interpolation method utilizes an inverse distance weighted interpolation method. The statistical filtering method and the inverse distance weighted interpolation method are prior art in this field and do not constitute the inventive solution of this application, and are not further elaborated here.

[0138] S1.3.10: Reproject each distortion-corrected crop image onto the digital surface model and fuse the overlapping areas using a weighted average algorithm to generate a seamless orthoimage by setting a distance-based weight function.

[0139] The specific steps of S2 include:

[0140] S2.1: Obtain a standardized orthophoto image, extract the red light band and the near-infrared band from the standardized orthophoto image, and calculate the Normalized Difference Vegetation Health Index (NDVI) based on the red light band and the near-infrared band;

[0141] Furthermore, the Normalized Difference Vegetation Health Index (NDVI) is obtained by dividing the difference between the near-infrared band reflectance and the red band reflectance by the sum of the near-infrared band reflectance and the red band reflectance.

[0142] If the Normalized Difference Vegetation Health Index (NDVI) is greater than or equal to the preset NDVI health threshold, the vegetation mask is set to 1;

[0143] If the normalized difference vegetation health index NDVI is less than the preset NDVI health threshold, the vegetation mask is set to 0;

[0144] S2.2: Calculate the gray-level co-occurrence matrix of the standardized orthoimage, count the occurrence frequencies of pixel pairs at preset directions and distances, and calculate texture feature parameters based on the occurrence frequencies; the texture feature parameters refer to contrast;

[0145] If the contrast change amplitude is greater than the preset amplitude fluctuation threshold, it is marked as a candidate pixel for pests and the mutation pixel mask is set to 1;

[0146] If the contrast change amplitude is less than or equal to the preset amplitude fluctuation threshold, it is marked as a non-pest candidate pixel and the mutation pixel mask is set to 0;

[0147] Furthermore, the contrast ratio is the cumulative sum of the product of the probability of occurrence of any two gray values ​​in the gray level co-occurrence matrix and the difference between the two gray values ​​in the gray level level.

[0148] S2.3: Calculate the product of the vegetation mask and the mutation pixel mask to obtain the pest candidate area mask;

[0149] S2.4: Output the pest candidate area mask as an image file to obtain the pest candidate area.

[0150] The specific steps of S3 include:

[0151] S3.1: Obtain a pest candidate region mask; the pest candidate region mask is a binary image; pest candidate pixels in the binary image are marked as 1, and non-candidate pixels are marked as 0;

[0152] S3.2: Set diamond-shaped structural elements based on the minimum expected size of the infestation patch;

[0153] The diamond structure element is:

[0154] Side length: 2a+1, where a is the radius of the minimum pest patch and The result of rounding down the ratio of . Note that the length of the diagonal of the rhombus corresponds to the diameter of the circle.

[0155] S3.3: Based on the diamond structuring element, perform an opening operation and a closing operation on the binary image; the opening operation is first erosion followed by dilation; the closing operation is first dilation followed by erosion;

[0156] The opening operation includes:

[0157] Remove the foreground pixels adjacent to the background in the binary image to obtain a binary image after preliminary denoising;

[0158] Dilate the binary image after preliminary denoising to restore the main shape of the foreground area, thereby obtaining a binary image after initial dilation. The dilation operation is a prior art in this field and does not constitute an inventive solution of the present application, and is not described in detail here.

[0159] The closing operation includes:

[0160] Based on the binary image after the initial expansion, the foreground area is first expanded and the adjacent pest patches are connected to obtain the binary image after the secondary expansion;

[0161] The binary image after the secondary expansion is eroded to restore the original region contour, thereby obtaining a binary image after the opening and closing operation. The erosion operation is a prior art in this field and is not an inventive solution of the present application, and is not described in detail here.

[0162] S3.4: Using an edge detection algorithm to extract the contours of all connected regions in the binary image after the opening and closing operations. The edge detection algorithm is prior art in this field and does not constitute an inventive solution of the present application, and is not described in detail here.

[0163] S3.5: Combined with the GNSS positioning data, the pixel coordinate sequence of each contour is converted into a geographic coordinate sequence to construct a closed vector polygon, i.e., the vector boundary of the preliminary pest area; each polygon in the closed vector polygon corresponds to a preliminary pest area.

[0164] The specific steps of S4 include:

[0165] S4.1: Load a pre-trained deep learning model, input the standardized orthophoto image into the pre-trained deep learning model, and output a predicted insect pest binary mask. The deep learning model is prior art in this field and does not constitute an inventive solution of the present application, and is not described in detail here.

[0166] S4.2: Convert the preliminary pest area vector boundary generated in S3.5 into a binary mask with the same resolution as the standardized orthophoto through rasterization operation to obtain a preliminary vector boundary raster mask;

[0167] Furthermore, in the present invention, vector to raster conversion is achieved through the Python library GDAL.

[0168] S4.3: Perform a logical AND operation on the predicted insect pest binary mask and the preliminary vector boundary raster mask to obtain the final insect pest area mask;

[0169] The logical AND operation only retains the areas that are simultaneously identified as pests by the model and are located within the preliminary vector boundary, eliminating isolated noise points or areas that are falsely detected by the model to improve boundary accuracy.

[0170] S4.4: Perform connected region analysis on the final pest area mask, extract the contours and convert them into vector polygons, which are the final pest boundaries.

[0171] Example 2

[0172] See also Figure 4 Another embodiment of the present invention provides a millimeter-level crop image deep learning analysis system, comprising:

[0173] Data processing module, candidate region generation module, boundary generation module, boundary fusion module;

[0174] The data processing module is used to collect data of the target farmland and perform geometric and radiation correction to generate standardized orthophoto images;

[0175] The candidate region generation module is used to screen candidate pest areas based on vegetation health index and texture features;

[0176] A boundary generation module is used to optimize the candidate area through mathematical morphological operations and convert it into a preliminary pest area vector boundary;

[0177] The boundary fusion module is used to determine the pest area using the pre-trained model and fuse it with the preliminary vector boundary to generate the final pest boundary.

[0178] The data processing module includes: image acquisition unit, distortion correction unit, image stitching unit, and radiation correction unit;

[0179] An image acquisition unit, used to acquire crop image data and digital surface model data using a drone equipped with a millimeter-level resolution camera;

[0180] Distortion correction unit, used to obtain radial / tangential distortion parameters using camera calibration tools, correct image pixel coordinates, and eliminate lens distortion errors;

[0181] An image stitching unit is used to align multiple rectified images into a unified coordinate system using a structure-from-motion algorithm, feature point matching, and bundle adjustment to generate a stitched orthoimage.

[0182] The radiation correction unit is used to convert the DN value of the crop image into actual radiation brightness and true surface reflectivity, eliminating atmospheric and light interference.

[0183] The candidate region generation module includes: vegetation health index calculation unit, texture feature analysis unit, and candidate region fusion unit;

[0184] Vegetation health index calculation unit, used to extract red and near-infrared bands, calculate the normalized vegetation health index, and generate vegetation masks;

[0185] A texture feature analysis unit is used to calculate texture features based on the gray-level co-occurrence matrix, detect texture mutation pixels, and generate mutation pixel masks;

[0186] The candidate region fusion unit is used to perform a logical multiplication operation on the vegetation mask and the mutation pixel mask to generate a pest candidate region mask and mark the pest candidate pixels.

[0187] The boundary generation module includes: a morphological operation unit and a vector conversion unit;

[0188] A morphological operation unit is used to perform opening and closing operations on the binary candidate regions using diamond structure elements, eliminate noise, and connect adjacent patches;

[0189] The vector conversion unit is used to extract the connected area contours of the image after the opening and closing operations, convert the pixel coordinates into geographic coordinates in combination with the GNSS positioning data, and generate the vector polygon boundary of the preliminary insect infestation area.

[0190] The boundary fusion module includes: model inference unit, logic operation unit, and vector boundary optimization unit;

[0191] Model inference unit, used to load pre-trained deep learning models, input standardized orthophotos, and output binary pest masks;

[0192] The logic operation unit is used to rasterize the preliminary vector boundary into a binary mask of the same resolution, perform a logical AND operation with the model prediction mask, and retain the area that meets both the model prediction and morphological screening;

[0193] The vector boundary optimization unit is used to perform connected area analysis on the fused mask, extract the contour and convert it into the final vector polygon, eliminate isolated noise points, and generate the final pest boundary.

[0194] Furthermore, a millimeter-level crop image deep learning analysis system also includes: a pest classification module;

[0195] The pest grading module quantifies the extent of pest damage and generates a grading report based on the final pest boundary and GNSS positioning data.

[0196] The pest grading module includes: density calculation unit, damage degree assessment unit, area calculation unit, and grading report generation unit;

[0197] A density calculation unit is used to count pests using a target detection algorithm and calculate the number of pests per unit area;

[0198] The damage degree assessment unit is used to combine the vegetation health index and texture feature changes to construct a damage degree index, which is divided into light, moderate and severe;

[0199] An area calculation unit is used to calculate the plane projection area or curved surface area of ​​the pest area using a digital surface model, and to calculate the area proportions of areas of different levels;

[0200] The classification report generation unit is used to integrate pest density, damage index and area data and output classification statistical reports.

[0201] Example 3

[0202] An electronic device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, it implements the steps of a deep learning analysis method for millimeter-level crop images. For details, please refer to the above method embodiment and will not be repeated here.

[0203] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also change, modify, replace and modify the above-mentioned embodiments without departing from the purpose and scope of protection of the present invention. These are all protected by the present invention.

Claims

1. A deep learning analysis method for millimeter-level crop images, characterized in that: include: S1: Use a drone to acquire data from a target farmland, synchronously acquire crop image data, a digital surface model, and GNSS positioning data, and obtain a standardized orthophoto image after geometric and radiometric correction of the acquired crop image data; the drone is equipped with a millimeter-level resolution camera; S2: Based on the standardized orthophoto image, the vegetation health index and texture characteristics are calculated, and pixels with abnormally low vegetation health index or sudden changes in texture characteristics are screened out and marked as candidate areas for pests; S3: Perform opening and closing operations on the candidate pest area to generate a preliminary vector boundary of the pest area; S4: Load the pre-trained deep learning model, perform a logical AND operation on the pest area output by the deep learning model and the generated preliminary pest area vector boundary to generate the final pest boundary.

2. The deep learning analysis method for millimeter-level crop images according to claim 1, characterized in that: The process of obtaining the standardized orthophoto image in S1 includes: S1.1: Acquire crop image data and digital surface model data; S1.2: Use the camera calibration tool to obtain radial distortion parameters and tangential distortion parameters , the original coordinates of the pixels of the crop image data Perform distortion correction to obtain N distortion-corrected crop images; the pixel coordinates in the distortion-corrected crop images are ; S1.3: Using the structure-from-motion algorithm, N distortion-corrected crop images are aligned to a unified coordinate system through feature point matching and bundle adjustment to generate a stitched orthoimage. S1.4: Before shooting, obtain the camera response function and convert the pixel DN values ​​of the stitched orthoimage into actual radiance values ​​L to obtain a radiance image. S1.5: Based on the radiance value L, combined with the distance between the sun and the earth d, the solar irradiance at the top of the atmosphere , solar zenith angle , and obtain the apparent reflectance of the radiance image ; S1.6: Use the FLAASH model to convert the apparent reflectance of the radiance image to the true reflectance of the surface , after normalization processing, the standardized orthophoto image is obtained, where Indicates atmospheric transmittance.

3. The deep learning analysis method for millimeter-level crop images according to claim 2, characterized in that: The specific steps of S1.3 include: S1.3.1: Obtain N distortion-corrected crop images, and use the Gaussian difference function to detect local extreme points in each distortion-corrected crop image, and use them as feature points; S1.3.2: Generate a 128-dimensional vector for each feature point by calculating the gradient direction histogram in the neighborhood of the feature point, and obtain the feature descriptor corresponding to the feature point; S1.3.3: Compute the nearest neighbor distances between feature descriptors of crop images with different distortion corrections and next nearest neighbor distance , combined with the preset distance threshold , will satisfy The feature points corresponding to the feature descriptors are taken as feature matching point pairs; S1.3.4: Use the random sampling consensus algorithm to calculate the basic matrix F from the feature matching point pairs, combine it with the camera intrinsic parameter matrix, and obtain the essential matrix. Then, after the singular value decomposition of the essential matrix, the relative rotation matrix and translation vector are obtained.

4. The deep learning analysis method for millimeter-level crop images according to claim 3, characterized in that: The specific steps of S1.3 also include: S1.3.5: Based on the relative rotation matrix and translation vector, combined with the known camera extrinsic parameters of each distortion-corrected crop image, construct a function to minimize the reprojection error. Obtain the initial 3D point cloud by solving the function to minimize the reprojection error. S1.3.6: Based on the initial 3D point cloud, the camera intrinsic parameter matrix, and the camera extrinsic parameter parameters, construct a total reprojection error function, and obtain the 3D point cloud and camera pose by minimizing the total reprojection error function. S1.3.7: Use the chain rule to compute the partial derivatives of the total reprojection error function with respect to the 3D point cloud, the camera intrinsic matrix, and the camera extrinsic parameters, constructing a sparse linear system to obtain the optimized camera pose. S1.3.8: Based on the optimized camera pose, apply a semi-global matching algorithm to all distortion-corrected crop images and generate a dense point cloud by calculating the similarity between pixels. S1.3.9: Filter and interpolate the dense point cloud to generate a digital surface model in the form of a regular grid and obtain elevation data for each grid point; S1.3.10: Reproject each distortion-corrected crop image onto the digital surface model and fuse the overlapping areas using a weighted average algorithm to generate a seamless orthoimage by setting a distance-based weight function.

5. The deep learning analysis method for millimeter-level crop images according to claim 4, characterized in that: The specific steps of S2 include: S2.1: Obtain a standardized orthophoto image, extract the red light band and the near-infrared band from the standardized orthophoto image, and calculate the Normalized Difference Vegetation Health Index (NDVI) based on the red light band and the near-infrared band; If the Normalized Difference Vegetation Health Index (NDVI) is greater than or equal to the preset NDVI health threshold, the vegetation mask is set to 1; If the normalized difference vegetation health index NDVI is less than the preset NDVI health threshold, the vegetation mask is set to 0; S2.2: Calculate the gray-level co-occurrence matrix of the standardized orthoimage, count the occurrence frequencies of pixel pairs at preset directions and distances, and calculate texture feature parameters based on the occurrence frequencies; the texture feature parameters refer to contrast; If the contrast change amplitude is greater than the preset amplitude fluctuation threshold, it is marked as a candidate pixel for pests and the mutation pixel mask is set to 1; If the contrast change amplitude is less than or equal to the preset amplitude fluctuation threshold, it is marked as a non-pest candidate pixel and the mutation pixel mask is set to 0; S2.3: Calculate the product of the vegetation mask and the mutation pixel mask to obtain the pest candidate area mask; S2.4: Output the pest candidate area mask as an image file to obtain the pest candidate area.

6. The deep learning analysis method for millimeter-level crop images according to claim 5, characterized in that: The specific steps of S3 include: S3.1: Obtain a pest candidate region mask; the pest candidate region mask is a binary image; pest candidate pixels in the binary image are marked as 1, and non-candidate pixels are marked as 0; S3.2: Set diamond-shaped structural elements based on the minimum expected size of the infestation patch; S3.3: Based on the diamond structuring element, perform an opening operation and a closing operation on the binary image to obtain a binary image after the opening and closing operations; the opening operation is first erosion followed by dilation; the closing operation is first dilation followed by erosion; S3.4: Extract the contours of all connected regions in the binary image after the opening and closing operations; S3.5: Combined with the GNSS positioning data, the pixel coordinate sequence of each contour is converted into a geographic coordinate sequence to construct a closed vector polygon, i.e., the vector boundary of the preliminary pest area; each polygon in the closed vector polygon corresponds to a preliminary pest area.

7. The deep learning analysis method for millimeter-level crop images according to claim 6, characterized in that: The specific steps of S4 include: S4.1: Load the pre-trained deep learning model, input the standardized orthophoto image into the pre-trained deep learning model, and output the predicted pest binary mask; S4.2: Convert the preliminary pest area vector boundary generated in S3.5 into a binary mask with the same resolution as the standardized orthophoto through rasterization operation to obtain a preliminary vector boundary raster mask; S4.3: Perform a logical AND operation on the predicted insect pest binary mask and the preliminary vector boundary raster mask to obtain the final insect pest area mask; S4.4: Perform connected region analysis on the final pest area mask, extract the contours and convert them into vector polygons, which are the final pest boundaries.

8. A millimeter-level crop image deep learning analysis system, used to implement the millimeter-level crop image deep learning analysis method according to any one of claims 1 to 7, characterized in that: include: Data processing module, candidate region generation module, boundary generation module, boundary fusion module, pest classification module; The data processing module is used to collect data on the target farmland and perform geometric and radiation correction to generate a standardized orthophoto image; The candidate area generation module is used to screen candidate pest areas based on vegetation health index and texture features; The boundary generation module is used to optimize the candidate area through mathematical morphological operations and convert it into a preliminary pest area vector boundary; The boundary fusion module is used to determine the pest area using the pre-trained model and fuse it with the preliminary vector boundary to generate the final pest boundary.

9. The millimeter-level crop image deep learning analysis system according to claim 8, characterized in that: The boundary generation module includes: a morphological operation unit and a vector conversion unit; The morphological operation unit is used to perform opening and closing operations on the binary candidate region using a diamond structure element; The vector conversion unit is used to extract the connected area contour of the image after the opening and closing operations, convert the pixel coordinates into geographic coordinates in combination with the GNSS positioning data, and generate the vector polygon boundary of the preliminary insect infestation area.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the deep learning analysis method for millimeter-level crop images described in any one of claims 1 to 7 are implemented.

Images