Millimeter-level crop image deep learning analysis method and system

Abstract

The application discloses a kind of millimeter level crop image deep learning analysis method and system, belong to crop image processing technical field, it specifically includes: using unmanned aerial vehicle with millimeter level resolution camera to collect target farmland crop image data, digital surface model and GNSS positioning data, after being corrected by geometry and radiation, generate standard orthographic image;Vegetation health index and texture feature are calculated based on standard orthographic image, mark pest candidate area after screening;Morphological open or close operation is carried out on pest candidate area, generate preliminary pest area vector boundary, and combine the pest area output by pre-training deep learning model, determine final pest boundary;In final pest boundary, the pest level is divided into light, medium, heavy three levels and statistical report is output by comprehensively area, density and damage degree index;The method realizes the high-precision positioning and grading of pest.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of crop image processing, in particular to a millimeter-level crop image deep learning analysis method and system. BACKGROUND

[0002] In modern agricultural production, field crops such as rice, wheat and corn are prone to be attacked by pests such as rice planthoppers, wheat aphids and corn borers, resulting in yield and quality decline. Traditional pest detection mainly relies on manual field patrol or ordinary optical image analysis, which is difficult to identify millimeter-level pest characteristics such as egg, larval morphology and leaf damage texture in real time and accurately. On the one hand, the processing accuracy of single spectrum or low-resolution image on complex pest symptoms is limited; on the other hand, the existing methods lack comprehensive utilization of historical pest data, sensor characteristics and crop growth stages, resulting in high false negative and false positive rates. Therefore, it is necessary to use high-resolution images, multi-source data fusion and deep learning algorithms to improve the accuracy and robustness of pest positioning. SUMMARY

[0003] In view of the deficiencies of the prior art, the application provides a millimeter-level crop image deep learning analysis method and system, which uses a unmanned aerial vehicle (UAV) equipped with a millimeter-level resolution camera to collect crop image data, digital surface model (DSM) and GNSS positioning data of a target farmland, generates a standardized orthographic image after geometric and radiometric correction; calculates vegetation health index and texture features based on the standardized orthographic image, marks pest candidate regions after screening; performs morphological opening or closing operation on the pest candidate regions to generate a preliminary pest region vector boundary, and determines the final pest boundary in combination with the pest region output by the pre-trained deep learning model; within the final pest boundary, the pest level is divided into three levels of light, medium and heavy and a statistical report is output by comprehensively considering the area, density and damage degree index; the method realizes high-precision positioning and grading of pests.

[0004] To achieve the above object, the application provides the following technical scheme:

[0005] A millimeter-level crop image deep learning analysis method, comprising:

[0006] S1: using a unmanned aerial vehicle (UAV) to acquire data of a target farmland, synchronously acquiring crop image data, digital surface model (DSM) and GNSS positioning data, and obtaining a standardized orthographic image after geometric correction and radiometric correction of the acquired crop image data; the UAV is equipped with a millimeter-level resolution camera;

[0007] S2: based on the standardized orthographic image, calculating vegetation health index and texture features, and screening pixels with abnormally low vegetation health index or mutated texture features to mark as pest candidate regions;

[0008] S3: Perform open operation and close operation on the pest candidate area to generate a preliminary pest area vector boundary;

[0009] S4: Load a pre-trained deep learning model, perform logical AND operation on the pest area output by the deep learning model and the generated preliminary pest area vector boundary to generate a final pest boundary.

[0010] Specifically, the acquisition process of the normalized orthographic image in S1 includes:

[0011] S1.1: Obtain crop image data and digital surface model data;

[0012] S1.2: Obtain radial distortion parameters and tangential distortion parameters using a camera calibration tool, and perform distortion correction on the pixel point original coordinates of the crop image data to obtain N pieces of distortion-corrected crop images; the pixel point coordinates in the distortion-corrected crop images are ;

[0013] S1.3: Use a structure from motion algorithm to align the N pieces of distortion-corrected crop images to a unified coordinate system through feature point matching and bundle adjustment to generate a spliced orthographic image;

[0014] S1.4: Obtain the response function of the camera before shooting, convert the pixel DN value of the spliced orthographic image into an actual radiance value L, and obtain a radiance image;

[0015] S1.5: Based on the radiance value L, combine the distance d between the sun and the earth, the atmospheric top layer solar irradiance , and the solar zenith angle to obtain the apparent reflectance of the radiance image;

[0016] S1.6: Convert the apparent reflectance of the radiance image into the ground true reflectance using the FLAASH model, and after normalization, obtain the normalized orthographic image, wherein represents the atmospheric transmittance.

[0017] Specifically, the specific steps of S1.3 include:

[0018] S1.3.1: Obtain N pieces of distortion-corrected crop images, detect local extreme points using a Gaussian difference function for each distortion-corrected crop image, and use the local extreme points as feature points;

[0019] S1.3.2: Calculate the gradient direction histogram in the neighborhood of the feature points to generate a 128-dimensional vector for each feature point to obtain the feature descriptor corresponding to the feature point;

[0020] S1.3.3: Calculate the nearest neighbor distance of feature descriptors between different distortion-corrected crop images and the second nearest neighbor distance , combined with a preset distance threshold , the feature points corresponding to the feature descriptors satisfying are taken as feature matching point pairs;

[0021] S1.3.4: Calculate the fundamental matrix F from the feature matching point pairs using the random sample consensus algorithm, combine the camera intrinsic matrix to obtain the essential matrix, and then decompose the essential matrix by singular value to obtain the relative rotation matrix and the translation vector.

[0022] Specifically, the specific steps of S1.3 further comprise:

[0023] S1.3.5: According to the relative rotation matrix and the translation vector, combined with the known camera extrinsic parameters of each distortion-corrected crop image, construct a minimum reprojection error function, and obtain the initial 3D point cloud by solving the minimum reprojection error function;

[0024] S1.3.6: Based on the initial 3D point cloud, the camera intrinsic matrix and the camera extrinsic parameters, construct a total reprojection error function, and obtain the 3D point cloud and the camera pose by minimizing the total reprojection error function;

[0025] S1.3.7: Calculate the partial derivative of the total reprojection error function with respect to the 3D point cloud, the camera intrinsic matrix and the camera extrinsic parameters using the chain rule, construct a sparse linear system, and obtain the optimized camera pose;

[0026] S1.3.8: Based on the optimized camera pose, use the semi-global matching algorithm on all distortion-corrected crop images to generate a dense point cloud by calculating the similarity between pixels;

[0027] S1.3.9: Filter and interpolate the dense point cloud to generate a digital surface model in the form of a regular grid, and obtain the elevation data of each grid point;

[0028] S1.3.10: Project each distortion-corrected crop image onto the digital surface model, and use a weighted average algorithm to fuse the overlapping regions to generate a seamless orthographic image by setting a distance-based weight function.

[0029] Specifically, the specific steps of S2 comprise:

[0030] S2.1: Obtain a standardized orthographic image, extract the red light band and the near-infrared band from the standardized orthographic image, and calculate the normalized vegetation health index NDVI according to the red light band and the near-infrared band;

[0031] If the normalized vegetation health index NDVI is greater than or equal to a preset NDVI health threshold, the vegetation mask is set to 1;

[0032] If the normalized vegetation health index NDVI is less than the preset NDVI health threshold, the vegetation mask is set to 0;

[0033] S2.2: Calculate the gray level co-occurrence matrix of the normalized orthographic image, count the occurrence frequency of pixel pairs in a preset direction and distance, and calculate the texture feature parameter according to the occurrence frequency; the texture feature parameter refers to contrast;

[0034] If the contrast variation amplitude is greater than a preset amplitude fluctuation threshold, it is marked as a pest candidate pixel, and the abrupt change pixel mask is set to 1;

[0035] If the contrast variation amplitude is less than or equal to the preset amplitude fluctuation threshold, it is marked as a non-pest candidate pixel, and the abrupt change pixel mask is set to 0;

[0036] S2.3: Calculate the product of the vegetation mask and the abrupt change pixel mask to obtain a pest candidate area mask;

[0037] S2.4: Output the pest candidate area mask as an image file to obtain a pest candidate area.

[0038] Specifically, the specific steps of S3 include:

[0039] S3.1: Obtain a pest candidate area mask; the pest candidate area mask is a binary image; pest candidate pixels in the binary image are marked as 1, and non-candidate pixels are marked as 0;

[0040] S3.2: According to the minimum expected size of the pest patch, a rhombus structure element is set;

[0041] S3.3: Based on the rhombus structure element, open operation and close operation are performed on the binary image to obtain an open-close operated binary image; the open operation is erosion first and then dilation; the close operation is dilation first and then erosion;

[0042] S3.4: Extract the contours of all connected regions in the open-close operated binary image;

[0043] S3.5: Combine the GNSS positioning data to convert the pixel coordinate sequence of each contour into a geographic coordinate sequence, and construct a closed vector polygon, i.e. a preliminary pest area vector boundary; each polygon in the closed vector polygon corresponds to a preliminary pest area.

[0044] Specifically, the specific steps of S4 include:

[0045] S4.1: load a pre-trained deep learning model, input the standardized orthographic image into the pre-trained deep learning model, and output a predicted pest binary mask;

[0046] S4.2: convert the preliminary pest area vector boundary generated in S3.5 into a binary mask with the same resolution as the standardized orthographic image through a rasterization operation, and obtain a preliminary vector boundary raster mask;

[0047] S4.3: perform a logical AND operation on the predicted pest binary mask and the preliminary vector boundary raster mask to obtain a final pest area mask;

[0048] S4.4: perform connected region analysis on the final pest area mask, extract the contour and convert it into a vector polygon, i.e. the final pest boundary.

[0049] A millimeter-level crop image deep learning analysis system, comprising: a data processing module, a candidate region generation module, a boundary generation module, a boundary fusion module, and a pest classification module;

[0050] The data processing module is configured to collect data of a target farmland and perform geometric and radiometric correction to generate a standardized orthographic image.

[0051] The candidate region generation module is configured to screen pest candidate regions based on vegetation health index and texture features.

[0052] The boundary generation module is configured to optimize the candidate regions through mathematical morphology operation and convert them into preliminary pest area vector boundaries.

[0053] The boundary fusion module is configured to determine pest areas using a pre-trained model and fuse them with the preliminary vector boundaries to generate final pest boundaries.

[0054] The boundary generation module comprises a morphological operation unit and a vector conversion unit.

[0055] The morphological operation unit is configured to perform opening and closing operations on the binary candidate regions using a diamond structural element.

[0056] The vector conversion unit is configured to extract the connected region contour of the image after opening and closing operations, convert the pixel coordinates into geographic coordinates in combination with GNSS positioning data, and generate a vector polygon boundary of the preliminary pest area.

[0057] An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a millimeter-level crop image deep learning analysis method when executing the computer program.

[0058] Compared with the prior art, the present application has the following advantages:

[0059] 1. The present application provides a millimeter level crop image deep learning analysis system, and the optimization and improvement of the architecture, operation steps and process are carried out, the system has the advantages of simple process, low investment and operation cost, and low production cost.

[0060] 2. The present application provides a millimeter level crop image deep learning analysis method, in the aspect of data processing and analysis accuracy, high definition crop image is obtained by unmanned aerial vehicle carrying millimeter level resolution camera, and combined with digital surface model data, geometric correction and radiation correction are carried out to ensure the spatial position accuracy and spectral information authenticity of the image; based on the standardized orthographic image, the candidate area of insect pest is screened by fusing the vegetation health index and texture feature, and then the accurate insect pest boundary is generated by morphological operation and deep learning model reasoning, which effectively combines the advantages of traditional image processing and deep learning, improves the accuracy and robustness of insect pest treatment, and reduces the false detection and missed detection problems of single feature or algorithm.

[0061] 3. The present application provides a millimeter level crop image deep learning analysis method, in the aspect of insect pest quantification and decision support, based on the final insect pest boundary and positioning data, the insect pest density, damage degree index and insect pest area can be accurately calculated, and the insect pest is divided into light, medium and heavy three levels according to multi-dimensional indexes, and the grading statistical report is output, which realizes the automatic and quantitative analysis from image data to insect pest information, and is convenient for timely formulating targeted prevention and control strategy, optimizing resource allocation and improving the intelligent management level of agricultural production. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 It is a schematic diagram of the millimeter level crop image deep learning analysis method of the present application;

[0063] Figure 2 It is a principle flow chart of the millimeter level crop image deep learning analysis method of the present application;

[0064] Figure 3 It is a standardized orthographic image acquisition flow chart of the millimeter level crop image deep learning analysis method of the present application;

[0065] Figure 4 It is a system architecture diagram of the millimeter level crop image deep learning analysis system of the present application. DETAILED DESCRIPTION

[0066] Example 1

[0067] Please refer to Figures 1-3 The present application provides an embodiment: a millimeter level crop image deep learning analysis method, comprising the following steps:

[0068] S1: using a unmanned aerial vehicle to obtain target farmland for data collection, synchronously obtaining crop image data, digital surface model and GNSS positioning data, and obtaining a standardized orthographic image after geometric correction and radiation correction of the collected crop image data; the unmanned aerial vehicle is equipped with a millimeter-level resolution camera;

[0069] Further, in addition to the millimeter-level resolution camera, the unmanned aerial vehicle is also configured with a digital surface model acquisition module and a GNSS positioning module, wherein the digital surface model acquisition module integrates a LiDAR sensor, and the GNSS positioning module is equipped with a real-time dynamic receiver, achieving centimeter-level positioning accuracy.

[0070] It should be noted that when using the unmanned aerial vehicle for data collection, flight route planning and flight parameter setting are required, wherein the flight route planning is to design a grid flight route using Pix4Dcapture software to ensure full coverage of the farmland, including an overlap rate of heading overlap ≥80% and an overlap rate of lateral overlap ≥70%; the flight parameter setting includes setting the flight height and the shooting mode, wherein the flight height is calculated according to the ground resolution requirement, mainly by multiplying the camera pixel size by the ground resolution and then dividing by the lens focal length, and the shooting mode adopts an equidistance trigger shooting, which is 1 second per shot in the present application, ensuring the time sequence consistency of the images.

[0071] S2: based on the standardized orthographic image, calculating the vegetation health index and the texture feature, and screening out pixels with abnormally low vegetation health index or mutated texture feature, and marking them as pest candidate areas;

[0072] S3: performing open operation and closed operation on the pest candidate areas to generate a preliminary pest area vector boundary;

[0073] S4: loading a pre-trained deep learning model, performing logical AND operation on the pest area output by the deep learning model and the generated preliminary pest area vector boundary to generate a final pest boundary;

[0074] Further, a millimeter-level crop image deep learning analysis method further comprises:

[0075] S5: based on the final pest boundary and the GNSS positioning data, calculating the pest density, the damage degree index and the pest area area;

[0076] Wherein, the GNSS positioning data refers to Global Navigation Satellite System (GNSS) positioning data.

[0077] Further, the specific steps of S5 include:

[0078] (1) obtaining the final pest boundary and the GNSS positioning data;

[0079] (2) Using a deep learning model to detect pest targets in the image area within the final pest boundary, counting the number of pest individuals, combining GNSS positioning data to determine the geographic range of the pest area, and through the conversion relationship between pixel number and actual area, calculating the number of pests per unit area;

[0080] (3) Obtain the normalized vegetation health index NDVI and the contrast change amplitude, and obtain the damage degree index by weighted fusion of the normalized vegetation health index NDVI and the contrast change amplitude index;

[0081] (4) Combine the digital surface model and GNSS positioning data to calculate the area, including:

[0082] Planar area: directly calculate the projected area through the planar coordinates of the vector boundary, which is realized by using the area calculation tool in GIS in the present application;

[0083] Terrain area: based on the elevation data of the digital surface model, the vector boundary is converted into a three-dimensional surface, and the surface area is calculated by the triangular meshing method, wherein the triangular meshing method is a prior art content in the art and is not the inventive scheme of the present application, and will not be described here.

[0084] S6: According to the pest area, pest density and crop damage degree index, the pest area is divided into light, medium and heavy levels, and a grading statistical report is output, wherein a threshold setting method is used to divide the pest area;

[0085] If at least two of the pest area, pest density and crop damage degree index are light, and no index reaches moderate or severe, the pest area is determined to be light level;

[0086] If at least two of the pest area, pest density and crop damage degree index are moderate, or one is severe and one is light, the pest area is determined to be medium level;

[0087] If at least two of the pest area, pest density and crop damage degree index are severe, or three are moderate, the pest area is determined to be heavy level.

[0088] The acquisition process of the normalized orthographic image in S1 includes:

[0089] S1.1: Obtain crop image data and digital surface model data;

[0090] S1.2: Obtain radial distortion parameters and tangential distortion parameters using a camera calibration tool, and use the formula to the original coordinates of the pixel points of the crop image data N pieces of crop images after distortion correction are obtained, wherein, represents the pixel point coordinates in the crop image after distortion correction, r represents the radial distance of the original coordinate point to the center of the image, and satisfies ;

[0091] In the present application, the camera calibration tool adopts MATLAB Camera Calibrator, and it is emphasized that the crop image after distortion correction is a single image.

[0092] S1.3: Aligning N pieces of crop images after distortion correction to a unified coordinate system by feature point matching and bundle adjustment through a structure from motion algorithm to generate a spliced orthographic image;

[0093] Among them, the feature point matching is used to find the corresponding points between different images, the bundle adjustment optimizes the camera parameters and three-dimensional point coordinates, and the accurate splicing of multiple images is realized.

[0094] S1.4: The pixel DN value of the spliced orthographic image is converted into the actual radiometric brightness value through the formula to obtain a radiometric brightness image, wherein L represents the radiometric brightness value, B represents the black level offset, and respectively represent the minimum value and the maximum value of the radiometric brightness corresponding to the camera dynamic range, and respectively represent the minimum value and the maximum value of the pixel DN value of the spliced orthographic image;

[0095] Among them, the DN value is a quantized value (Digital Number, DN), which is a quantized value of a pixel in a digital image, represents the sampling result of the sensor to the target ground object radiation or reflected energy, and reflects the relative radiation intensity of the pixel corresponding area, which is a dimensionless digital signal value. For example, assuming that in an 8-bit grayscale image, the DN value range is 0~255. Among them, 0 represents the darkest, and 255 represents the brightest; in a multispectral or hyperspectral image, each waveband pixel has an independent DN value.

[0096] It should be noted that the pixel DN value of the spliced orthographic image is derived from the original record in the data acquisition stage, and the specific process includes:

[0097] (1) Using a drone equipped with a millimeter resolution camera to collect data on the target farmland, during the shooting process, the camera sensor converts the received light signal into an electrical signal and quantizes it into a digital signal, which is stored in a matrix form, and each value in the matrix is the DN value of the corresponding pixel of the original crop image, thereby obtaining crop image data containing DN values;

[0098] (2) When the DN value is still retained after lens distortion correction, according to S1.2, the radial distortion parameters and tangential distortion parameters are obtained by using a camera calibration tool to correct the distortion of each crop image. In this process, only the geometric position of the crop image is adjusted, and the DN value of the pixel is not changed, so that the crop image after distortion correction still contains the DN value;

[0099] (3) Finally, when the DN value is integrated by image registration and splicing, the N crop images after distortion correction are aligned to a unified coordinate system by using a structure from motion algorithm, feature point matching and bundle adjustment to generate a spliced orthographic image. In this process, the pixels and corresponding DN values of each crop image are integrated into the same image to form a spliced orthographic image containing complete DN value information. Therefore, when calculating the radiance value, the pixel DN value can be directly extracted.

[0100] S1.5: Based on the radiance value L, the apparent reflectance of the radiance image is calculated , wherein, represents the circumference of a circle, d represents the distance from the earth to the sun, represents the solar irradiance at the top of the atmosphere, and cos represents the cosine function, represents the solar zenith angle;

[0101] It should be explained that the solar irradiance is obtained by a sensor or real-time observation; the solar zenith angle represents the angle between the sunlight and the normal of the earth's surface, which can be calculated by the imaging time, geographical position and solar position model, and is used to correct the influence of the solar incident angle; the apparent reflectance, also known as atmospheric apparent reflectance or top space reflectance, refers to the reflectance of the earth's surface at the entrance pupil of the sensor when the atmospheric multiple scattering is ignored.

[0102] S1.6: The apparent reflectance of the radiance image is converted into the true reflectance of the earth's surface by using the FLAASH model , after normalization, the normalized orthographic image is obtained, wherein, represents the atmospheric transmittance, and the FLAASH model is a prior art content in the field and is not the inventive scheme of the present application, and will not be described here.

[0103] Wherein, FLAASH (Fast Line-of-sight Atmospheric Analysis of Spectral Hypercubes) is an atmospheric correction model based on radiation transfer theory, which is used to eliminate the influence of atmospheric scattering and absorption on satellite or aerial images, and convert the apparent reflectance into the true reflectance of the earth's surface. Its core principle is to solve the radiation transfer equation, considering the influence of atmospheric molecules, aerosols, water vapor and other factors on radiation.

[0104] The specific steps of S1.3 include:

[0105] S1.3.1: Obtain N pieces of crop images after distortion correction, detect local extreme points using a Gaussian difference function for each piece of crop image after distortion correction, and take the local extreme points as feature points;

[0106] Further, the specific steps of S1.3.1 include:

[0107] (1) Obtain N pieces of crop images after distortion correction, and perform grayscale processing on each piece of crop image after distortion correction to obtain a crop grayscale image;

[0108] (2) Take the crop grayscale image as the bottom layer of a pyramid, perform convolution on each layer of the crop grayscale image using a Gaussian kernel with different standard deviations to generate b Gaussian blurred images of the layer, and perform down-sampling on the Gaussian blurred image of the current layer as the input of the next layer of the pyramid, and repeat the process to generate a D-layer pyramid, wherein the convolution using the Gaussian kernel is a prior art content in the field and is not the creative scheme of the present application, and will not be described here;

[0109] (3) Subtract two adjacent Gaussian blurred images of each layer of the pyramid to obtain a Gaussian difference response image;

[0110] (4) In the Gaussian difference response image, check whether each pixel is a local maximum or minimum point, i.e., the response value of the point is greater than or less than the 18 adjacent points of the 8-neighborhood and the corresponding positions of the upper and lower two layers, a total of 26 adjacent points;

[0111] If any point is an extreme point in the neighborhood, it is marked as a candidate feature point;

[0112] (6) Set a response threshold, if the absolute value of the response value of the point is less than the response threshold, the point is removed, and finally the feature points are obtained.

[0113] S1.3.2: Generate a 128-dimensional vector for each feature point by calculating the gradient direction histogram in the neighborhood of the feature point, to obtain the feature descriptor corresponding to the feature point;

[0114] Further, the specific process of calculating the gradient direction histogram in the neighborhood of the feature point includes:

[0115] (1) Take a square neighborhood centered on the feature point;

[0116] (2) For each pixel in the neighborhood, calculate the coordinate offset relative to the feature point by difference calculation with the pixel of the feature point;

[0117] (3) Rotate the coordinate offset to obtain the coordinates of the rotated pixel;

[0118] (4) using Sobel operator to calculate the gradient magnitude and direction of the pixels in the neighborhood, wherein the Sobel operator is the prior art in the field and is not the creative scheme of the present application, and thus is not described herein;

[0119] (5) using statistical method to construct the histogram of gradient direction according to the gradient magnitude and direction of the coordinates of the rotated pixels in each sub-block.

[0120] S1.3.3: calculating the nearest neighbor distance of the feature descriptors between the crop images after different distortion correction and the second nearest neighbor distance , combining the preset distance threshold , taking the feature points corresponding to the feature descriptors satisfying as the feature matching point pairs, wherein the Euclidean distance formula is used to calculate the distance, and the Euclidean distance formula is the prior art in the field and is not the creative scheme of the present application, and thus is not described herein;

[0121] S1.3.4: using the random sample consensus algorithm to calculate the fundamental matrix F from the feature matching point pairs, combining the camera intrinsic matrix , obtaining the essential matrix , and then performing singular value decomposition to obtain the relative rotation matrix and the translation vector , wherein T represents transposition, and the random sample consensus algorithm and the singular value decomposition are the prior art in the field and are not the creative scheme of the present application, and thus are not described herein;

[0122] Further, the solving process of the relative rotation matrix and the translation vector includes:

[0123] (1) performing singular value decomposition on the essential matrix, i.e. , to obtain the orthogonal matrices U and V, and the diagonal matrix S;

[0124] (2) obtaining the rotation matrix or and the translation vector or by combining U and V, wherein W represents the preset skew-symmetric matrix, represents the third column of U;

[0125] (3) according to each set of relative rotation matrix and translation vector, calculating the initial 3D point cloud by using the triangulation method, if the vertical coordinates of each 3D point in all camera coordinate systems are greater than 0, the set of relative rotation matrix and translation vector is the correct solution, i.e. the final relative rotation matrix and the translation vector wherein the triangulation is a prior art content in the field, not the inventive solution of the present application, and is not described herein.

[0126] S1.3.5: according to the relative rotation matrix and the translation vector , combined with the known camera extrinsic parameters A and C of each distortion-corrected crop image, the triangulation is used to solve the minimization reprojection error problem: to obtain the initial 3D point cloud P, wherein represents the observed position of the i-th feature point in the distortion-corrected crop image, represents the projection function, which is to map the 3D point to the image plane, represents the coordinates of the i-th feature point in the three-dimensional space, and represent the camera extrinsic parameters of the i-th feature point, represents the camera intrinsic matrix of the i-th feature point, and n represents the number of feature points, represents the Euclidean distance, and min represents the minimum function;

[0127] S1.3.6: based on the initial 3D point cloud P, the camera intrinsic matrix and the camera extrinsic parameters , a total reprojection error function is constructed, and the total reprojection error function is minimized by the Levenberg-Marquardt optimization algorithm to obtain the 3D point cloud and the camera pose, wherein m represents the number of camera views; the number of feature points in the j-th camera view, represents the pixel coordinates of the i-th feature point observed in the j-th camera view, and represent the camera extrinsic parameters in the j-th camera view, and the Levenberg-Marquardt optimization algorithm is a prior art content in the field, not the inventive solution of the present application, and is not described herein;

[0128] S1.3.7: the total reprojection error function is calculated using the chain rule to calculate the partial derivative of the 3D point cloud, the camera intrinsic matrix and the camera extrinsic parameters, and a sparse linear system is constructed, and the optimized camera pose is obtained, wherein J represents the Jacobian matrix, e represents the error vector, represents the parameter increment, wherein the chain rule is a prior art content in the field, not the inventive solution of the present application, and is not described herein;

[0129] S1.3.8: based on the optimized camera pose, a semi-global matching algorithm is used for all distortion-corrected crop images to generate a dense point cloud by calculating the similarity between pixels;

[0130] Further, the specific steps of S1.3.8 include:

[0131] (1) Perform brightness normalization on all the crop images after distortion correction to reduce the difference in illumination;

[0132] (2) Correct the image pair to the same plane by epipolar rectification using the optimized camera pose, so that the corresponding points only need to be searched in the horizontal direction, wherein the epipolar rectification is a prior art content in the field and is not the creative scheme of the present application, and will not be described here;

[0133] (3) For each pixel and each possible disparity value, calculate its matching cost using the normalized cross-correlation method, wherein the normalized cross-correlation method is a prior art content in the field and is not the creative scheme of the present application, and will not be described here;

[0134] (4) Calculate the cumulative matching cost along 8 directions;

[0135] (5) Select the disparity value with the minimum cumulative cost for each pixel;

[0136] (6) Calculate the 3D coordinates corresponding to each pixel by triangulation according to the disparity value with the minimum cumulative cost and the camera intrinsic matrix and camera extrinsic parameters, and obtain the 3D coordinates, i.e. the dense point cloud.

[0137] S1.3.9: Perform filtering and interpolation processing on the dense point cloud to generate a digital surface model in the form of a regular grid, and obtain the elevation data of each grid point, wherein the filtering uses a statistical filtering method and the interpolation uses an inverse distance weighted interpolation method, and the statistical filtering method and the inverse distance weighted interpolation method are prior art contents in the field and are not the creative scheme of the present application, and will not be described here;

[0138] S1.3.10: Project each crop image after distortion correction onto the digital surface model, and fuse the overlapping areas using a weighted average algorithm to generate a seamless orthographic image by setting a distance-based weight function.

[0139] The specific steps of S2 include:

[0140] S2.1: Obtain a standardized orthographic image, extract the red light band and near-infrared band from the standardized orthographic image, and calculate the normalized vegetation health index NDVI according to the red light band and the near-infrared band;

[0141] Further, the normalized vegetation health index NDVI is obtained by dividing the difference between the near-infrared band reflectance and the red light band reflectance by the sum of the near-infrared band reflectance and the red light band reflectance.

[0142] If the normalized vegetation health index NDVI is greater than or equal to a preset NDVI health threshold, the vegetation mask is set to 1;

[0143] If the normalized vegetation health index NDVI is less than the preset NDVI health threshold, the vegetation mask is set to 0;

[0144] S2.2: Calculate the gray level co-occurrence matrix of the normalized orthographic image, count the occurrence frequency of pixel pairs in a preset direction and distance, and calculate the texture feature parameter according to the occurrence frequency; the texture feature parameter refers to contrast;

[0145] If the contrast variation amplitude is greater than a preset amplitude fluctuation threshold, it is marked as a pest candidate pixel, and the abrupt change pixel mask is set to 1;

[0146] If the contrast variation amplitude is less than or equal to the preset amplitude fluctuation threshold, it is marked as a non-pest candidate pixel, and the abrupt change pixel mask is set to 0;

[0147] Further, the contrast is the cumulative sum of the product of the occurrence probability of any two gray values in the gray level co-occurrence matrix and the difference between the two gray values in the gray level.

[0148] S2.3: Calculate the product of the vegetation mask and the abrupt change pixel mask to obtain a pest candidate region mask;

[0149] S2.4: Output the pest candidate region mask as an image file to obtain a pest candidate region.

[0150] The specific steps of S3 include:

[0151] S3.1: Obtain a pest candidate region mask; the pest candidate region mask is a binary image; the pest candidate pixels in the binary image are marked as 1, and the non-candidate pixels are marked as 0;

[0152] S3.2: According to the minimum expected size of the pest patch, a diamond structure element is set;

[0153] The diamond structure element is:

[0154] The side length is 2a+1, where a is the integer part of the ratio of the minimum pest patch radius to It should be noted that the length of the diagonal of the diamond corresponds to the diameter of the circle.

[0155] S3.3: Based on the diamond structure element, open operation and close operation are performed on the binary image; the open operation is first erosion and then dilation; the close operation is first dilation and then erosion;

[0156] The open operation includes:

[0157] Remove the foreground pixels adjacent to the background in the binary image to obtain a binary image after preliminary denoising;

[0158] Perform dilation on the binary image after preliminary denoising to restore the main shape of the foreground region, to obtain a binary image after initial dilation, wherein the dilation operation is a prior art in the field and is not the inventive scheme of the present application, and will not be described here;

[0159] The closing operation includes:

[0160] Based on the binary image after initial dilation, the foreground region is first expanded to connect adjacent pest patches to obtain a binary image after secondary dilation;

[0161] Perform erosion on the binary image after secondary dilation to restore the original region profile to obtain a binary image after opening and closing operation, wherein the erosion operation is a prior art in the field and is not the inventive scheme of the present application, and will not be described here;

[0162] S3.4: Use an edge detection algorithm to extract the profile of all connected regions in the binary image after opening and closing operation, wherein the edge detection algorithm is a prior art in the field and is not the inventive scheme of the present application, and will not be described here;

[0163] S3.5: Combine GNSS positioning data to convert the pixel coordinate sequence of each profile into a geographic coordinate sequence to construct a closed vector polygon, i.e. a preliminary pest region vector boundary; each polygon in the closed vector polygon corresponds to a preliminary pest region.

[0164] The specific steps of S4 include:

[0165] S4.1: Load a pre-trained deep learning model, input the normalized orthographic image into the pre-trained deep learning model, and output a predicted pest binary mask, wherein the deep learning model is a prior art in the field and is not the inventive scheme of the present application, and will not be described here;

[0166] S4.2: Convert the preliminary pest region vector boundary generated in S3.5 into a binary mask with the same resolution as the normalized orthographic image through a rasterization operation to obtain a preliminary vector boundary raster mask;

[0167] Further, in the present application, the vector-to-raster conversion is realized through the Python library GDAL.

[0168] S4.3: Perform a logical AND operation on the predicted pest binary mask and the preliminary vector boundary raster mask to obtain a final pest region mask;

[0169] Wherein, the logical AND operation only retains the area which is determined as pest by the model and located in the preliminary vector boundary, eliminates isolated noise points or model misjudgment area, and improves the boundary accuracy.

[0170] S4.4: performing connected region analysis on the final pest area mask, extracting the contour and converting into a vector polygon, i.e. the final pest boundary.

[0171] Embodiment 2

[0172] Please refer to Figure 4 , the application provides another embodiment: a millimeter-level crop image deep learning analysis system, comprising:

[0173] a data processing module, a candidate area generation module, a boundary generation module, and a boundary fusion module.

[0174] The data processing module is configured to collect data of a target farmland and perform geometric and radiometric correction to generate a standardized orthographic image.

[0175] The candidate area generation module is configured to screen pest candidate areas by vegetation health index and texture features.

[0176] The boundary generation module is configured to optimize the candidate areas by mathematical morphology operation and convert them into preliminary pest area vector boundaries.

[0177] The boundary fusion module is configured to determine pest areas by using a pre-trained model and fuse them with the preliminary vector boundaries to generate final pest boundaries.

[0178] The data processing module comprises an image acquisition unit, a distortion correction unit, an image stitching unit, and a radiometric correction unit.

[0179] The image acquisition unit is configured to acquire crop image data and digital surface model data by using a drone equipped with a millimeter-level resolution camera.

[0180] The distortion correction unit is configured to acquire radial / tangential distortion parameters by using a camera calibration tool, correct image pixel coordinates, and eliminate lens distortion errors.

[0181] The image stitching unit is configured to align multiple corrected images to a unified coordinate system by using feature point matching and bundle adjustment based on a structure from motion algorithm to generate a stitched orthographic image.

[0182] The radiometric correction unit is configured to convert crop image DN values into actual radiometric brightness and ground true reflectance to eliminate atmospheric and light interference.

[0183] The candidate area generation module comprises a vegetation health index calculation unit, a texture feature analysis unit, and a candidate area fusion unit.

[0184] a vegetation health index calculation unit configured to extract red and near-infrared bands, calculate a normalized difference vegetation index, and generate a vegetation mask;

[0185] a texture feature analysis unit configured to calculate texture features based on a gray level co-occurrence matrix, detect texture mutation pixels, and generate a mutation pixel mask;

[0186] a candidate region fusion unit configured to perform a logical multiplication operation on the vegetation mask and the mutation pixel mask, generate a pest candidate region mask, and mark pest candidate pixels.

[0187] the boundary generation module includes a morphological operation unit and a vector conversion unit;

[0188] the morphological operation unit is configured to perform an opening operation and a closing operation on the binary candidate region using a diamond structural element, eliminate noise, and connect adjacent patches;

[0189] the vector conversion unit is configured to extract a connected region contour of the image after the opening and closing operation, convert pixel coordinates into geographic coordinates in combination with GNSS positioning data, and generate a vector polygon boundary of a preliminary pest region.

[0190] the boundary fusion module includes a model inference unit, a logical operation unit, and a vector boundary optimization unit;

[0191] the model inference unit is configured to load a pre-trained deep learning model, input a standardized orthographic image, and output a pest binary mask;

[0192] the logical operation unit is configured to rasterize the preliminary vector boundary into a binary mask with the same resolution, and perform a logical AND operation with the model predicted mask to retain a region that satisfies both the model prediction and the morphological screening;

[0193] the vector boundary optimization unit is configured to perform connected region analysis on the fused mask, extract a contour and convert it into a final vector polygon, remove isolated noise points, and generate a final pest boundary.

[0194] Further, a millimeter-level crop image deep learning analysis system further includes a pest grading module;

[0195] the pest grading module is configured to quantify the degree of pest damage and generate a grading report according to the final pest boundary and GNSS positioning data.

[0196] the pest grading module includes a density calculation unit, a damage degree assessment unit, an area calculation unit, and a grading report generation unit;

[0197] the density calculation unit is configured to count pests by a target detection algorithm and calculate the number of pests per unit area;

[0198] The damage degree evaluation unit is configured to combine the vegetation health index and the texture feature change to construct a damage degree index, and divide the damage degree index into light, medium and severe degrees.

[0199] The area calculation unit is configured to calculate the planar projection area or curved surface area of the pest damage area by using the digital surface model, and calculate the area proportion of different levels.

[0200] The grading report generation unit is configured to integrate the pest density, the damage degree index and the area data, and output a grading statistical report.

[0201] Embodiment 3

[0202] An electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the millimeter-level crop image deep learning analysis method when executing the computer program. For details, refer to the above method embodiments, which will not be repeated here.

[0203] The embodiments of the application are described above in combination with the drawings, but the application is not limited to the above specific embodiments, and the above specific embodiments are only illustrative but not restrictive. Those skilled in the art can make changes, modifications, replacements and variations to the above embodiments without departing from the purpose of the application and the scope of protection, and these are all within the protection of the application.

Claims

1. A method for millimeter-level crop image deep learning analysis, characterized in that, The application relates to an agricultural pest detection method based on unmanned aerial vehicle (UAV) and deep learning. The method comprises the following steps: S1: using an unmanned aerial vehicle to acquire target farmland for data collection, synchronously acquiring crop image data, a digital surface model and GNSS positioning data, and obtaining a standardized orthographic image after geometric correction and radiation correction of the acquired crop image data; the unmanned aerial vehicle is provided with a millimeter-level resolution camera; S2: based on the standardized orthographic image, calculating a vegetation health index and a texture feature, and screening out pixels with abnormally low vegetation health index or texture feature mutation, and marking the pixels as pest candidate areas; S3: performing open operation and closed operation on the pest candidate areas to generate a preliminary pest area vector boundary; S4: loading a pre-trained deep learning model, performing logical AND operation on the pest area output by the deep learning model and the generated preliminary pest area vector boundary to generate a final pest edge; The specific steps of S2 comprise the following steps: S2.1: acquiring the standardized orthographic image, extracting a red light band and a near-infrared band from the standardized orthographic image, and calculating a normalized vegetation health index NDVI according to the red light band and the near-infrared band; if the normalized vegetation health index NDVI is greater than or equal to a preset NDVI health threshold, the vegetation mask is set to 1; if the normalized vegetation health index NDVI is less than the preset NDVI health threshold, the vegetation mask is set to 0; S2.2: calculating a gray level co-occurrence matrix of the standardized orthographic image, counting the appearance frequency of pixel pairs in a preset direction and distance, and calculating a texture feature parameter according to the appearance frequency; the texture feature parameter is contrast; if the contrast variation amplitude is greater than a preset amplitude fluctuation threshold, the pixel is marked as a pest candidate pixel, and the mutation pixel mask is set to 1; if the contrast variation amplitude is less than or equal to the preset amplitude fluctuation threshold, the pixel is marked as a non-pest candidate pixel, and the mutation pixel mask is set to 0; S2.3: calculating the product of the vegetation mask and the mutation pixel mask to obtain a pest candidate area mask; S2.4: outputting the pest candidate area mask as an image file to obtain a pest candidate area; The specific steps of S3 comprise the following steps: S3.1: acquiring the pest candidate area mask; the pest candidate area mask is a binary image; in the binary image, the pest candidate pixel is marked as 1, and the non-candidate pixel is marked as 0; S3.2: setting a diamond structure element according to a minimum expected size of a pest patch; S3.3: based on the diamond structure element, performing open operation and closed operation on the binary image to obtain an open-closed operation binary image; the open operation is first erosion and then dilation; the closed operation is first dilation and then erosion; S3.4: extracting the contour of all connected regions in the open-closed operation binary image; 2. The millimeter-scale crop image deep learning analysis method of claim 1, wherein, S3.5: combining the GNSS positioning data, converting the pixel coordinate sequence of each contour into a geographic coordinate sequence, and constructing a closed vector polygon, that is, a preliminary pest area vector boundary; each polygon in the closed vector polygon corresponds to a preliminary pest area. The acquisition process of the standardized orthographic image in S1 comprises the following steps: S1.1: acquiring crop image data and digital surface model data; S1.2: Obtain radial distortion parameters using camera calibration tools and tangential distortion parameters The original coordinates of the pixels in the crop image data Distortion correction is performed to obtain N distortion-corrected crop images; the pixel coordinates in the distortion-corrected crop images are... ; S1.3: Using structure from motion algorithm, align N distortion-corrected crop images to a unified coordinate system through feature point matching and bundle adjustment, and generate a seamless orthographic image; S1.4: Obtain the response function of the camera before shooting, convert the pixel DN value of the orthographic image to the actual radiance value L, and obtain the radiance image; S1.5: Based on the radiance value L, in combination with the distance d, the top-of-atmosphere solar irradiance , the solar zenith angle , the apparent reflectivity of the radiance image is obtained ; S1.6: Convert apparent reflectance of radiance image to ground true reflectance using FLAASH model , after normalization, the normalized orthoimage is obtained, wherein, represents the atmospheric transmittance. 3.The millimeter-level crop image deep learning analysis method of claim 2, wherein, The specific steps of S1.3 include: S1.3.1: Obtain N distortion-corrected crop images, detect local extreme points as feature points for each distortion-corrected crop image using a Gaussian difference function; S1.3.2: Calculate the gradient direction histogram in the neighborhood of the feature points to generate a 128-dimensional vector for each feature point, and obtain the feature descriptor corresponding to the feature point; S1.3.3: calculating the nearest neighbor distance of feature descriptors between different distortion-corrected crop images and the second nearest neighbor distance , combined with a preset distance threshold , feature points corresponding to feature descriptors satisfying are taken as feature matching point pairs; S1.3.4: Calculate the fundamental matrix F from the feature matching point pairs using the random sample consensus algorithm, combine the camera intrinsic matrix to obtain the essential matrix, and then decompose the essential matrix by singular value to obtain the relative rotation matrix and translation vector. 4.The millimeter-level crop image deep learning analysis method of claim 3, wherein, The specific steps of S1.3 also include: S1.3.5: According to the relative rotation matrix and translation vector, combine the known camera extrinsic parameters of each distortion-corrected crop image to construct a minimum reprojection error function, and obtain the initial 3D point cloud by solving the minimum reprojection error function; S1.3.6: Based on the initial 3D point cloud, the camera intrinsic matrix and the camera extrinsic parameters, construct a total reprojection error function, and obtain the 3D point cloud and camera pose by minimizing the total reprojection error function; S1.3.7: Calculate the partial derivative of the total reprojection error function with respect to the 3D point cloud, the camera intrinsic matrix and the camera extrinsic parameters using the chain rule, construct a sparse linear system, and obtain the optimized camera pose; S1.3.8: Based on the optimized camera pose, use the semi-global matching algorithm on all distortion-corrected crop images to generate a dense point cloud by calculating the similarity between pixels; S1.3.9: Filter and interpolate the dense point cloud to generate a digital surface model in the form of a regular grid, and obtain the elevation data of each grid point; S1.3.10: Project each distortion-corrected crop image onto the digital surface model, and fuse the overlapping areas using a weighted average algorithm to generate a seamless orthographic image.

5. The millimeter-scale crop image deep learning analysis method of claim 4, wherein, The specific steps of S4 include: S4.1: Load the pre-trained deep learning model, input the standardized orthographic image into the pre-trained deep learning model, and output the predicted pest binary mask; S4.2: Convert the preliminary pest area vector boundary generated in S3.5 to a binary mask with the same resolution as the standardized orthographic image through rasterization operation to obtain a preliminary vector boundary raster mask; S4.3: Perform logical AND operation on the predicted pest binary mask and the preliminary vector boundary raster mask to obtain the final pest area mask; S4.4: Perform connected region analysis on the final pest area mask, extract the contour and convert it to a vector polygon, which is the final pest boundary.

6. A millimeter crop image deep learning analysis system for implementing the millimeter crop image deep learning analysis method of any one of claims 1-5, characterized in that, It includes: Data processing module, candidate region generation module, boundary generation module, boundary fusion module, pest classification module; The data processing module is configured to collect data of the target farmland, perform geometric and radiation correction, and generate a standardized orthographic image. The candidate region generation module is configured to screen a pest candidate region by using a vegetation health index and a texture feature. The boundary generation module is configured to optimize the candidate region by using mathematical morphology operation, and convert the candidate region into a preliminary pest region vector boundary. The boundary fusion module is configured to determine a pest region by using a pre-trained model, fuse the preliminary vector boundary, and generate a final pest boundary.

7. The millimeter-scale crop image deep learning analysis system of claim 6, wherein, The boundary generation module comprises a morphological operation unit and a vector conversion unit. The morphological operation unit is configured to perform open operation and close operation on the binary candidate region by using a diamond structural element. The vector conversion unit is configured to extract a connected region contour of the image after the open operation and the close operation, convert pixel coordinates into geographic coordinates by combining GNSS positioning data, and generate a vector polygon boundary of the preliminary pest region. 8.An electronic device comprising a memory and a processor, the memory storing a computer program, wherein, The processor executes the computer program to implement the steps of the millimeter-level crop image deep learning analysis method according to any one of claims 1-5.

Images