Image recognition-based plant protection unmanned aerial vehicle fruit tree spraying effect detection device and method

Abstract

This application provides a device and method for detecting the spraying effect of agricultural drones on fruit trees based on image recognition. It collects multispectral images of vegetation before and after spraying, performs sub-pixel-level image registration based on feature point matching on the pre- and post-spraying multispectral images, and generates a temporal difference feature map highlighting canopy changes. It then extracts the reflectance of the tracer's characteristic bands and the vegetation's low-reflectance reference bands from the post-spraying multispectral images, generating a spectral enhancement feature map highlighting pesticide deposition. Based on the signal-to-noise ratio of the temporal difference feature map and the spectral enhancement feature map, pixel-level adaptive weighted fusion is performed to generate a comprehensive feature map. Based on the effective area in the comprehensive feature map, the pesticide coverage and deposition uniformity index at the canopy scale are determined. Using the scheme of this application, both temporal difference and multispectral enhancement features can be fused to quantitatively analyze the pesticide deposition area of ​​sprayed vegetation.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to a device and method for detecting the spraying effect of agricultural drones on fruit trees based on image recognition. Background Technology

[0002] Currently, the detection of the spraying effect of agricultural drones mainly relies on traditional manual visual assessment or ground sampling methods based on collection cards. Among these methods, manual assessment is highly subjective, inefficient, and cannot be quantified. While ground sampling methods can provide quantitative analysis, they are cumbersome, time-consuming, labor-intensive, have limited sample representativeness, and are difficult to obtain information on the overall spatial depositional distribution of the canopy.

[0003] In recent years, visual inspection based on UAV platforms has provided a new approach to detecting spraying effects. However, existing technologies still have the following problems: First, in complex field environments, the visual or spectral signals of pesticide deposits are weak and easily drowned out by vegetation background and various environmental noises. Second, they either only utilize spatial spectral features or only utilize temporal variation features, failing to coordinate the use of multi-dimensional information to jointly constrain and enhance the target, resulting in poor anti-interference ability of the detection results. Third, most methods passively analyze natural scenes without marking or enhancing the detection target in a way that can be effectively identified by sensors, resulting in fundamental limitations due to the inherent low contrast between the target and the background. Therefore, how to integrate temporal difference and multispectral enhancement features to quantitatively analyze the pesticide deposit area of ​​sprayed vegetation has become a challenge for the industry. Summary of the Invention

[0004] Based on this, this application provides an image recognition-based device and method for detecting the spraying effect of agricultural drones on fruit trees, which integrates temporal difference and multispectral enhancement features to quantitatively analyze the pesticide deposition area of ​​sprayed vegetation.

[0005] In a first aspect, this application provides a method for detecting the spraying effect of agricultural drones on fruit trees based on image recognition, comprising the following steps: Multispectral images of vegetation were collected before and after spraying by agricultural drones. The pesticide solution used in the spraying operation was premixed with an inert tracer that has a characteristic reflection peak in the short-wave infrared band. Subpixel-level image registration based on feature point matching is performed on the multispectral image set of vegetation before spraying and the multispectral image set of vegetation after spraying. Then, differential operation is performed on the reflectance images of the same temporal band before and after spraying on the registered image pairs to generate a temporal differential feature map that highlights the changes in the canopy. The characteristic bands of the tracer and the reflectance of the low-reflectance reference band of the vegetation are extracted from the multispectral images of the vegetation after spraying, and then a spectral enhancement feature map that highlights the deposition of the pesticide solution is generated. Pixel-level adaptive weighted fusion is performed based on the signal-to-noise ratio of the temporal difference feature map and the spectral enhancement feature map to generate a comprehensive feature map, and then the effective area of ​​drug deposition is extracted from the comprehensive feature map; The pesticide coverage rate at the canopy scale is determined based on the pixel ratio of the effective area to the entire canopy area, and the deposition uniformity index is determined based on the spatial statistical distribution of pixel values ​​within the effective area, thereby generating a test report on the spraying effect of the plant protection drone on fruit trees.

[0006] In some embodiments, subpixel-level image registration based on feature point matching for the multispectral image set of the vegetation before spraying and the multispectral image set of the vegetation after spraying specifically includes: The images in the multispectral image set of the vegetation before spraying are paired with the images in the multispectral image set of the vegetation after spraying that correspond to the geographical locations, to obtain multiple pairs of images to be registered. For each pair of images to be registered, scale-invariant feature transform points are extracted from the multispectral images before and after spraying, and initial matching is performed. Based on the random sampling consensus algorithm, the geometric transformation model from the post-spraying multispectral image to the pre-spraying multispectral image is estimated from the initially matched feature point pairs, and mismatched feature point pairs are eliminated. The multispectral image after spraying is resampled by bilinear interpolation according to the geometric transformation model, so as to align the multispectral image after spraying with the multispectral image before spraying at the subpixel level to obtain a registered image pair.

[0007] In some embodiments, performing a difference operation on the reflectance images of the same temporal band before and after spraying for the registered image pairs to generate a temporal difference feature map highlighting canopy changes specifically includes: From the registered image pairs, select the band reflectance of the same time phase that is sensitive to changes in vegetation canopy; The initial difference image is generated by subtracting the band reflectance of the multispectral image before spraying from the band reflectance of the multispectral image after spraying pixel by pixel. The initial difference image is processed using a mean filtering algorithm to remove shot noise; The filtered image is used as a temporal difference feature map to highlight canopy variations.

[0008] In some embodiments, extracting the reflectance of the tracer's characteristic bands and the vegetation's low-reflectance reference bands from the multispectral image set of the sprayed vegetation specifically includes: The specific band number corresponding to the characteristic reflection peak of the inert tracer in the short-wave infrared band is determined and used as the characteristic band; The red light band, where vegetation chlorophyll has strong absorption characteristics, was identified as the low-reflection reference band. From each image in the multispectral image set of the post-spray vegetation, the reflectance of the characteristic band and the reflectance of the low-reflectance reference band are read respectively.

[0009] In some embodiments, generating a spectral enhancement feature map that highlights drug deposition specifically includes: For each image in the multispectral image set of the sprayed vegetation, a pixel-by-pixel normalized difference index calculation is performed based on the reflectance of the feature band and the reflectance of the low reflectance reference band extracted from the image to obtain the index value of each pixel. The calculated exponent values ​​of all pixels are combined into a single-channel image; The single-channel image was used as a spectral enhancement feature map to highlight drug deposition. The pixel regions with higher index values ​​correspond to regions where the reflectance of the feature band is significantly higher than that of the reference band, i.e., suspected drug deposition regions.

[0010] In some embodiments, generating a comprehensive feature map by performing pixel-level adaptive weighted fusion based on the signal-to-noise ratio of the temporal difference feature map and the spectral enhancement feature map specifically includes: The temporal difference feature map and the spectral enhancement feature map are normalized respectively to make the pixel value range of the two feature maps uniform; Calculate the signal-to-noise ratio estimates of the temporal difference feature map and the spectral enhancement feature map within the local window, respectively; Based on the signal-to-noise ratio estimate of the two feature maps at each pixel location, the fusion weights of the temporal difference feature map and the spectral enhancement feature map at that pixel location are dynamically allocated; Based on the assigned fusion weights, the temporal difference feature map and the spectral enhancement feature map are subjected to pixel-level weighted summation to generate a comprehensive feature map.

[0011] In some embodiments, extracting the effective region of drug deposition from the comprehensive feature map specifically includes: The comprehensive feature map is subjected to adaptive threshold segmentation to initially segment connected regions with pixel values ​​higher than those of the background region; Morphological opening operations are performed on the connected regions obtained from the initial segmentation to eliminate noise points with excessively small areas; Perform morphological closing operations on connected regions that have undergone opening operations to fill small holes inside the regions; The connected regions after morphological opening and closing operations are used as the effective regions for drug deposition.

[0012] Secondly, this application provides an image recognition-based device for detecting the spraying effect of agricultural drones on fruit trees, comprising: The acquisition module is used to acquire multispectral image sets of vegetation before and after spraying by agricultural drones. The pesticide solution used in the spraying operation is premixed with an inert tracer that has a characteristic reflection peak in the short-wave infrared band. The processing module is used to perform sub-pixel-level image registration based on feature point matching on the multispectral image set of the vegetation before spraying and the multispectral image set of the vegetation after spraying. Then, it performs differential operation on the reflectance images of the same time band before and after spraying on the registered image pairs to generate a temporal differential feature map that highlights the changes in the canopy. The processing module is also used to extract the characteristic bands of the tracer and the reflectance of the low-reflectance reference band of the vegetation from the multispectral image set of the sprayed vegetation, and then generate a spectral enhancement feature map that highlights the deposition of the pesticide solution. The processing module is further configured to perform pixel-level adaptive weighted fusion based on the signal-to-noise ratio of the temporal difference feature map and the spectral enhancement feature map to generate a comprehensive feature map, and then extract the effective area of ​​drug deposition from the comprehensive feature map; The execution module is used to determine the pesticide coverage rate at the canopy scale based on the pixel ratio of the effective area to the entire canopy area, and to determine the deposition uniformity index based on the spatial statistical distribution of pixel values ​​within the effective area, thereby generating a report on the fruit tree spraying effect of the plant protection drone.

[0013] Thirdly, this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the above-described method for detecting the spraying effect of agricultural drones on fruit trees based on image recognition.

[0014] Fourthly, this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for detecting the spraying effect of fruit trees by an image recognition-based agricultural drone.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The image recognition-based device and method for detecting the spraying effect of agricultural drones on fruit trees provided in this application first collects multispectral image sets of vegetation before and after spraying. The pesticide solution used in the spraying operation is premixed with an inert tracer that has a characteristic reflection peak in the short-wave infrared band. This step actively adds the inert tracer with a characteristic reflection peak in the short-wave infrared band to the pesticide solution, thereby creating a unique and easily detectable spectral "fingerprint" for the pesticide deposition area in the post-spraying multispectral image. This significantly improves the spectral separability between the target and the complex vegetation background, addressing the limitations of traditional methods. The visual method addresses the fundamental problem of low signal-to-noise ratio due to the similarity between pesticide and natural vegetation reflectance characteristics. Secondly, it performs sub-pixel-level image registration based on feature point matching on the multispectral image sets of vegetation before and after spraying. Then, it performs differential operations on the reflectance images of the same temporal bands before and after spraying on the registered image pairs to generate a temporal difference feature map highlighting canopy changes. This step achieves high-precision sub-pixel-level registration and differential analysis of multispectral images before and after spraying, thereby accurately extracting the effects caused by the spraying event itself (such as pesticide adhesion and leaf wetting). The method effectively filters out interference from light fluctuations, minor differences in camera angles, and natural canopy variations caused by non-spraying factors by analyzing the "changes" in canopy reflectance, generating pure temporal variation characteristics. Subsequently, the reflectance of the tracer's characteristic bands and the vegetation's low-reflectance reference bands are extracted from the multispectral images of the sprayed vegetation, thereby generating a spectral enhancement feature map highlighting pesticide deposition. This step directly extracts and enhances the tracer's spectral signal from the post-spray images, thereby directly highlighting the pesticide deposition area containing the tracer on a single temporal image based on spectral calculations using the characteristic bands and the vegetation's low-reflectance reference bands, greatly enhancing the effect. The target object has spectral specific characterization capabilities; then, pixel-level adaptive weighted fusion is performed based on the signal-to-noise ratio of the temporal difference feature map and the spectral enhancement feature map to generate a comprehensive feature map, and then the effective area of ​​drug deposition is extracted from the comprehensive feature map. This step can realize the adaptive fusion of the temporal difference feature map and the spectral enhancement feature map based on the local signal-to-noise ratio, thereby intelligently making full use of the dual information advantages of the time dimension and the spectral dimension to generate a comprehensive feature map with more significant features and stronger anti-interference ability, which significantly improves the robustness and accuracy of accurately segmenting the effective area of ​​drug deposition from complex orchard scenes;Finally, the pesticide coverage rate at the canopy scale is determined based on the pixel ratio of the effective area to the entire canopy area, and the deposition uniformity index is determined based on the spatial statistical distribution of pixel values ​​within the effective area. This process generates a report on the spraying effect of the agricultural drone on fruit trees. This step enables quantitative index calculation and automated report generation based on image segmentation results, transforming the extracted effective pesticide deposition area into two intuitive and quantifiable evaluation indicators at the canopy scale: pesticide coverage rate and deposition uniformity index. A graphical and textual report is then generated, providing an objective and efficient basis for evaluating the quality of spraying operations. In summary, the scheme of this application can integrate temporal difference and multispectral enhancement features to quantitatively analyze the pesticide deposition area of ​​sprayed vegetation. Attached Figure Description

[0016] Figure 1 This is an exemplary flowchart of a method for detecting the spraying effect of agricultural drones on fruit trees based on image recognition, according to some embodiments of this application. Figure 2 This is a schematic diagram illustrating an application scenario of a spraying effect detection data processing system according to some embodiments of this application; Figure 3 This is a schematic flowchart illustrating the generation of temporal difference feature maps according to some embodiments of this application; Figure 4 This is a schematic diagram of the structure of an image recognition-based plant protection drone fruit tree spraying effect detection device according to some embodiments of this application; Figure 5 This is a schematic diagram of the structure of a computer device for implementing an image recognition-based method for detecting the spraying effect of agricultural drones on fruit trees, according to some embodiments of this application. Detailed Implementation

[0017] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0018] refer to Figure 1 The figure is an exemplary flowchart of an image recognition-based method for detecting the spraying effect of agricultural drones on fruit trees, according to some embodiments of this application. This image recognition-based method for detecting the spraying effect of agricultural drones on fruit trees mainly includes the following steps: In step 101, a multispectral image set of vegetation before and after spraying by the agricultural drone is acquired. The pesticide solution used in the spraying operation is premixed with an inert tracer that has a characteristic reflection peak in the short-wave infrared band.

[0019] In practice, the acquisition of multispectral image sets of vegetation before and after spraying by the agricultural drone can be achieved in the following way: First, select an agricultural drone platform equipped with a high-precision global navigation satellite system real-time dynamic differential positioning module, and carry a multispectral camera that includes at least the blue, green, red, red-edge, near-infrared, and short-wave infrared bands; then, before the spraying operation, plan an autonomous flight path covering the canopy of the target fruit trees through the ground control station software, set a fixed flight altitude, forward overlap rate, and lateral overlap rate, and save the flight path file; next, control the agricultural drone to perform its first flight along the saved flight path, and the multispectral camera will... During the process, original images are automatically acquired at preset time intervals or distance intervals, and the position and attitude data of each image are recorded synchronously to obtain a multispectral image set of vegetation before spraying. During the spraying operation, an inert tracer is uniformly mixed into the plant protection liquid at a predetermined ratio. The inert tracer is a material that has a characteristic reflection peak at a specific wavelength in the short-wave infrared band and is safe and non-toxic to the environment and crops. After the spraying operation is completed, under ambient light conditions similar to those before the acquisition, the plant protection drone is controlled to call the saved flight path file to perform a second flight, acquiring a multispectral image set of vegetation after spraying at the same flight altitude, speed and camera parameters. Other methods can also be used in other embodiments, which are not limited here.

[0020] It should be noted that in this application, radiometric calibration and geometric correction are also required for the multispectral image sets of vegetation collected before and after spraying. Radiometric calibration uses standard reflectance whiteboard images taken before and after flight to convert the original digital quantization values ​​into surface reflectance values. Geometric correction is based on the recorded location and attitude data to eliminate image distortion and generate a reflectance image set with geographic reference as the basis for subsequent processing. Other methods can also be used in other embodiments, which are not limited here.

[0021] In addition, it should be noted that the above steps can actively add an inert tracer with a characteristic reflection peak in the short-wave infrared band to the pesticide solution, thereby creating a unique and easily detectable spectral "fingerprint" for the pesticide deposition area in the multispectral image after spraying. This significantly improves the spectral separability between the target and the complex vegetation background, providing a prerequisite for solving the fundamental problem of low detection signal-to-noise ratio caused by the similar natural reflection characteristics of pesticide solution and vegetation in traditional visual methods.

[0022] In some embodiments, reference Figure 2As shown in the figure, this figure is a schematic diagram of the application scenario of the spraying effect detection data processing system shown in some embodiments of this application. The figure includes three main components: acquisition device, server and data storage device. The acquisition device is responsible for collecting multispectral image sets of vegetation before and after spraying by the plant protection drone, and sending the collected multispectral image sets of vegetation before and after spraying to the server through the communication network. The spraying effect detection data processing system runs on the server, and the server stores the processing results in the data storage device and visualizes them.

[0023] In step 102, subpixel-level image registration based on feature point matching is performed on the multispectral image set of vegetation before spraying and the multispectral image set of vegetation after spraying. Then, differential operation is performed on the reflectance images of the same time band before and after spraying on the registered image pairs to generate a temporal differential feature map that highlights the changes in the canopy.

[0024] In some embodiments, subpixel-level image registration based on feature point matching for the multispectral image set of the vegetation before spraying and the multispectral image set of the vegetation after spraying can be achieved by the following steps: The images in the multispectral image set of the vegetation before spraying are paired with the images in the multispectral image set of the vegetation after spraying that correspond to the geographical locations, to obtain multiple pairs of images to be registered. For each pair of images to be registered, scale-invariant feature transform points are extracted from the multispectral images before and after spraying, and initial matching is performed. Based on the random sampling consensus algorithm, the geometric transformation model from the post-spraying multispectral image to the pre-spraying multispectral image is estimated from the initially matched feature point pairs, and mismatched feature point pairs are eliminated. The multispectral image after spraying is resampled by bilinear interpolation according to the geometric transformation model, so as to align the multispectral image after spraying with the multispectral image before spraying at the subpixel level to obtain a registered image pair.

[0025] It should be noted that the subpixel-level image registration in this application can be used to eliminate subpixel-level spatial misalignment between two images before and after spraying due to minor differences in UAV positioning, attitude, etc., and is a prerequisite for achieving high-precision temporal difference operations; the bilinear interpolation resampling can be used to smoothly estimate the value of new pixels, avoiding jagged edges or information loss caused by direct rounding.

[0026] In specific implementation, pairing images from the pre-spray vegetation multispectral image set with geographically corresponding images from the post-spray vegetation multispectral image set to obtain multiple image pairs to be registered can be achieved in the following way: read the GPS positioning and attitude data attached to each image in the pre-spray vegetation multispectral image set, and the GPS positioning and attitude data attached to each image in the post-spray vegetation multispectral image set; calculate the Euclidean distance between each post-spray image and all pre-spray images; for each post-spray image, select the pre-spray image with the smallest Euclidean distance to it, and combine the two into an image pair to be registered; traverse all post-spray images to complete the above matching, and finally obtain multiple image pairs to be registered that correspond one-to-one in space. Other methods can also be used in other embodiments, which are not limited here.

[0027] In specific implementation, for each image pair to be registered, the extraction of scale-invariant feature transform (SMT) feature points from the pre-spray and post-spray multispectral images and the initial matching can be achieved as follows: For the pre-spray and post-spray multispectral images in the image pair to be registered, they are first converted from multi-band reflectance images to grayscale images; then, the SMT algorithm is run on the two grayscale images respectively. This algorithm detects stable keypoint positions and scales by constructing a Gaussian difference scale space and calculates a 128-dimensional feature descriptor based on the gradient orientation histogram for each keypoint; next, for each keypoint in the post-spray multispectral image, the Euclidean distance between its feature descriptor and the feature descriptors of all keypoints in the pre-spray multispectral image is calculated; the pair of keypoints with the closest distance and a distance less than a set empirical threshold is recorded as the initial matched feature point pair. Other methods can also be used in other embodiments, which are not limited here.

[0028] In specific implementation, the geometric transformation model from the post-spray multispectral image to the pre-spray multispectral image is estimated from the initially matched feature point pairs based on the random sampling consensus algorithm, and mismatched feature point pairs are eliminated. This can be achieved in the following way: First, a distance threshold is preset to determine whether a feature point pair is an interior point; then, four non-collinear sample point pairs are randomly selected from the initially matched feature point pairs, and a homography matrix from the post-spray multispectral image coordinates to the pre-spray multispectral image coordinates is calculated as a candidate geometric transformation model; this candidate geometric transformation model is used to transform the multispectral image after spraying... The coordinates of the initially matched feature points are transformed, and the projection error between the transformed coordinates and the corresponding feature point coordinates in the multispectral image before spraying is calculated. Feature point pairs with projection errors less than the distance threshold are marked as inliers of the current candidate model. The above random sampling, model calculation, and inlier statistics process is repeated for a preset number of iterations, and finally the candidate model with the most inliers is selected as the optimal geometric transformation model. All initially matched feature point pairs that are not determined as inliers by the optimal geometric transformation model are discarded as mismatched feature point pairs. Other methods can be used in other embodiments, which are not limited here.

[0029] In specific implementation, bilinear interpolation resampling is performed on the multispectral image after spraying according to the geometric transformation model to achieve sub-pixel-level spatial alignment between the multispectral image after spraying and the multispectral image before spraying, resulting in a registered image pair. This can be achieved in the following way: using the optimal geometric transformation model, the source coordinates (usually non-integer precision) corresponding to each integer pixel coordinate point in the multispectral image before spraying are calculated in the multispectral image after spraying; for each non-integer source coordinate, its four nearest integer pixel coordinate points in the multispectral image after spraying are found. Based on the relative distance between the non-integer source coordinates and the four surrounding integer pixel coordinates, bilinear weighted interpolation is performed on the pixel reflectance values ​​of these four points to calculate a new reflectance value at that location. This new reflectance value is then assigned to the corresponding integer pixel coordinates in the pre-spray multispectral image, thereby generating a resampled post-spray multispectral image that is spatially perfectly aligned with the pre-spray multispectral image. This resampled post-spray multispectral image and the original pre-spray multispectral image together constitute a registered image pair. Other methods can also be used in other embodiments, which are not limited here.

[0030] In some embodiments, reference Figure 3 As shown, this figure is a flowchart illustrating the generation of temporal difference feature maps in some embodiments of this application. In this embodiment, the difference operation is performed between the reflectance images of the same temporal band before and after spraying of the registered image pairs to generate a temporal difference feature map highlighting canopy changes. This can be achieved by the following steps: In step 1031, from the registered image pair, the band reflectance of the same time phase that is sensitive to changes in vegetation canopy is selected; In step 1032, the band reflectance corresponding to the multispectral image before spraying is subtracted pixel by pixel from the band reflectance corresponding to the multispectral image after spraying to generate an initial differential image. In step 1033, a mean filtering algorithm is applied to the initial difference image to remove shot noise; In step 1034, the filtered image is used as a temporal difference feature map that highlights the changes in the canopy.

[0031] It should be noted that the band reflectance in this application is the value of each pixel in a specific spectral band in the image after radiometric calibration. It quantifies the reflectivity of ground objects, such as vegetation canopy, for electromagnetic waves in that band and serves as a standardized and comparable basic data unit for all spectral analyses and calculations. The initial difference image is an intermediate result image obtained by directly subtracting pixels from the registered post-spraying image and the pre-spraying image during the generation of the temporal difference feature map. The temporal difference feature map is a single-channel image in which the value of each pixel represents the change in reflectance of the same geographical location in a specific band before and after spraying. Its function is to highlight the area of ​​change in canopy reflectance characteristics caused by the spraying event, while suppressing the unchanged environmental background.

[0032] In specific implementation, selecting the reflectance of the same temporal band sensitive to changes in vegetation canopy from the registered image pair can be achieved in the following way: based on the spectral characteristics of vegetation and the spectral characteristics of the inert tracer, the near-infrared band is selected as the same temporal band sensitive to changes in vegetation canopy, because this band is sensitive to changes in leaf internal structure, biomass and water content, and the spraying solution and tracer may affect the reflectance characteristics of this band; the reflectance image of the near-infrared band is extracted from the multispectral image before spraying in the registered image pair, and the reflectance image of the same near-infrared band is extracted from the resampled multispectral image after spraying in the registered image pair. Other methods can also be used in other embodiments, which are not limited here.

[0033] In specific implementation, the initial difference image is generated by subtracting the band reflectance of the multispectral image before spraying from the band reflectance of the multispectral image after spraying pixel by pixel. This can be achieved in the following way: create an empty matrix with the same size as the near-infrared band reflectance image as the initial difference image; sequentially traverse the coordinates of each pixel in the near-infrared band reflectance image, read the reflectance value of the near-infrared band at that coordinate in the resampled multispectral image after spraying, and read the reflectance value of the near-infrared band at the same coordinate in the multispectral image before spraying; subtract the latter from the former, and store the difference in the corresponding coordinate position of the empty matrix; after traversing all pixels, the empty matrix stores the difference value of the near-infrared reflectance at each pixel position after spraying and before spraying. This matrix is ​​the initial difference image. Other methods can also be used in other embodiments, which are not limited here.

[0034] In specific implementation, the application of the median filtering algorithm to remove shot noise in the initial difference image can be implemented in the following way: Define a sliding window centered on the target pixel, with a window size of 3 pixels by 3 pixels; traverse the sliding window to every pixel of the initial difference image, and for each pixel at the center of the window, extract the reflectance difference of all pixels within the window; sort the extracted reflectance differences and take the value at the middle position of the sorted list, i.e., the median; replace the original reflectance difference of the pixel at the center of the window in the initial difference image with this median; after the traversal is completed, the image after median filtering is obtained. Other methods can also be used in other embodiments, which are not limited here.

[0035] In specific implementation, the filtered image can be used as a temporal difference feature map highlighting canopy changes in the following way: the image after median filtering is used as the final output image characterizing the degree of change in near-infrared reflectance of the canopy before and after spraying. This image is the temporal difference feature map highlighting canopy changes. In this temporal difference feature map, areas with pixel values ​​significantly greater than zero may correspond to canopy areas with increased reflectance due to pesticide deposition, areas with pixel values ​​significantly less than zero may correspond to areas with decreased reflectance caused by other factors, and areas with pixel values ​​close to zero represent canopy backgrounds that have not changed significantly. Other methods can also be used in other embodiments, which are not limited here.

[0036] It should be noted that the above steps can achieve high-precision sub-pixel level registration and differentiation of multispectral images before and after spraying, thereby accurately extracting the "change in canopy reflectance" caused by the spraying event itself (such as pesticide adhesion and leaf wetting), effectively filtering out interferences such as light fluctuations, slight differences in camera angles, and natural changes in the canopy caused by non-spraying, and generating pure temporal variation characteristics.

[0037] In step 103, the characteristic bands of the tracer and the reflectance of the low-reflectance reference band of the vegetation are extracted from the multispectral image set of the vegetation after spraying, and then a spectral enhancement feature map highlighting the deposition of the pesticide solution is generated.

[0038] In some embodiments, extracting the reflectance of the tracer's characteristic bands and the vegetation's low-reflectance reference bands from the multispectral image set of the sprayed vegetation can be achieved by the following steps: The specific band number corresponding to the characteristic reflection peak of the inert tracer in the short-wave infrared band is determined and used as the characteristic band; The red light band, where vegetation chlorophyll has strong absorption characteristics, was identified as the low-reflection reference band. From each image in the multispectral image set of the post-spray vegetation, the reflectance of the characteristic band and the reflectance of the low-reflectance reference band are read respectively.

[0039] It should be noted that the characteristic band in this application is a specific channel of a multispectral camera pre-selected based on the spectral characteristics of the inert tracer used. Its function is to serve as a key detection window, so that the image data acquired in this band can generate the strongest response signal to the tracer component in the drug solution, thereby distinguishing it from the background such as vegetation in the spectrum. The low reflectance reference band is a multispectral camera channel selected in the low reflectance region of the vegetation spectral characteristics. Its function is to provide a stable background reference. The reflectance of the vegetation itself is very low in this band, so when performing spectral calculations with the characteristic band, the high reflectance signal caused by the tracer in the characteristic band can be maximized.

[0040] In specific implementation, the specific band number corresponding to the characteristic reflection peak of the inert tracer in the short-wave infrared band is determined as the characteristic band. This can be achieved in the following way: using a spectrometer to measure the reflectance spectrum curve of the inert tracer sample in the short-wave infrared band; analyzing the reflectance spectrum curve to identify the center wavelength corresponding to the peak position with the highest reflectance; comparing the center wavelength with the center wavelength of each channel of the multispectral camera in the short-wave infrared band, selecting the channel whose center wavelength is closest to the center wavelength of the peak position, and recording the channel number as the characteristic band. This characteristic band is the characteristic band used for subsequent reflectance extraction. Other methods can also be used in other embodiments, which are not limited here.

[0041] In specific implementation, determining the red light band with strong absorption characteristics of vegetation chlorophyll as the low-reflection reference band can be achieved in the following way: based on common knowledge in vegetation spectroscopy, it is determined that chlorophyll has a strong absorption valley in the visible red light region, and the typical wavelength range corresponding to this strong absorption valley is 650 nm to 680 nm; the center wavelength of each band channel of the multispectral camera is compared with this typical wavelength range, and the channel whose center wavelength falls within the range of 650 nm to 680 nm is selected, and this channel is used as the low-reflection reference band. This low-reflection reference band has a low reflectivity value in the vegetation area. Other methods can also be used in other embodiments, which are not limited here.

[0042] In specific implementation, the reflectance of the characteristic band and the reflectance of the low-reflectance reference band can be read from each image of the multispectral image set of the sprayed vegetation in the following manner: For each reflectance image of the sprayed vegetation that has undergone radiometric calibration and geometric correction in the multispectral image set, the reflectance data matrix of the corresponding characteristic band is located and read from multiple band data layers of the reflectance image based on the characteristic band. Based on the low-reflectance reference band, the reflectance data matrix of the corresponding low-reflectance reference band is located and read from multiple band data layers of the same reflectance image, thereby obtaining the reflectance data of the characteristic band and the reflectance data of the low-reflectance reference band corresponding to each image. Other methods can also be used in other embodiments, which are not limited here.

[0043] In some embodiments, generating a spectral enhancement feature map highlighting drug deposition can be achieved by the following steps: For each image in the multispectral image set of the sprayed vegetation, a pixel-by-pixel normalized difference index calculation is performed based on the reflectance of the feature band and the reflectance of the low reflectance reference band extracted from the image to obtain the index value of each pixel. The calculated exponent values ​​of all pixels are combined into a single-channel image; The single-channel image was used as a spectral enhancement feature map to highlight drug deposition. The pixel regions with higher index values ​​correspond to regions where the reflectance of the feature band is significantly higher than that of the reference band, i.e., suspected drug deposition regions.

[0044] It should be noted that the spectral enhancement feature map in this application is a single-channel image generated by performing a specific spectral index calculation on the reflectance of the characteristic band and the low-reflectance reference band in the post-spray image. Its function is to directly enhance and highlight the drug deposition area containing tracer in the spectral dimension, so that its pixel value forms a significant contrast with the background area.

[0045] In specific implementation, for each image in the multispectral image set of the sprayed vegetation, a pixel-by-pixel normalized difference exponent calculation is performed based on the feature band reflectance and low-reflectance reference band reflectance extracted from the image to obtain the exponent value of each pixel. This can be achieved in the following way: For each image in the multispectral image set of the sprayed vegetation, obtain the feature band reflectance data matrix and the low-reflectance reference band reflectance data matrix extracted from the image. These two matrices have the same number of rows and columns. Iterate through each pixel position in the matrix, read the reflectance value of the pixel position in the feature band reflectance data matrix, and the reflectance value of the same pixel position in the low-reflectance reference band reflectance data matrix. Subtract the low-reflectance reference band reflectance value from the feature band reflectance value of the pixel, and then divide by the sum of the feature band reflectance value and the low-reflectance reference band reflectance value to obtain the exponent value of the normalized difference of the pixel position. Other methods can also be used in other embodiments, which are not limited here.

[0046] In specific implementation, combining the calculated exponent values ​​of all pixels into a single-channel image can be achieved in the following way: after traversing all pixel positions in the matrix, an exponent value matrix with the same size as the input reflectance data matrix is ​​obtained, which stores the exponent value of each pixel. The calculated exponent value matrix is ​​stored in a standard image format, which is a single-channel floating-point or integer raster image. The gray value or pixel value of each pixel in the image is the exponent value of the normalized difference calculated for that pixel position. Other methods can also be used in other embodiments, which are not limited here.

[0047] In specific implementation, the single-channel image is used as a spectral enhancement feature map to highlight drug deposition. Pixel regions with higher index values ​​correspond to areas where the reflectance of the characteristic band is significantly higher than that of the reference band, i.e., suspected drug deposition areas. This can be achieved by outputting the index value matrix stored in a single-channel image format as a spectral enhancement feature map. In this spectral enhancement feature map, regions with higher pixel values ​​indicate a greater positive difference between the reflectance of the characteristic band and the low-reflectance reference band in the image after spraying. Based on the spectral characteristics of the inert tracer, such regions with significantly higher reflectance of the characteristic band than the reference band are more likely to be interpreted as drug deposition areas. Therefore, this spectral enhancement feature map serves to highlight suspected drug deposition areas. Other methods can also be used in other embodiments, which are not limited here.

[0048] It should be noted that the above steps can directly extract and enhance the spectral signal of the tracer from the post-spraying image. Based on the spectral calculation of the characteristic band and the low-reflectance reference band of the vegetation, the area of ​​pesticide deposition containing the tracer can be directly highlighted on a single temporal image, which greatly enhances the spectral specificity characterization ability of the target.

[0049] In step 104, pixel-level adaptive weighted fusion is performed based on the signal-to-noise ratio of the temporal difference feature map and the spectral enhancement feature map to generate a comprehensive feature map, and then the effective area of ​​drug deposition is extracted from the comprehensive feature map.

[0050] In some embodiments, generating a comprehensive feature map by performing pixel-level adaptive weighted fusion based on the signal-to-noise ratio of the temporal difference feature map and the spectral enhancement feature map can be achieved through the following steps: The temporal difference feature map and the spectral enhancement feature map are normalized respectively to make the pixel value range of the two feature maps uniform; Calculate the signal-to-noise ratio estimates of the temporal difference feature map and the spectral enhancement feature map within the local window, respectively; Based on the signal-to-noise ratio estimate of the two feature maps at each pixel location, the fusion weights of the temporal difference feature map and the spectral enhancement feature map at that pixel location are dynamically allocated; Based on the assigned fusion weights, the temporal difference feature map and the spectral enhancement feature map are subjected to pixel-level weighted summation to generate a comprehensive feature map.

[0051] It should be noted that the signal-to-noise ratio (SNR) estimate in this application is a quantitative evaluation of the ratio of signal intensity to noise intensity within a local region of the image. It is used to measure the reliability and quality of the temporal difference feature map and the spectral enhancement feature map at different spatial locations. The higher the value, the more reliable the feature information at that location and the less noise interference. The integrated feature map is generated by adaptively weighting and fusing the temporal difference feature map and the spectral enhancement feature map according to their respective SNRs to produce a new, integrated feature image. Its function is to integrate the advantages of the two complementary features to generate a more robust, more significant, and more conducive fused feature representation for subsequent accurate segmentation.

[0052] In specific implementation, the temporal difference feature map and the spectral enhancement feature map are normalized to unify the pixel value range of the two feature maps. This can be achieved in the following way: find the minimum and maximum values ​​of all pixels in the temporal difference feature map and the minimum and maximum values ​​of all pixels in the spectral enhancement feature map; for each pixel value in the temporal difference feature map, subtract the minimum value of the feature map and divide it by the difference between the maximum and minimum values ​​of the feature map, thereby linearly transforming the pixel value to the range of 0 to 1; for each pixel value in the spectral enhancement feature map, use the same linear transformation method, that is, subtract the minimum value of its own feature map and divide it by the difference between the maximum and minimum values, also transforming it to the range of 0 to 1; after this processing, the two feature maps have a unified pixel value range. Other methods can also be used in other embodiments, which are not limited here.

[0053] In specific implementation, the signal-to-noise ratio (SNR) estimates of the temporal difference feature map and the spectral enhancement feature map within the local window can be calculated as follows: Define a rectangular sliding window centered on the target pixel, for example, with a size of 5 pixels multiplied by 5 pixels; traverse this sliding window on the normalized temporal difference feature map and the normalized spectral enhancement feature map respectively; for each pixel position at the center of the window, calculate the average value of all pixel values ​​within the window as an estimate of the signal strength, and calculate the standard deviation of all pixel values ​​within the window as an estimate of the noise strength; divide the estimated signal strength at this position by the estimated noise strength, and the quotient is the local SNR estimate of that pixel position in the current feature map; perform the above calculations for each pixel position of each feature map, thereby generating a temporal difference feature map SNR estimation matrix and a spectral enhancement feature map SNR estimation matrix with the same size as the feature map. Other methods can also be used in other embodiments, which are not limited here.

[0054] In specific implementation, the dynamic allocation of fusion weights between the temporal difference feature map and the spectral enhancement feature map at each pixel location, based on the signal-to-noise ratio (SNR) estimates of the two feature maps at each pixel location, can be achieved in the following way: For each pixel location in the image, read the SNR estimate of that location in the SNR estimation matrix of the temporal difference feature map and the SNR estimate of the spectral enhancement feature map; divide the SNR estimate of the temporal difference feature map at that pixel location by the SNR estimate of the temporal difference feature map and the spectral enhancement feature map at that pixel location. The sum of the signal-to-noise ratio estimates is used as the fusion weight of the temporal difference feature map at that pixel location. Since the sum of the fusion weights of the two feature maps at that pixel location should be 1, the fusion weight of the spectral enhancement feature map at that pixel location is 1 minus the fusion weight of the temporal difference feature map at that pixel location. Performing this calculation on all pixel locations yields two weight matrices of the same size as the feature maps, representing the fusion weights of each pixel location for the temporal difference feature map and the spectral enhancement feature map, respectively. Other methods can also be used in other embodiments, which are not limited here.

[0055] In specific implementation, based on the assigned fusion weights, a pixel-level weighted summation operation is performed on the temporal difference feature map and the spectral enhancement feature map to generate a comprehensive feature map. This can be achieved in the following way: First, an empty matrix with the same size as the temporal difference feature map is created to store the data of the comprehensive feature map. Then, each pixel position is traversed sequentially, and the pixel value of that position in the normalized temporal difference feature map and the pixel value of that position in the normalized spectral enhancement feature map are read. At the same time, the fusion weights of the temporal difference feature map and the fusion weights of the spectral enhancement feature map corresponding to that position are read. The pixel value of the temporal difference feature map is multiplied by its fusion weight, and the pixel value of the spectral enhancement feature map is multiplied by its fusion weight. Then, these two products are added together, and the result is the pixel value of that pixel position in the comprehensive feature map. This result is stored in the corresponding position of the empty matrix. After traversing all pixel positions, the matrix storing all the calculation results is the generated comprehensive feature map. Other methods can also be used in other embodiments, which are not limited here.

[0056] In some embodiments, extracting the effective region of drug deposition from the comprehensive feature map can be achieved by the following steps: The comprehensive feature map is subjected to adaptive threshold segmentation to initially segment connected regions with pixel values ​​higher than those of the background region; Morphological opening operations are performed on the connected regions obtained from the initial segmentation to eliminate noise points with excessively small areas; Perform morphological closing operations on connected regions that have undergone opening operations to fill small holes inside the regions; The connected regions after morphological opening and closing operations are used as the effective regions for drug deposition.

[0057] In specific implementation, adaptive threshold segmentation is performed on the comprehensive feature map to initially segment connected regions with pixel values ​​higher than the background region. This can be achieved in the following way: the Otsu method is used to process the comprehensive feature map. This algorithm determines an optimal segmentation threshold by calculating the inter-class variance. The comprehensive feature map is then binarized using this optimal segmentation threshold. Pixels with pixel values ​​greater than or equal to the threshold are set as foreground points, and pixels with pixel values ​​less than the threshold are set as background points. After binarization, all interconnected foreground pixels constitute one or more connected regions. These connected regions are the initially segmented connected regions with pixel values ​​higher than the background region. Other methods can also be used in other embodiments, which are not limited here.

[0058] In specific implementation, morphological opening operations are performed on the initially segmented connected regions to eliminate noise points with excessively small areas. This can be achieved as follows: Define a structuring element for morphological operations, such as a circular structuring element with a radius of 2 pixels; First, perform an erosion operation on the initially segmented binary image, that is, let the structuring element slide on the image. Only when the structuring element is completely covered by the foreground region is the center pixel retained as the foreground; otherwise, it is set as the background. This operation will shrink the boundaries of all connected regions; Then, perform a dilation operation on the eroded image, that is, let the same structuring element slide on the image. As long as the structuring element intersects with the foreground region, the center pixel is set as the foreground. This operation will restore part of the size of the shrunken area, but isolated small noise points will be completely eliminated through the entire process of erosion followed by dilation. Other methods can also be used in other embodiments, which are not limited here.

[0059] In specific implementation, morphological closing operations can be performed on connected regions that have undergone opening operations to fill small holes within the regions. This can be achieved by using a structuring element of the same size and shape as the opening operation, such as a circular structuring element with a radius of 2 pixels. First, a dilation operation is performed on the image after the opening operation, which expands the boundaries of all foreground regions. Then, an erosion operation is performed on the image after the dilation operation, which shrinks the expanded regions back to roughly their original boundaries. After the closing operation process of dilation followed by erosion, the small holes within the connected regions will be filled, while the overall outline of the regions remains essentially unchanged. Other methods can also be used in other embodiments, which are not limited here.

[0060] In specific implementation, the connected regions after morphological opening and closing operations can be used as the effective regions for drug deposition in the following way: the binary image obtained after the final morphological closing operation is directly used as the output mask image, and the white connected regions with all pixel values ​​of 1 are defined as the effective regions for drug deposition. Other methods can also be used in other embodiments, which are not limited here.

[0061] It should be noted that the above steps can achieve adaptive fusion of temporal difference feature map and spectral enhancement feature map based on local signal-to-noise ratio, thereby intelligently making full use of the dual information advantages of the time dimension and spectral dimension to generate a comprehensive feature map with more significant features and stronger anti-interference ability, which significantly improves the robustness and accuracy of accurately segmenting the effective area of ​​pesticide deposition from complex orchard scenes.

[0062] In step 105, the pesticide coverage rate at the canopy scale is determined based on the pixel ratio of the effective area to the entire canopy area, and the deposition uniformity index is determined based on the spatial statistical distribution of pixel values ​​within the effective area, thereby generating a test report on the fruit tree spraying effect of the plant protection drone.

[0063] In some embodiments, determining the drug coverage at the canopy scale based on the pixel ratio of the effective region to the entire canopy region can be achieved by the following steps: The normalized vegetation index was calculated based on the multispectral images after spraying, and the entire canopy area was segmented. Determine the logical intersection between the effective area of ​​drug deposition and the entire canopy region to obtain the actual deposition area within the canopy. Count the total number of pixels in the actual deposition area and the total number of pixels in the entire canopy region; Divide the total number of pixels in the actual deposition area by the total number of pixels in the entire canopy area and multiply by 100% to obtain the drug coverage rate at the canopy scale.

[0064] It should be noted that the actual deposition area in this application refers to the area obtained by performing a logical AND operation between the effective deposition area of ​​the pesticide solution segmented from the image and the entire canopy area determined by the vegetation index. This defines the actual pesticide solution deposition range attached to the fruit tree canopy vegetation, excluding areas deposited on non-target backgrounds such as soil and branches, and is the effective statistical range for calculating the coverage rate. The pesticide solution coverage rate is a percentage quantitative indicator used to evaluate the size of the coverage area of ​​the sprayed pesticide solution on the surface of the target fruit tree canopy, and is one of the core effect indicators for measuring the quality of spraying operations.

[0065] In specific implementation, calculating the normalized vegetation index (NDI) and segmenting the entire canopy region based on the multispectral images after spraying can be achieved in the following way: Select the near-infrared reflectance image and the red reflectance image from the multispectral image set of the sprayed vegetation after radiometric calibration and geometric correction; for each pixel in the image, subtract the red reflectance value from the near-infrared reflectance value, and then divide by the sum of the near-infrared and red reflectance values ​​to obtain the NDI value for each pixel; then, calculate the NDI value for all the images... The normalized vegetation index (NVI) values ​​of the pixels are combined to form a single-channel NVI map. An empirical threshold is used to perform binarization segmentation on the NVI map. All pixels with NVI values ​​greater than the empirical threshold are identified as vegetation canopy, and all pixels with NVI values ​​less than or equal to the empirical threshold are identified as non-vegetation background. In the resulting binary image, the region formed by all pixels identified as vegetation canopy is the entire canopy region. Other methods can also be used in other embodiments, which are not limited here.

[0066] In specific implementation, determining the logical intersection of the effective area of ​​drug deposition and the entire canopy region to obtain the actual deposition area within the canopy can be achieved in the following way: Obtain a binary image representing the entire canopy region, where the pixel value of the canopy region is 1 and the pixel value of the background region is 0; simultaneously obtain a binary image representing the effective area of ​​drug deposition, where the pixel value of the effective area is 1 and the pixel value of the background region is 0; perform a pixel-by-pixel logical AND operation on these two binary images, i.e., only when the pixel value of the same pixel position is 1 in both images is the pixel value of the resulting image 1, otherwise it is 0; the new binary image obtained after the logical AND operation, where the area with a pixel value of 1 represents the portion of the effective area of ​​drug deposition that is simultaneously located within the entire canopy region, is the actual deposition area within the canopy. Other methods can also be used in other embodiments, which are not limited here.

[0067] In some embodiments, determining the deposition uniformity index based on the spatial statistical distribution of pixel values ​​within the effective region can be achieved using the following steps: Extract the pixel values ​​of all pixels corresponding to the effective area of ​​drug deposition from the comprehensive feature map; The mean and standard deviation of all extracted pixel values ​​are determined, and then the standard deviation is divided by the mean to obtain the coefficient of variation; The deposition uniformity index is calculated based on the coefficient of variation.

[0068] It should be noted that the coefficient of variation in this application is used to measure the proportion of the dispersion of the values ​​of all pixels in the comprehensive feature map within the actual deposition area relative to its average level. Its function is to characterize the relative fluctuation or non-uniformity of the deposition intensity of the pesticide in spatial distribution, and it is an intermediate measure for calculating the deposition uniformity index. The deposition uniformity index is a comprehensive evaluation index that quantifies the uniformity of the pesticide distribution on the canopy surface. The higher the index value, the more uniform the deposition; the lower the value, the more concentrated the deposition or the greater the distribution difference. It is another core performance indicator for measuring the quality of spraying operations.

[0069] In specific implementation, extracting the pixel values ​​of all pixels corresponding to the effective area of ​​drug deposition from the comprehensive feature map can be achieved in the following way: using the binary image of the effective area of ​​drug deposition as a mask, and registering and aligning it with the comprehensive feature map; sequentially traversing each pixel position in the comprehensive feature map, and checking the value of the binary image of the effective area of ​​drug deposition at the same position; if the value at that position is 1, then reading the pixel value at that position from the comprehensive feature map and recording it in a value list; after traversal, all the values ​​stored in the value list are the pixel values ​​of all pixels corresponding to the effective area of ​​drug deposition extracted from the comprehensive feature map. Other methods can also be used in other embodiments, which are not limited here.

[0070] In specific implementation, the average and standard deviation of all extracted pixel values ​​can be determined as follows: sum all pixel values ​​stored in the numerical list to obtain the total sum of pixel values; divide the total sum of pixel values ​​by the total number of pixel values ​​in the numerical list to obtain the average of all extracted pixel values; calculate the difference between each pixel value in the numerical list and the average, square each difference, and then sum all squared differences to obtain the sum of squared differences; divide the sum of squared differences by the total number of pixel values ​​in the numerical list to obtain the variance; and take the square root of the variance to obtain the standard deviation of all extracted pixel values. Other methods can also be used in other embodiments, which are not limited here.

[0071] In specific implementation, the deposition uniformity index can be calculated based on the coefficient of variation in the following way: subtract the coefficient of variation from the value 1, and the difference is taken as the deposition uniformity index. The value of the deposition uniformity index is between 0 and 1. The closer the value is to 1, the more uniform the spatial distribution of pixel values ​​in the effective area of ​​the drug deposition. The closer the value is to 0, the more uneven the distribution. Other methods can also be used in other embodiments, which are not limited here.

[0072] In specific implementation, generating a report on the spraying effect of agricultural drones on fruit trees can be achieved in the following way: First, the calculated pesticide coverage rate and deposition uniformity index at the canopy scale, along with basic information about the flight data collection mission, including collection time, geographical location, drone and camera model, are integrated into a structured text data summary; then, based on the binary image of the actual deposition area within the canopy and the binary image of the entire canopy area, an overlay display image is generated, in which different colors or fill patterns are used to distinguish the canopy background from the actual deposition area; simultaneously, the pesticide coverage rate corresponding to the pesticide in the comprehensive feature image is... The pixel values ​​of the effective deposition area are mapped to a color gradient to generate a spatial distribution heatmap of the pesticide deposition intensity. Further, the pesticide coverage rate at the canopy scale and the deposition uniformity index are plotted as a bar chart. Finally, the text data summary, the overlay display map, the spatial distribution heatmap, and the bar chart are formatted and combined according to a preset report template to output a comprehensive report on the fruit tree spraying effect of the agricultural drone, containing text, data, and visualization charts. This report is used to quantitatively and objectively evaluate the effect of this spraying operation. Other methods can also be used in other embodiments, which are not limited here.

[0073] It should be noted that the above steps can achieve adaptive fusion of temporal difference feature map and spectral enhancement feature map based on local signal-to-noise ratio, thereby intelligently making full use of the dual information advantages of the time dimension and spectral dimension to generate a comprehensive feature map with more significant features and stronger anti-interference ability, which significantly improves the robustness and accuracy of accurately segmenting the effective area of ​​pesticide deposition from complex orchard scenes.

[0074] In another aspect, in some embodiments, this application provides an image recognition-based device for detecting the spraying effect of agricultural drones on fruit trees, referring to... Figure 4 The figure is a schematic diagram of the structure of an image recognition-based agricultural drone fruit tree spraying effect detection device according to some embodiments of this application. The image recognition-based agricultural drone fruit tree spraying effect detection device includes: a data acquisition module 401, a processing module 402, and an execution module 403, which are described below: The acquisition module 401 in this application is mainly used to acquire a multispectral image set of vegetation before spraying by the plant protection drone and a multispectral image set of vegetation after spraying. The pesticide solution used in the spraying operation is premixed with an inert tracer that has a characteristic reflection peak in the short-wave infrared band. Processing module 402, in this application, is mainly used to perform sub-pixel-level image registration based on feature point matching on the multispectral image set of the vegetation before spraying and the multispectral image set of the vegetation after spraying. Then, for the registered image pair, differential operation is performed between the reflectance images of the same time band before and after spraying to generate a temporal differential feature map that highlights the changes in the canopy. The processing module 402 described in this application is also used to extract the characteristic bands of the tracer and the reflectance of the low-reflectance reference band of the vegetation from the multispectral image set of the sprayed vegetation, and then generate a spectral enhancement feature map that highlights the deposition of the pesticide solution. The processing module 402 described in this application is further configured to perform pixel-level adaptive weighted fusion based on the signal-to-noise ratio of the temporal difference feature map and the spectral enhancement feature map to generate a comprehensive feature map, and then extract the effective area of ​​drug deposition from the comprehensive feature map; The execution module 403 in this application is mainly used to determine the pesticide coverage rate at the canopy scale based on the pixel ratio of the effective area to the entire canopy area, and to determine the deposition uniformity index based on the spatial statistical distribution of pixel values ​​within the effective area, thereby generating a test report on the fruit tree spraying effect of the plant protection drone.

[0075] Each module in the aforementioned image recognition-based agricultural drone fruit tree spraying effect detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0076] In another embodiment, this application provides a computer device, which may be a server, and its internal structure diagram may be as follows. Figure 5 As shown, the computer device includes a processor, memory, and network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data on the effectiveness of fruit tree spraying by image recognition-based agricultural drones. The network interface communicates with external terminals via a network connection. When the processor executes the computer program, it implements a method for detecting the effectiveness of fruit tree spraying by image recognition-based agricultural drones.

[0077] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0078] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above embodiment of the method for detecting the effect of fruit tree spraying by an agricultural drone based on image recognition.

[0079] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps described in the above embodiment of the method for detecting the effect of fruit tree spraying by an image recognition-based agricultural drone.

[0080] In one embodiment, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps described in the embodiment of the image recognition-based method for detecting the effect of spraying fruit trees by an agricultural drone.

[0081] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0082] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0083] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for detecting the spraying effect of agricultural drones on fruit trees based on image recognition, characterized in that, Includes the following steps: Multispectral images of vegetation were collected before and after spraying by agricultural drones. The pesticide solution used in the spraying operation was premixed with an inert tracer that has a characteristic reflection peak in the short-wave infrared band. Subpixel-level image registration based on feature point matching is performed on the multispectral image set of vegetation before spraying and the multispectral image set of vegetation after spraying. Then, differential operation is performed on the reflectance images of the same temporal band before and after spraying on the registered image pairs to generate a temporal differential feature map that highlights the changes in the canopy. The characteristic bands of the tracer and the reflectance of the low-reflectance reference band of the vegetation are extracted from the multispectral images of the vegetation after spraying, and then a spectral enhancement feature map that highlights the deposition of the pesticide solution is generated. Pixel-level adaptive weighted fusion is performed based on the signal-to-noise ratio of the temporal difference feature map and the spectral enhancement feature map to generate a comprehensive feature map, and then the effective area of ​​drug deposition is extracted from the comprehensive feature map; The pesticide coverage rate at the canopy scale is determined based on the pixel ratio of the effective area to the entire canopy area, and the deposition uniformity index is determined based on the spatial statistical distribution of pixel values ​​within the effective area, thereby generating a test report on the spraying effect of the plant protection drone on fruit trees.

2. The method as described in claim 1, characterized in that, The sub-pixel-level image registration based on feature point matching for the multispectral image sets of the vegetation before and after spraying specifically includes: The images in the multispectral image set of the vegetation before spraying are paired with the images in the multispectral image set of the vegetation after spraying that correspond to the geographical locations, to obtain multiple pairs of images to be registered. For each pair of images to be registered, scale-invariant feature transform points are extracted from the multispectral images before and after spraying, and initial matching is performed. Based on the random sampling consensus algorithm, the geometric transformation model from the post-spraying multispectral image to the pre-spraying multispectral image is estimated from the initially matched feature point pairs, and mismatched feature point pairs are eliminated. The multispectral image after spraying is resampled by bilinear interpolation according to the geometric transformation model, so as to align the multispectral image after spraying with the multispectral image before spraying at the subpixel level to obtain a registered image pair.

3. The method as described in claim 1, characterized in that, For the registered image pairs, a difference operation is performed between reflectance images of the same time phase band before and after spraying to generate a temporal difference feature map highlighting canopy changes. Specifically, this includes: From the registered image pairs, select the band reflectance of the same time phase that is sensitive to changes in vegetation canopy; The initial difference image is generated by subtracting the band reflectance of the multispectral image before spraying from the band reflectance of the multispectral image after spraying pixel by pixel. The initial difference image is processed using a mean filtering algorithm to remove shot noise; The filtered image is used as a temporal difference feature map to highlight canopy variations.

4. The method as described in claim 1, characterized in that, Extracting the reflectance of the tracer's characteristic bands and the vegetation's low-reflectance reference bands from the multispectral images of the vegetation after spraying specifically includes: The specific band number corresponding to the characteristic reflection peak of the inert tracer in the short-wave infrared band is determined and used as the characteristic band; The red light band, where vegetation chlorophyll has strong absorption characteristics, was identified as the low-reflection reference band. From each image in the multispectral image set of the post-spray vegetation, the reflectance of the characteristic band and the reflectance of the low-reflectance reference band are read respectively.

5. The method as described in claim 1, characterized in that, The generation of spectral enhancement feature maps highlighting drug deposition specifically includes: For each image in the multispectral image set of the sprayed vegetation, a pixel-by-pixel normalized difference index calculation is performed based on the reflectance of the feature band and the reflectance of the low reflectance reference band extracted from the image to obtain the index value of each pixel. The calculated exponent values ​​of all pixels are combined into a single-channel image; The single-channel image was used as a spectral enhancement feature map to highlight drug deposition. The pixel regions with higher index values ​​correspond to regions where the reflectance of the feature band is significantly higher than that of the reference band, i.e., suspected drug deposition regions.

6. The method as described in claim 1, characterized in that, Pixel-level adaptive weighted fusion is performed based on the signal-to-noise ratio of the temporal difference feature map and the spectral enhancement feature map to generate a comprehensive feature map, specifically including: The temporal difference feature map and the spectral enhancement feature map are normalized respectively to make the pixel value range of the two feature maps uniform; Calculate the signal-to-noise ratio estimates of the temporal difference feature map and the spectral enhancement feature map within the local window, respectively; Based on the signal-to-noise ratio estimate of the two feature maps at each pixel location, the fusion weights of the temporal difference feature map and the spectral enhancement feature map at that pixel location are dynamically allocated; Based on the assigned fusion weights, the temporal difference feature map and the spectral enhancement feature map are subjected to pixel-level weighted summation to generate a comprehensive feature map.

7. The method as described in claim 1, characterized in that, The effective region for drug deposition extracted from the comprehensive feature map specifically includes: The comprehensive feature map is subjected to adaptive threshold segmentation to initially segment connected regions with pixel values ​​higher than those of the background region; Morphological opening operations are performed on the connected regions obtained from the initial segmentation to eliminate noise points with excessively small areas; Perform morphological closing operations on connected regions that have undergone opening operations to fill small holes inside the regions; The connected regions after morphological opening and closing operations are used as the effective regions for drug deposition.

8. A device for detecting the spraying effect of agricultural drones on fruit trees based on image recognition, characterized in that, include: The acquisition module is used to acquire multispectral image sets of vegetation before and after spraying by agricultural drones. The pesticide solution used in the spraying operation is premixed with an inert tracer that has a characteristic reflection peak in the short-wave infrared band. The processing module is used to perform sub-pixel-level image registration based on feature point matching on the multispectral image set of the vegetation before spraying and the multispectral image set of the vegetation after spraying. Then, it performs differential operation on the reflectance images of the same time band before and after spraying on the registered image pairs to generate a temporal differential feature map that highlights the changes in the canopy. The processing module is also used to extract the characteristic bands of the tracer and the reflectance of the low-reflectance reference band of the vegetation from the multispectral image set of the sprayed vegetation, and then generate a spectral enhancement feature map that highlights the deposition of the pesticide solution. The processing module is further configured to perform pixel-level adaptive weighted fusion based on the signal-to-noise ratio of the temporal difference feature map and the spectral enhancement feature map to generate a comprehensive feature map, and then extract the effective area of ​​drug deposition from the comprehensive feature map; The execution module is used to determine the pesticide coverage rate at the canopy scale based on the pixel ratio of the effective area to the entire canopy area, and to determine the deposition uniformity index based on the spatial statistical distribution of pixel values ​​within the effective area, thereby generating a report on the fruit tree spraying effect of the plant protection drone.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method for detecting the spraying effect of plant protection drones on fruit trees based on image recognition, as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for detecting the spraying effect of plant protection drones on fruit trees based on image recognition as described in any one of claims 1 to 7.

Images