Apple yield prediction method based on unmanned aerial vehicle and satellite multispectral differential fusion

Abstract

The invention discloses an apple yield prediction method based on unmanned aerial vehicle and satellite multispectral differential fusion. The method comprises the following steps: acquiring an unmanned aerial vehicle multispectral image and a satellite image of an apple tree in a spring shoot stopping long period; apple tree crown spectral information in the unmanned aerial vehicle orchard multispectral image and the satellite image is extracted; differential fusion is carried out on the sensitive spectrum variable in the unmanned aerial vehicle multispectral image and the spectrum variable in the satellite image to obtain an unmanned aerial vehicle-satellite differential fusion spectrum variable; and constructing an apple yield prediction model through unmanned aerial vehicle-satellite differential fusion spectrum variables for apple yield prediction. An unmanned aerial vehicle multispectral image and a satellite image are combined to obtain accurate fruit tree canopy spectral information, and the influence of background information in the unmanned aerial vehicle image is reduced, so that sensitive spectral variables related to the apple yield are screened out from the spectral information to obtain differential fusion spectral variables; a more accurate yield prediction model can be constructed and screened out for apple yield prediction, and the accuracy of apple yield prediction is improved.

Description

Technical Field

[0001] This application relates to the technical field of apple yield prediction, and particularly relates to an apple yield prediction method based on multi-spectral differential fusion of drones and satellites. Background Art

[0002] The information management level of some orchards is relatively low, lacking large-area and real-time apple yield prediction technology, which leads to problems such as planting decision-making mistakes, increased market risks, and industrial chain fluctuations.

[0003] Currently, the main means of fruit tree yield prediction is machine vision technology, but this method is only applicable to the yield prediction of single fruit trees and is difficult to achieve rapid prediction of fruit tree yields at the regional scale. Satellite remote sensing technology has the ability to obtain spectra over a large range, providing new possibilities for rapid non-destructive prediction of apple yields at the regional scale. The fruit tree yield prediction models constructed using satellite remote sensing technology can be roughly divided into two types: "crop growth models" and "empirical models". Rahman et al. used high-resolution WorldView-3 (WV3) satellite images during the critical growth period of mangoes to extract the canopy spectral reflectance and crown area of mango trees, and used an artificial neural network method to estimate the mango yield in Australia. Bai et al. based on long-term Planet multi-spectral images, extracted and calculated the integrated data of the canopy vegetation index of single apple trees, and used the random forest algorithm to achieve the prediction of apple yields at the township scale. Using high-resolution satellite remote sensing images, although relatively accurate fruit tree canopy spectra can be extracted, the image cost is relatively high, restricting its popularization and application in daily agricultural situation monitoring. Sentinel-2 satellite remote sensing images are outstanding in monitoring the growth status of vegetation due to their advantages such as high revisit period, open-source acquisition, and sensitive red-edge monitoring of vegetation. However, due to the limitation of spatial resolution, the pixels often contain orchard interference information such as soil and shadows, weakening the spectral signal of the fruit tree canopy and reducing the prediction accuracy of fruit tree yields. Therefore, how to apply Sentinel-2 images to extract accurate fruit tree canopy spectral information to improve the prediction accuracy of apple yields at the county level is an urgent problem to be solved.

[0004] To improve the accuracy of vegetation spectral information extraction, current research can be roughly divided into three methods. The first method is to separate vegetation signals from non-vegetation signals. For example, different bands of remote sensing images are subjected to mathematical operations such as ratio and normalization to construct vegetation indices, so as to distinguish vegetation from non-vegetation information and achieve the purpose of accurately extracting vegetation spectra. This method is simple and fast, and can effectively distinguish vegetation from non-vegetation, which is suitable for large-area remote sensing images. However, it is prone to overfitting in high-density vegetation areas, and when the pixel contains multiple ground object information, the vegetation spectrum will be greatly interfered. To solve the problem of mixed pixels, some scholars combine vegetation indices with mixed pixel decomposition methods to improve the accuracy of vegetation spectral information extraction through linear and non-linear mixed pixel decomposition models. However, due to the need for a high-quality spectral library as a reference for the mixed pixel decomposition model and the complex model calculation, its application in many scenarios is limited. The second method is to use machine learning or deep learning algorithms to automatically extract the spectral features of vegetation. This method can automatically extract vegetation information from complex spectral data based on the input samples and performs excellently in processing large-scale data sets. However, this method requires a large amount of labeled data for training, which is time-consuming and the interpretability of the model is poor. The third method is to use high-resolution spectral data to correct or fuse satellite remote sensing data. Zhang et al. used the average reflectance values of each band of UAV spectral data simulated by near-surface hyperspectral data to correct the spectral information of wheat in Sentinel-2 and drew a county-level wheat SPAD prediction map. This method is relatively simple and can achieve rapid correction of satellite spectral information in the case of homogeneous distribution and small differences between samples. Due to the influence of factors such as tree age and nutrient status on different apple trees, the spectral information presented is different, resulting in difficulty in accurately correcting the spectral data of different apple trees with the average reflectance value of the samples. The NMF algorithm can realize dimensionality reduction and feature extraction of the matrix, and then obtain the correction coefficient of the samples based on the differences between the samples. Chen et al. applied NMF to decompose the spectral variables in near-surface hyperspectral data and Sentinel-2 data respectively. Then, the basis matrix obtained from the near-surface hyperspectral data was multiplied by the correction coefficient in Sentinel-2 data to reconstruct the spectral variables. This fusion method improves the accuracy of Sentinel-2 data in land feature monitoring and highlights the potential of NMF in integrating multi-source spectral data sets.

[0005] Compared with near-earth hyperspectral sensors, low-altitude UAV sensors are more flexible and can provide ultra-high-resolution spectral images of fruit tree canopies, capturing the spatial variability and health status of apple tree canopies in more detail. However, there is currently little research on fusing UAV multispectral data with satellite spectral data for regional apple yield prediction. Moreover, when the direct solar direction is inconsistent with the UAV sensor direction, there are often shadows inside the fruit tree canopy, weakening the spectral signal of the fruit tree canopy. At the same time, the planting intervals of fruit trees result in the inclusion of background information such as soil in UAV images, which will further interfere with the accurate extraction of UAV fruit tree canopy spectral information. Summary of the Invention

[0006] To solve the above technical problems, the present application proposes the following technical solutions:

[0007] In a first aspect, an apple yield prediction method based on differential fusion of UAV and satellite multispectral data provided by an embodiment of the present application includes:

[0008] Obtain UAV multispectral images and satellite images of apple trees during the spring shoot cessation period;

[0009] Extract the spectral information of apple tree canopies from UAV orchard multispectral images and satellite images;

[0010] Perform differential fusion of sensitive spectral variables in the UAV multispectral image with spectral variables in the satellite image to obtain UAV-satellite differential fusion spectral variables;

[0011] Construct an apple yield prediction model through the UAV-satellite differential fusion spectral variables to predict apple yield.

[0012] In a possible implementation manner, the obtaining of UAV multispectral images and satellite images of apple trees during the spring shoot cessation period includes:

[0013] Use a UAV to carry a multispectral camera, and perform image mosaicking and radiometric calibration on the captured images to obtain UAV multispectral images;

[0014] Select satellite images in a preset time period before and after the acquisition of UAV images, and perform screening, mosaicking, and cropping on the satellite images to obtain true-color satellite images.

[0015] In a possible implementation manner, the extraction of the spectral information of apple tree canopies from the UAV orchard multispectral image includes:

[0016] First, use the threshold method to determine the optimal threshold of the normalized difference canopy shadow index NDCSI;

[0017] Based on the optimal threshold, use Python to remove the orchard background of the UAV multispectral image of the experimental plot;

[0018] The ground objects in the apple orchard UAV images are classified into three categories: fruit tree canopies, bare soil, and shadows. The ENVI classification accuracy verification tool is used to evaluate the accuracy of the orchard background removal result under the optimal threshold of NDCSI;

[0019] Finally, the spectral information of the apple tree canopies is extracted from the UAV images after removing the orchard background in ENVI.

[0020] In a possible implementation, the obtaining of the UAV-satellite differential fusion spectral variables by differentiating and fusing the sensitive spectral variables in the UAV multispectral images and the spectral variables in the satellite images includes:

[0021] Respectively extract the sensitive spectral variables for estimating apple yield in the UAV multispectral images and the satellite images;

[0022] Use non-negative matrix factorization to fuse the sensitive spectral variables extracted from the UAV multispectral images and the satellite images to obtain the UAV-satellite differential fusion spectral variables.

[0023] In a possible implementation, the respectively extracting the sensitive spectral variables for estimating apple yield in the UAV multispectral images and the satellite images includes:

[0024] Perform spectral transformations on the spectral image data such as 1 / (R i +R j ), 1 / (R i ―R j ), R i +R j , R i ·R j , R i / (R i +R j ), R i / (R i ―R j ), and R i ·R j ·R q where: R i , R j , R q are any of the five bands respectively;

[0025] Respectively analyze the relationships between the spectral variables of different sensors and apple yield;

[0026] Use the correlation coefficient method to preliminarily screen the sensitive spectral variables of apple yield, and then use the Cars algorithm for further screening.

[0027] In a possible implementation, the step of fusing the sensitive spectral variables extracted from the UAV multispectral image and the satellite image by using non-negative matrix factorization to obtain the UAV-satellite differential fusion spectral variables includes:

[0028] First, represent the sensitive spectral variables as a non-negative matrix V of n×m, where n represents the number of bands or variables, and m represents the number of samples;

[0029] Given a positive integer r, where r < min(n, m), according to the NMF algorithm, decompose the two sets of sensitive spectral variables into two non-negative matrices, namely W ∈ R n×r , H ∈ R r×m , such that V ≈ WH, where r represents the number of decompositions, that is, the number of main sources in matrix V, W ∈ R n×r represents the spectral feature matrix, and each column represents the spectral variables of a main source, that is, the basis matrix, H ∈ R r×m is the abundance matrix, and each column represents the fraction of the main source in the sample;

[0030] Use the NMF algorithm to determine the correction coefficient H of the sensitive spectral variables of the satellite image MSISV ;

[0031] Decompose the sensitive spectral variables of the UAV multispectral image to obtain the basis spectral matrix W UAVSVs ;

[0032] Adopt the NMF algorithm combining non-negative least squares NNLS and projection gradient method to alternately decompose the sensitive spectral variables of the UAV and the sensitive spectral variables of the satellite until the optimal W UAVSVs and H MSISVs ;

[0033] Multiply the optimal W UAVSVs and H MSISVs obtained by decomposition to get the UAV-satellite differential fusion spectral variables.

[0034] In a possible implementation, during the decomposition process, the decomposition number r determines the size of the data volume and the approximation degree to the original data. The peak signal-to-noise ratio PSNR is used as an evaluation parameter to determine r, and the specific calculation method is as follows:

[0035]

[0036] In the formula, i is the main factor, MAX i is the maximum spectral input of i, N is the number of samples, and i, k are the values of the main factor i of sample k.

[0037] In a possible implementation, the step of constructing an apple yield prediction model through the UAV-satellite differential fusion spectral variables includes:

[0038] Randomly divide the collected samples into a modeling set and a validation set according to a ratio of 2:1.

[0039] Use PLSR, SVM, RF, and BPNN algorithms to construct multiple apple yield prediction models by combining the UAV-satellite differential fusion spectral variables.

[0040] Use R 2 , nRMSE, and RPD to evaluate the accuracy of the apple yield prediction models and select the best apple yield prediction model.

[0041] In a possible implementation, using the apple yield prediction model for apple yield prediction includes:

[0042] Obtain a predicted apple yield sensitive spectral variable map using satellite images.

[0043] Construct a conversion variable for the apple yield sensitive spectral variables before and after differential fusion to generate a differential fusion apple yield sensitive spectral variable map.

[0044] Finally, input the fused apple yield sensitive spectral variable map into the best apple yield prediction model to draw an apple yield prediction map for the prediction area.

[0045] In the embodiments of the present application, by combining UAV multispectral images and satellite images to obtain accurate spectral information of the fruit tree canopy, the influence of background information in the UAV images is reduced. In this way, sensitive spectral variables related to apple yield can be screened out from the spectral information to obtain differential fusion spectral variables, and a more accurate yield prediction model can be constructed. Finally, the optimal yield prediction model is selected to predict the apple yield, improving the accuracy of apple yield prediction. Description of the Drawings

[0046] Figure 1 It is a schematic flowchart of a method for predicting apple yield based on UAV and satellite multispectral differential fusion provided by the embodiments of the present application;

[0047] Figure 2 It is the UAV multispectral image of the test plot provided by the embodiments of the present application;

[0048] Figure 3 It is a comparison diagram of the calculation of NDVI and NDCSI of the UAV image provided by the embodiments of the present application;

[0049] Figure 4 It is a schematic diagram for determining the orchard background removal threshold of the NDCSI UAV image provided by the embodiments of the present application;

[0050] Figure 5Schematic diagram of the comparison of the spectral characteristics of the fruit tree canopy extracted from the pre- and post-background-removed images of the drone provided in the embodiment of the present application and the Sentinel-2 image before fusion;

[0051] Figure 6 Schematic diagram of the correlation between the spectral variables of different remote sensing data and the apple yield provided in the embodiment of the present application;

[0052] Figure 7 Schematic diagram of the change of the peak signal-to-noise ratio with the decomposition number r provided in the embodiment of the present application;

[0053] Figure 8 Schematic diagram of the comparison of the correlation between the spectral variables before and after differential fusion and the apple yield provided in the embodiment of the present application;

[0054] Figure 9 Schematic diagram of the comparison of the accuracies of different models before and after the fusion of the spectral variables of the drone-satellite provided in the embodiment of the present application;

[0055] Figure 10 Schematic diagram of the evaluation result of the feature importance for the identification of apple trees provided in the embodiment of the present application;

[0056] Figure 11 Schematic diagram of the spatial distribution of the apple orchard yield in the study area provided in the embodiment of the present application;

[0057] Figure 12 Schematic diagram of the statistical results of the apple yields of the yield estimation units in different experimental plots provided in the embodiment of the present application. Detailed implementation manners

[0058] The following elaborates on this solution in combination with the accompanying drawings and the detailed implementation manners.

[0059] Refer to Figure 1 , the apple yield prediction method based on the differential fusion of the multi-spectral data of the drone and the satellite provided in this embodiment includes:

[0060] S101, obtaining the multi-spectral images of the drone and the satellite images during the period when the spring shoots of the apple trees stop growing.

[0061] Before obtaining the image data, first determine the study area. The study area for obtaining the image data in this embodiment is located in Qixia City, Yantai, Shandong Province (37°05′-37°30′N, 120°33′-121°16′E), with a total area of 1,793.25 square kilometers. It is one of the famous apple-producing areas in China. It has a warm temperate monsoon climate, with distinct seasons and a relatively large temperature difference between day and night, which is conducive to the accumulation of sugar in the fruit trees. The soil is brown soil with good fertility and is suitable for the growth of fruit trees. To ensure the comprehensiveness and representativeness of the sampling data, five experimental plots are set up in the east, west, south, and north of the study area to conduct experiments.

[0062] The spring shoot growth cessation period (from the end of May to the beginning of July) is a crucial period for the nutrient transformation of apple trees. The number of fruits has been basically determined, which is closely related to the yield during the apple tree harvest period. Therefore, in this embodiment, the multi-spectral images of unmanned aerial vehicles (UAVs) and Sentinel-2 images during the spring shoot growth cessation period of apple trees are used as data sources.

[0063] Select the time in mid-June. Use a DJI Matrix 300 RTK UAV (DJI Technology Co., Ltd., Shenzhen, China) equipped with an MS600 PRO multi-spectral sensor to obtain the multi-spectral images of the experimental plot. The MS600 PRO camera has six bands, including blue, green, red, red edge 1, red edge 2, and near-infrared bands. The maximum endurance of the UAV is 55 minutes, the maximum payload is 2.7 kg, the flight speed is set at 7 m / s, the flight altitude is set at 100 m, and the spatial resolution is 0.06 m. The side overlap rate for each flight is set at 75%.

[0064] Before the UAV flight, use the MS600 PRO camera to take gray board images for radiometric calibration. Import the collected data into Pix4Dmapper software for image mosaicking and radiometric calibration. The final obtained UAV multi-spectral images of the experimental plot are as Figure 2 shown.

[0065] In this study, Sentinel-2 images 10 days before and after the UAV image acquisition are selected, and the images are screened, mosaicked, and cropped using the GEE platform to generate the true color Sentinel-2 images of the apple trees in the study area during the spring shoot growth cessation period. The band information of the UAV sensor and Sentinel-2 is shown in Table 1. Since the B3, B4, B5, B6, and B8 bands of Sentinel-2 are consistent with the UAV image bands, these bands are applied to the construction and screening of subsequent apple yield-sensitive spectral variables.

[0066] Table 1 Comparison of UAV and Sentinel-2 band information

[0067]

[0068] In this embodiment, relying on the GEE platform, Sentinel-2 images of the study area in April, May, and October 2023 are screened. Combining spectral features such as vegetation indices and texture features, the best feature combination for apple tree identification is screened using the Out-Of-Bag error (OOB) algorithm of the random forest, and the apple orchards in the study area are extracted using the random forest classification algorithm. The classification results are verified using the Kappa coefficient and confusion matrix. The apple orchard identification features used in this embodiment are shown in Table 2.

[0069] Table 2 Apple orchard identification features

[0070]

[0071] S102, extract the spectral information of apple tree canopies from the multi - spectral images of drones in the orchard and satellite images.

[0072] In this embodiment, in order to accurately extract the spectral information of the fruit tree canopy in the drone images, the optimal threshold of NDCSI is first determined by the threshold method; subsequently, based on the optimal threshold, the orchard background in the multi - spectral images of drones in the experimental plot is removed using Python. To verify the accuracy of the NDCSI threshold, the ground objects in the drone images of the apple orchard are divided into three categories: fruit tree canopy, bare soil, and shadow. The accuracy of the orchard background removal result under the optimal NDCSI threshold is evaluated using the ENVI classification accuracy verification tool. Finally, the spectral information of apple tree canopies is extracted from the drone images after removing the orchard background in ENVI and applied to the construction and screening of subsequent apple yield - sensitive spectral variables. As Figure 3 shown, (A) is the true - color composite image of the drone, (B) is the calculation result of NDVI, and (C) is the calculation result of NDCSI. It can be seen that NDCSI can more effectively distinguish the apple tree canopy from the orchard background.

[0073] Use the threshold method with NDCSI to remove the orchard background in the multi - spectral images of drones, as Figure 4 shown, Figure 4 in which (A) is the original multi - spectral image of a single fruit tree, (B) is the calculation result of NDCSI of a single fruit tree, (C) is the extraction result of the single - fruit - tree canopy when the threshold is 0.40, (D) is the extraction result of the single - fruit - tree canopy when the threshold is 0.45, and (E) is the extraction result of the single - fruit - tree canopy when the threshold is 0.5. Taking the single fruit tree O1 as an example, NDCSI can effectively highlight the fruit tree canopy part. When the NDCSI threshold is set to 0.4, although the overall shape of the canopy is retained and the soil and some shadows are removed, there are still some shadows that are difficult to remove; when the NDCSI threshold is set to 0.5, although the soil and most of the shadows are effectively removed, the fruit tree canopy is too fragmented; when the NDCSI threshold is set to 0.45, most of the orchard soil and shadows within the canopy are removed, and the overall shape of the fruit tree canopy is retained. Therefore, 0.45 is set as the threshold of O1 NDCSI. Finally, the threshold of O2 is determined to be 0.43, the threshold of O3 is 0.48, the threshold of O4 is 0.43, and the threshold of O5 is 0.45.

[0074] To verify the reliability of the orchard background removal results, the orchard background removal results at the optimal NDCSI threshold were evaluated. The results showed that the accuracy of orchard background removal for all UAV images reached 98%. Subsequently, based on the optimal threshold, a mask was established to remove the orchard background from all UAV images of the experimental plots, and the spectral information of the fruit tree canopies in the yield estimation units was extracted.

[0075] Figure 5 Figure 4 shows the spectral reflectance curves of apple tree canopies before and after background removal extracted from UAV images, as well as the spectral reflectance curves of apple tree canopies extracted from Sentinel-2 images. After applying the NDCSI method to remove the background, the spectral reflectance values of the apple tree canopies captured by the UAV increased significantly, especially in the red-edge2 and Nir bands. Among them, the reflectance in the red-edge2 band increased by 0.09, and the reflectance in the Nir band increased by 0.08. These results indicate that the interference of the background on the red-edge and near-infrared bands is more significant. Table 3 presents the correlation comparison between each UAV band and apple yield before and after NDCSI background removal. After applying the NDCSI method, the correlation between the UAV bands and apple yield increased significantly. Among them, the Nir band and the red-edge2 band had the largest increases, and the correlation coefficients of the two bands increased by 0.142 and 0.128, respectively. These results indicate that applying NDCSI to remove background interference can obtain more accurate spectral data of apple tree canopies. Based on the UAV images after background removal, the spectral information of apple tree canopies was extracted to construct spectral variables.

[0076] Comparing the spectral information of apple tree canopies extracted from UAV and Sentinel-2 images shows that the trends of the spectral information of apple tree canopies in the two images are generally the same. Comparing the spectral reflectance values of the two data sets, the results show that except for the red band, the spectral reflectance values of other bands in the UAV images are higher than those in the Sentinel-2 image data.

[0077] Table 3 Correlation comparison between each band and apple yield before and after removing the orchard background from UAV images using NDCSI

[0078]

[0079] S103, the sensitive spectral variables in the UAV multispectral images were differentially fused with the spectral variables in the satellite images to obtain the UAV-satellite differential fusion spectral variables.

[0080] After separately extracting the spectral information of apple tree canopies in the yield estimation units from the UAV multispectral images and Sentinel-2 images. The spectral data was processed with 1 / (R i +R j ) and 1 / (R i ―R j ), Ri +R j ,R i ·R j ,R i / (R i +R j ),R i / (R i ―R j ) and R i ·R j ·R q Spectral transformation, where: R i , R j , R q are any bands in five bands respectively. Analyze the relationships between spectral variables of different sensors and apple yield respectively, and initially screen the sensitive spectral variables of apple yield by using the correlation coefficient method, and then further screen by using the Cars algorithm.

[0081] The finally screened sensitive spectral variables of apple yield are Red-edge1-R, G+R, 1 / (G+R), R / (Red-edge2-R), Red-edge1 / (Red-edge2-G), Red-edge2 / (Nir-R), and The correlation results between the screened sensitive spectral variables and apple yield are as Figure 6 shown. Figure 6 In (A), the spectral variables are constructed from UAV spectral data, and in (B), the spectral variables are constructed from Sentinel-2 spectral data before fusion. Figure 6 In it, V1-V8 are the sensitive spectral variables Red-edge1-R, G+R, 1 / (G+R), R / (Red-edge2-R), Red-edge1 / (Red-edge2-G), Red-edge2 / (Nir-R),

[0082] The results show that the absolute value of the correlation coefficient between the UAV spectral variables and apple yield is between 0.750 and 0.768, while the absolute value of the correlation coefficient between the Sentinel-2 spectral variables and apple yield is between 0.629 and 0.643. Compared with Sentinel-2, the correlation between the sensitive spectral variables screened by UAV spectral data and apple yield is 0.109-0.132 higher. Therefore, the NMF algorithm is used to decompose the spectral variables of the two remote sensing data, extract the basis matrix of the UAV spectral variables with higher correlation, and multiply it by the weight matrix of the satellite spectral variables to generate the sensitive spectral variables of apple yield after star-air differential fusion.

[0083] The non-negative matrix factorization (NMF) algorithm is used to differentially fuse the UAV spectral variables (UAVSVs) after background information removal with the Sentinel-2 spectral variables (MSISVs) to improve the Sentinel-2 apple yield prediction accuracy. The basic principle of NMF is that if all elements in a matrix are non-negative, the original matrix can be decomposed into two non-negative matrices through NMF, and their product can approximate the input matrix, thus generating a part-based representation. Therefore, as long as the spectral variables of apple yield are non-negative matrices, this method can decompose them.

[0084] First, the sensitive spectral variables of apple yield are represented as a non-negative matrix V of n×m, where n represents the number of bands or variables, and m represents the number of samples. Given a positive integer r (r < min(n, m)), according to the NMF algorithm, the two sets of spectral variables can be decomposed into two non-negative matrices respectively, that is, W ∈ R n×r , H ∈ R r×m , such that V ≈ WH. Here, r represents the number of decompositions, that is, the number of main sources in matrix V. W ∈ R n×r represents the spectral feature matrix, and each column represents the spectral variables of a main source, that is, the basis matrix; H ∈ R r×m is the abundance matrix, and each column represents the fraction of the main source in the sample. Therefore, H ∈ R r×m can be used as the weight value of the sample, that is, the correction coefficient of the sample. V can be regarded as a linear combination of each column of matrix W with H as the weight factor.

[0085] The correction coefficient H of the Sentinel-2 spectral variables is determined using the NMF algorithm MSISV . Since the UAV sensor is less affected by the atmosphere and has a higher spatial resolution than the satellite sensor, the decomposed W UAVSVs is used as the basis spectral matrix to participate in the fusion. The NMF algorithm combining the non-negative least squares (NNLS) and the projection gradient method is used to alternately decompose the UAV spectral variables and the satellite spectral variables until the optimal W UAVSVs and H MSISVs are obtained. The optimal W UAVSVs and H MSISVs obtained by decomposition are multiplied to obtain the spectral variables after satellite-UAV differential fusion.

[0086] During the decomposition process, the decomposition number r determines the size of the data volume and the approximation degree to the original data. Therefore, the peak signal-to-noise ratio (PSNR) is used as an evaluation parameter to determine the decomposition number r. Generally, the higher the PSNR, the smaller the image distortion. The specific calculation method is as follows:

[0087]

[0088] In the formula, i is the main factor, MAX i is the maximum spectral input of i, N is the number of samples, and i,k is the value of the main factor i of sample k.

[0089] Figure 7 shows the change trend of PSNR within different ranges of r values. When the r value is between 2 and 8, the PSNR value first gradually increases and then tends to converge. When r = 5, the PSNR value of the UAV spectral variable starts to decline, while the PSNR value of the Sentinel-2 spectral variable continues to rise, stabilizes at r = 7, and reaches the extreme value at r = 8. Based on this, to obtain a more accurate spectral fusion effect, the UAV spectral variable matrix is decomposed at r = 5 to extract the basic matrix, and the Sentinel-2 spectral variable is decomposed at r = 8 to extract the weight matrix. The two matrices are multiplied to achieve the differential fusion of the satellite-UAV spectral variables.

[0090] To evaluate the effect of the satellite-UAV differential fusion algorithm, the correlations between the spectral variables and apple yield before and after differential fusion were compared and analyzed, as Figure 8 shown. The results show that the correlations between the sensitive spectral variables after fusion and apple yield are all higher than those before fusion, and all reach above 0.7. Among them, the correlation between V1 and apple yield after fusion increased by 0.13, V5 increased by 0.1, and V6 increased by 0.09. It shows that the satellite-UAV spectral variable differential fusion can enhance the correlation between the spectral variables and apple yield.

[0091] S104, an apple yield prediction model is constructed through the UAV-satellite differential fusion spectral variables to predict the apple yield.

[0092] In this embodiment, October, i.e., the fruit maturity period of apple trees, is selected. 153 yield estimation units of 10m×10m are randomly selected in five experimental plots in the study area, and the number of fruits on each fruit tree in each yield estimation unit is counted. On the fruit trees at the four sides and the center position of each yield estimation unit, 8 apples are randomly picked, and a total of 40 apples are collected. They are weighed on-site using an electronic scale to calculate the average single fruit weight of the yield estimation unit. Based on the total number of fruits and the average single fruit weight of the yield estimation unit, the apple yield of the yield estimation unit is calculated. In the same way as above, in October of the following year, 73 yield estimation units are randomly selected in five experimental plots in the study area to obtain apple yield data for validating the yield prediction model.

[0093] Randomly divide the collected samples into a modeling set and a validation set according to a ratio of 2:1; use PLSR, SVM, RF, and BPNN algorithms to construct multiple apple yield prediction models by combining the UAV-satellite differential fusion spectral variables; use R 2 , nRMSE, and RPD to evaluate the accuracy of the apple yield prediction models and screen out the best apple yield prediction model.

[0094] Figure 9 Table 1 shows the accuracy evaluation results of the prediction models before and after the differential fusion of UAV-satellite spectral variables. The results show that the accuracy of the prediction models constructed by the spectral variables after differential fusion is higher than that before fusion. The RF model has the most significant improvement in accuracy. After fusion, the R 2 of the training set of this model increased by 0.1, the nRMSE decreased by 0.05, and the RPD increased by 0.41; the R 2 of the validation set increased by 0.12, the nRMSE decreased by 0.09, and the RPD increased by 0.64.

[0095] Comparing different modeling methods, the results show that the prediction models constructed by machine learning algorithms are all higher than the linear regression model. The best model for apple yield prediction is the RF model after the fusion of UAV-satellite spectral variables, and its scatter plot is shown in Figure 9 (E). The R 2 of the training set of this model is 0.81, the R 2 of the validation set is 0.84, the nRMSE is 0.14, and the RPD is 2.01. And most of the measured and predicted apple yield results of this model are distributed near the 1:1 line, indicating that this model has a good apple yield prediction effect. Use the apple yield data in 2024 to verify this model, and the obtained results are shown in Figure 9 (F). The results show that the R 2 of this model is 0.73, the nRMSE is 0.15, and the RPD is 1.68. It shows that this model has high accuracy and strong stability, and is suitable for apple yield prediction in different years.

[0096] Generate the spatial distribution map of apple yield in the study area based on the best apple yield prediction model. First, use Sentinel-2 images to calculate the apple yield sensitive spectral variable map in the study area; secondly, construct the conversion model of apple yield sensitive spectral variables before and after differential fusion to generate the apple yield sensitive spectral variable map after differential fusion; finally, input the fused apple yield sensitive spectral variable map into the best apple yield prediction model to draw the apple yield prediction map in the study area.

[0097] The statistical characteristics of apple yield of each prediction unit in the experimental plot are shown in Figure 10 . The highest recorded yield is 746.05 kg / m2 , the lowest recorded yield was 145.91 kg / m 2 . The average yield was 448.17 kg / m 2 , and the standard deviation was 148.46 kg / m 2 , and the coefficient of variation was 0.31. These results indicate that there are significant differences in apple yields among different prediction units, highlighting the urgent need for accurate apple yield prediction using remote sensing images.

[0098] Based on the Sentinel-2 images of the optimal phenological phase for apple tree identification, the apple tree identification features were extracted. The importance scores of the identification features were evaluated using the out-of-bag data error rate of the random forest, and the results are as Figure 11 shown. In images of different phenological phases, the optimal feature combinations for apple orchard identification are different. Based on the comprehensive features of all phenological phases, the top 10 features with higher importance scores were selected for combination, and the random forest algorithm was used to identify and extract apple orchards in the study area. The final classification Kappa coefficient was 85%, the producer accuracy (PA) was 78%, and the user accuracy (UA) was 85%.

[0099] Based on the extracted apple orchard distribution information, the RF apple yield prediction model established using the differential fusion spectral variables of unmanned aerial vehicle-satellite was used to draw the apple yield map of the study area. The correlation relationships of spectral variables before and after fusion are shown in Table 4. In Table 4, V1-V8 represent Red-edge1-R, G+R, 1 / (G+R), R / (Red-edge2-R), Red-edge1 / (Red-edge2-G), Red-edge2 / (Nir-R), and the correlation between the fused spectral variables and the corresponding unfused variables exceeds 0.6. Based on this, a linear regression transformation model between the two sets of spectral variables was constructed to generate the fused spectral variable map. Finally, the fused spectral variable map was input into the RF model to obtain the apple yield prediction map of Qixia City in 2023, as Figure 12 shown.

[0101] Table 4 Relationship between fused spectral variables and satellite spectral variables

[0102]

[0103] The differential fusion of unmanned aerial vehicle and satellite remote sensing data enhances the correlation between spectral variables and apple yields, thereby improving the prediction accuracy of the model. The correlation between the differential-fused spectral variables and apple yields increased by 0.06 - 0.13; the RF model constructed based on the differential-fused spectral variables is the best model for apple yield prediction, and this model has a higher accuracy than the R model before differential fusion. 2It has increased by 0.12, the nRMSE has decreased by 0.09, and the RPD has increased by 0.64. The cross-year validation of this model is carried out. R 2 has reached 0.73, indicating that the model has high precision and strong stability. The apple yield prediction map can provide a scientific basis and technical support for the reasonable planning of regional apple tree production and the improvement of orchard operation efficiency.

[0104] In the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships can exist. For example, A and / or B can represent the cases of A existing alone, A and B existing simultaneously, and B existing alone. Where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are an "or" relationship. "At least one of the following" and its similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, a - b, a - c, b - c, or a - b - c, where a, b, and c can be single or multiple.

[0105] As described above, the above are only the specific embodiments of the present application. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should all be covered by the protection scope of the present application. The protection scope of the present application shall be subject to the protection scope of the claimed rights.

Claims

1. An apple yield prediction method based on multi-spectral differential fusion of drones and satellites, characterized in that, Including: Obtaining multi - spectral images of an unmanned aerial vehicle (UAV) and satellite images during the spring shoot - stopping period of apple trees; Extracting the spectral information of apple tree canopies from the UAV orchard multi - spectral images and satellite images; Performing differential fusion on the sensitive spectral variables in the UAV multi - spectral images and the spectral variables in the satellite images to obtain UAV - satellite differential fusion spectral variables; Constructing an apple yield prediction model through the UAV - satellite differential fusion spectral variables to predict apple yield.

2. The method according to claim 1, characterized in that, The obtaining of multi - spectral images of an unmanned aerial vehicle and satellite images during the spring shoot - stopping period of apple trees includes: Using a UAV equipped with a multi - spectral camera, performing image mosaicking and radiometric calibration on the captured images to obtain UAV multi - spectral images; Selecting satellite images in a preset time period before and after the acquisition of UAV images, and performing screening, mosaicking and cropping on the satellite images to obtain true - color satellite images.

3. The method according to claim 1, wherein The extracting of the spectral information of apple tree canopies from the UAV orchard multi - spectral images includes: First, using the threshold method to determine the optimal threshold of the normalized difference canopy shadow index (NDCSI); Based on the optimal threshold, using Python to remove the orchard background from the UAV multi - spectral images of the experimental plots; Dividing the ground objects in the UAV images of the apple orchard into three categories: fruit tree canopies, bare soil and shadows, and using the ENVI classification accuracy verification tool to evaluate the accuracy of the orchard background removal results under the optimal threshold of NDCSI; Finally, extracting the spectral information of apple tree canopies from the UAV images after removing the orchard background in ENVI.

4. The method according to claim 3, wherein The performing of differential fusion on the sensitive spectral variables in the UAV multi - spectral images and the spectral variables in the satellite images to obtain UAV - satellite differential fusion spectral variables includes: Respectively extracting the sensitive spectral variables for estimating apple yield from the UAV multi - spectral images and satellite images; Using non - negative matrix factorization to fuse the sensitive spectral variables extracted from the UAV multi - spectral images and satellite images to obtain the UAV - satellite differential fusion spectral variables.

5. The method according to claim 4, wherein The respectively extracting of the sensitive spectral variables for estimating apple yield from the UAV multi - spectral images and satellite images includes: Perform spectral transformation on spectral image data by 1 / (R i +R j )、1 / (R i ―R j ), R i +R j 、R i ·R j 、R i / (R i +R j ), R i / R i ―R j and R i ·R j ·R q where: R i 、R j 、R q are any bands in five bands respectively; Respectively analyzing the relationships between the spectral variables of different sensors and apple yield; Using the correlation coefficient method to preliminarily screen the sensitive spectral variables of apple yield, and then using the Cars algorithm for further screening.

6. The method according to claim 4, wherein The using of non - negative matrix factorization to fuse the sensitive spectral variables extracted from the UAV multi - spectral images and satellite images to obtain the UAV - satellite differential fusion spectral variables includes: First, representing the sensitive spectral variables as a non - negative matrix V of n×m, where n represents the number of bands or variables, and m represents the number of samples; Given a positive integer r, where r < min(n, m), the two sets of sensitive spectral variables are decomposed into two non-negative matrices according to the NMF algorithm, namely W ∈ R n×r , H ∈ R r×m , such that V ≈ WH, where r represents the number of decompositions, that is, the number of main sources in matrix V, W ∈ R n×r represents the spectral feature matrix, and each column represents the spectral variables of a main source, that is, the basis matrix, H ∈ R r×m is the abundance matrix, and each column represents the fraction of the main source in the sample; Determine the correction coefficient H of the sensitive spectral variable of the satellite image using the NMF algorithm MSISV ; Decompose the sensitive spectral variables of the UAV multispectral image to obtain the basic spectral matrix W UAVSVs ; The NMF algorithm that combines the non - negative least squares (NNLS) method and the projection gradient method is used to alternately decompose the sensitive spectral variables of the UAV and the satellite until the optimal W UAVSVs and H MSISVs ; Multiply the optimal W obtained by decomposition UAVSVs and H MSISVs to obtain the UAV-satellite differential fusion spectral variable.

7. The method according to claim 6, characterized in that, During the decomposition process, the decomposition number r determines the size of the data volume and the approximation degree to the original data. Using the peak signal - to - noise ratio (PSNR) as an evaluation parameter to determine r, and the specific calculation method is as follows: where i is the main factor, MAX i is the maximum spectral input of i, N is the number of samples, and i, k are the values of the main factor i of sample k.

8. The method according to claim 1, wherein The constructing of an apple yield prediction model through the UAV - satellite differential fusion spectral variables includes: Randomly dividing the collected samples into a modeling set and a validation set according to a ratio of 2:1; Using the PLSR, SVM, RF, BPNN algorithms to construct multiple apple yield prediction models in combination with the UAV - satellite differential fusion spectral variables; Using R 2 and nRMSE, RPD to evaluate the accuracy of the apple yield prediction model and screen out the best apple yield prediction model.

9. The method according to claim 8, wherein Apple yield prediction using an apple yield prediction model includes: Obtaining a prediction apple yield sensitive spectral variable map using satellite imagery; Constructing a conversion variable for apple yield sensitive spectral variables before and after differential fusion to generate a differential fusion apple yield sensitive spectral variable map; Finally, inputting the fused apple yield sensitive spectral variable map into the best apple yield prediction model to draw an apple yield prediction map for the prediction area.

Images