Method for estimating vegetative growth condition of wine grapes by using phenotypic characteristics

Abstract

The invention discloses a method for estimating the vegetative growth condition of wine grapes by using phenotypic characteristics, and the method comprises the steps: collecting the remote sensing image data of the growth of grapes in a vineyard based on a remote sensing platform, and obtaining the environment parameter data of the growth of the grapes through an unmanned plane and a ground sensor; preprocessing the remote sensing image data and the environmental parameter data, and performing multi-source data fusion and feature extraction to obtain feature data; and constructing a target multi-modal network model, and monitoring the vegetative growth condition of the wine grapes to obtain a monitoring result. According to the method, through multi-source data fusion and self-adaptive feature extraction, the accuracy of grape growth condition evaluation is remarkably improved, the evaluation time is greatly shortened through the target multi-modal network model, the requirement of real-time monitoring is met, and efficient and accurate evaluation of the vegetative growth condition of wine grapes is achieved.

Description

Technical Field

[0001] The invention belongs to the field of agricultural science, and in particular relates to a method for estimating the nutritional growth status of wine grapes by utilizing phenotypic characteristics. Background Art

[0002] Wine grapes are an important economic crop, and their growth conditions directly affect the quality and yield of wine. Traditionally, the evaluation of grape growth conditions relies on manual observation and empirical judgment. This method is not only inefficient, but also highly subjective and difficult to accurately reflect the actual growth status of grapes. In recent years, with the development of remote sensing technology, image processing technology, and machine learning technology, people have begun to try to use these technologies to automatically evaluate the growth conditions of grapes. However, existing technologies have certain limitations in the extraction and analysis of grape phenotypic characteristics, such as poor adaptability to environmental conditions, lack of universality for different varieties, and low recognition accuracy in complex backgrounds. Specifically reflected in the following aspects:

[0003] Poor environmental adaptability: Existing technologies are unstable under different lighting and climatic conditions, making it difficult to accurately extract the phenotypic characteristics of grapes.

[0004] Lack of universality of varieties: Most existing technologies are optimized for specific varieties of grapes and are difficult to adapt to the growth characteristics of different varieties.

[0005] Low recognition accuracy in complex backgrounds: In complex farmland environments, existing technologies have difficulty accurately distinguishing grapes from other crops or backgrounds, resulting in reduced accuracy of evaluation results.

[0006] Low data processing efficiency: The existing technology has low computational efficiency when processing large amounts of grape image data, making it difficult to meet the needs of real-time monitoring. Summary of the invention

[0007] In order to solve the above technical problems, the present invention provides a method for estimating the nutritional growth status of wine grapes using phenotypic characteristics, comprising:

[0008] Collect remote sensing image data of grape growth in vineyards based on remote sensing platforms, and obtain environmental parameter data of grape growth through drones and ground sensors;

[0009] After preprocessing the remote sensing image data and environmental parameter data, multi-source data fusion and feature extraction are performed to obtain feature data;

[0010] Constructing an initial multimodal network model, and training the initial multimodal network model based on the feature data to obtain a target multimodal network model;

[0011] The nutritional growth status of wine grapes is monitored by the target multimodal network model to obtain monitoring results.

[0012] Preferably, the process of collecting remote sensing image data of grape growth in a vineyard based on a remote sensing platform and obtaining environmental parameter data of grape growth through drones and ground sensors includes:

[0013] Using satellite remote sensing technology to regularly obtain macroscopic remote sensing images of the vineyard, and obtaining macroscopic information of the vineyard based on the macroscopic remote sensing images; the macroscopic information includes but is not limited to vegetation coverage and chlorophyll content;

[0014] Using a drone equipped with a high-definition camera to regularly take low-altitude photos of the vineyard to obtain high-resolution images of the grape plants, and obtaining microscopic information of the grape plants based on the high-resolution images; the microscopic information includes but is not limited to the leaf area and fruit distribution of the grapes;

[0015] By deploying soil moisture sensors and weather stations in the vineyard, environmental parameter data can be acquired through real-time monitoring; the environmental parameter data include but are not limited to soil moisture, temperature, and light intensity.

[0016] Preferably, the process of preprocessing the remote sensing image data and environmental parameter data includes:

[0017] The remote sensing images are processed in sequence by radiation correction, geometric correction, image registration, image fusion, light and color uniformity, image stitching and image cropping to obtain the target remote sensing images;

[0018] The drone aerial images are subjected to pixel normalization, pixel centering, data enhancement, noise reduction, size adjustment and color space conversion in sequence, and the sensor data is filtered and calibrated.

[0019] Preferably, the process of performing multi-source data fusion and feature extraction includes:

[0020] Through wireless communication technology, the data collected by the remote sensing platform, UAV and ground sensors are transmitted to the central data processing system in real time. The multi-source data fusion of the target remote sensing image, UAV aerial image and sensor data is carried out through principal component analysis to obtain fused data. The fused data is then subjected to noise filtering, data cleaning and data enhancement again to extract features related to grape growth and complete feature extraction.

[0021] Preferably, the process of multi-source data fusion of target remote sensing images, drone aerial images and sensor data by principal component analysis includes:

[0022] The sensor data with different sampling rates and viewing angles, as well as the target remote sensing images and drone aerial images are synchronously aligned in time and space to obtain time series data;

[0023] The associated probability distribution of each sensor is synthesized by multi-Bayesian estimation method to provide a final fusion value, and multi-source data fusion is performed according to the final fusion value to obtain fused data.

[0024] Preferably, the process of performing noise filtering, data cleaning and data enhancement on the fused data again includes:

[0025] The sensor noise is removed by mean filtering. For missing values ​​in the sensor data, the missing values ​​are replaced with specific values ​​by interpolation through the nan_to_num function. Then, the outliers are identified and removed or corrected by the Z-score method.

[0026] The data is normalized by using the StandardScaler of the Scikit-learn library to convert the data into a form with a mean of 0 and a standard deviation of 1; then the data is standardized to convert the data into a form with zero mean and unit variance;

[0027] Filtering is performed based on a low-pass filter combined with an exponential smoothing method to remove high-frequency noise;

[0028] The target remote sensing image and the drone aerial image are subjected to brightness change, random rotation, image flipping and motion noise addition processing, wherein the motion noise is Gaussian noise.

[0029] Preferably, the process of performing multi-source data fusion according to the final fusion value to obtain fused data includes:

[0030] Assigning a prior probability distribution to each sensor, wherein the prior probability distribution reflects the knowledge of the target variable in the absence of observed data;

[0031] Determine a likelihood function for each sensor, where the likelihood function describes the probability distribution of the target variable given the observed data;

[0032] Using the Bayesian formula in combination with the prior probability distribution and the likelihood function, a posterior probability distribution is calculated; the posterior probability distribution is used to reflect each sensor's knowledge of the target variable when the observation data is available;

[0033] The posterior probability distributions of all sensors are combined into a joint posterior probability distribution, and a final data fusion result is obtained based on the joint posterior probability distribution; and a final fusion value is obtained by calculating the fused joint posterior probability distribution.

[0034] Preferably, the process of constructing the initial multimodal network model includes:

[0035] Construct an improved convolutional neural network model and a recurrent neural network model, and train the improved convolutional neural network model and the recurrent neural network model using image data and sensor parameter data respectively;

[0036] The trained convolutional neural network model and the recurrent neural network model are fused to obtain a multimodal neural network model.

[0037] Preferably, the process of constructing the improved convolutional neural network model and the recurrent neural network model includes:

[0038] Construct two identical convolutional neural network models, add a BatchNorm2d layer, a nonlinear ReLU layer and an attention mechanism module after the two-dimensional convolution layer in the convolutional neural network model to generate a twin adversarial network model; generate a convolutional neural network group based on the twin adversarial network model;

[0039] Constructing a recurrent neural network, and introducing an attention mechanism into the recurrent neural network for optimization, thereby generating an improved recurrent neural network;

[0040] The convolutional neural network model is a lightweight backbone network designed based on deep separable convolution to replace the backbone network of the YOLOv8 algorithm;

[0041] Design channel grouping multi-scale fusion convolution embedded into the C2f module;

[0042] Optimize the coordinated attention mechanism and embed it into a lightweight backbone network based on deep separable convolution.

[0043] An information sharing decoupling head is designed to obtain the convolutional neural network model.

[0044] Preferably, the target multimodal network model is used to monitor the nutritional growth status of wine grapes, and the process of obtaining the monitoring results includes:

[0045] The target multimodal network model is used to detect whether there are flaws on the surface of the grape fruit and abnormal texture on the surface of the fruit to obtain the shape and flatness of the grape fruit;

[0046] Measuring the geometric size of the grape fruit by using the target multimodal network model, determining whether the size of the grape fruit is within a preset error threshold, and obtaining the size of the grape fruit;

[0047] The target multimodal network model is used to analyze the RGB values ​​of the grape fruit to detect color deviation and obtain the color of the grape fruit.

[0048] Compared with the prior art, the present invention has the following advantages and technical effects:

[0049] The present invention significantly improves the accuracy of grape growth condition assessment through multi-source data fusion and adaptive feature extraction.

[0050] The present invention greatly shortens the evaluation time through the target multimodal network model and meets the needs of real-time monitoring.

[0051] The evaluation method of the present invention can adapt to different environmental conditions, ensure the stability of the evaluation results, and can also adapt to different varieties of grapes, thereby expanding the application scope of the evaluation method.

[0052] The present invention realizes efficient and accurate evaluation of the nutritional growth status of wine grapes, and can also provide scientific management suggestions for grape growers, thereby improving the yield and quality of grapes. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0054] Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present invention. DETAILED DESCRIPTION

[0055] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0056] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0057] like Figure 1 As shown, this embodiment provides a method for estimating the nutritional growth status of wine grapes using phenotypic characteristics, comprising:

[0058] Collect remote sensing image data of grape growth in vineyards based on remote sensing platforms, and obtain environmental parameter data of grape growth through drones and ground sensors;

[0059] After preprocessing the remote sensing image data and environmental parameter data, multi-source data fusion and feature extraction are performed to obtain feature data;

[0060] Constructing an initial multimodal network model, training the initial multimodal network model based on feature data, and obtaining a target multimodal network model;

[0061] The nutritional growth status of wine grapes is monitored through the target multimodal network model to obtain the monitoring results.

[0062] Furthermore, the process of collecting remote sensing image data of grape growth in the vineyard based on the remote sensing platform and obtaining environmental parameter data of grape growth through drones and ground sensors includes:

[0063] Using satellite remote sensing technology to regularly obtain macroscopic remote sensing images of the vineyard, and based on the macroscopic remote sensing images, obtain macroscopic information of the vineyard; the macroscopic information includes but is not limited to vegetation coverage and chlorophyll content;

[0064] Use drones equipped with high-definition cameras to regularly take low-altitude photos of vineyards to obtain high-resolution images of grape plants, and obtain microscopic information of grape plants based on the high-resolution images; the microscopic information includes but is not limited to the leaf area and fruit distribution of grapes;

[0065] By deploying soil moisture sensors and weather stations in the vineyard, environmental parameter data can be acquired through real-time monitoring; environmental parameter data include but are not limited to soil moisture, temperature, and light intensity.

[0066] Furthermore, the process of preprocessing the remote sensing image data and the environmental parameter data includes:

[0067] The remote sensing images are processed in sequence through radiation correction, geometric correction, image registration, image fusion, light and color uniformity, image stitching and image cropping to remove the influence of atmosphere, clouds, etc., obtain accurate surface information and target remote sensing images, and ensure the accuracy and availability of the data.

[0068] Furthermore, radiation correction refers to the process of correcting the systematic and random radiation distortion or aberration caused by external factors, data acquisition and transmission systems, and eliminating or correcting the image distortion caused by radiation errors.

[0069] When using a sensor to observe the reflected or radiated energy of a target, the difference between the measured value and the target's spectral reflectance or spectral radiant brightness and other physical quantities is called radiation error. Radiation error causes distortion of remote sensing images and affects the interpretation and interpretation of remote sensing images. Therefore, it must be eliminated or weakened. It should be pointed out that there are many factors that cause distortion of remote sensing image radiation. In addition to the distortion caused by the sensitivity characteristics of the remote sensor, there are also strong effects such as field of view angle, sun angle, terrain undulation, and atmospheric absorption and scattering.

[0070] Remote sensing image radiation correction mainly includes three aspects:

[0071] Radiation errors caused by the sensitivity characteristics of the sensor, such as edge dimming caused by the non-uniformity of the optical lens, radiation distortion caused by the sensitivity characteristics of the photoelectric conversion system, etc.

[0072] Radiation errors caused by differences in lighting conditions, such as radiation distortion correction caused by different solar altitude angles, radiation distortion correction caused by ground tilt and undulation, etc.

[0073] Correction of radiation errors caused by atmospheric scattering and absorption.

[0074] Radiation correction is divided into two parts: radiation calibration and atmospheric correction.

[0075] Radiometric calibration is when users need to calculate the spectral reflectance or spectral radiance of an object, or need to compare images acquired at different times and with different sensors. In this process, the brightness grayscale value of the image must be converted into absolute radiance. This process is called radiometric calibration.

[0076] The on-orbit calibration method based on sub-pixel targets of the present invention uses a reflective point source with a sufficiently small spatial scale and sufficiently bright radiation brightness as a sub-pixel reference target, and mainly relies on ground synchronous measurement, supplemented by simplified radiation transmission calculation, to achieve high-precision calibration of the full dynamic range of high-resolution optical remote sensing satellites.

[0077] The on-orbit PSF detection of high-resolution optical remote sensing satellite based on sub-pixel targets is a direct detection method based on the physical definition of the point spread function. The parameterized point spread model is used to perform surface fitting on the response data of each sub-pixel target to determine the peak position of each target response. Then, the sub-pixel interpolation point spread response is obtained by position registration of the sub-pixel target response value or count value (DN value) of the 3*3 array with non-integer pixel intervals to reduce the influence of system sampling effect and random noise. As shown in the figure below, X and Y represent the horizontal and vertical coordinate axes of the sub-pixel target image. The point spread function of the optical remote sensing satellite imaging system is obtained by fitting the parameterized Gaussian model, and the response signals of the sub-pixel target reflection and background radiation such as path radiation, ground-air coupling radiation and dark current are separated, so as to statistically analyze the radiation response of the optical remote sensing satellite to the sub-pixel target. If the obtained system point spread function is subjected to discrete Fourier transform, the modulus is taken and normalized, the modulation transfer function (MTF) of the optical remote sensing satellite imaging system can be obtained, which is used to evaluate the imaging quality of the optical remote sensor imaging system.

[0078] Atmospheric correction of satellite images in the visible and near-infrared bands includes atmospheric absorption correction, atmospheric radiation correction and proximity effect correction.

[0079] Based on the equivalent average background reflectivity, the present invention develops an adaptive atmospheric correction algorithm that can effectively remove the proximity effect in sub-meter optical satellite images. The adaptive feature of the adaptive atmospheric correction algorithm is that it can adjust the contribution weight value of each background pixel to the proximity effect according to the optical thickness of atmospheric molecules, the optical thickness of aerosols, the spatial distance between the background pixel and the target pixel, and the difference between the reflectivity of the background pixel and the reflectivity of the target pixel. The adaptive atmospheric correction algorithm can be implemented through the following four steps:

[0080] Step 1: Select satellite images with no clouds or very few thin clouds. Currently, satellite images with no clouds or very few thin clouds are manually selected through satellite images of blue band and near infrared band. Blue band and near infrared band satellite images are used to detect clouds over land and clouds over ocean respectively.

[0081] Step 2: Perform radiometric calibration on the original satellite image to obtain the apparent radiance satellite image, and then calculate the apparent reflectance corresponding to each pixel in the satellite image. The spatial coordinates of each pixel can be omitted here.

[0082] Step 3: Perform atmospheric absorption and atmospheric intrinsic reflectivity correction on the reflectivity satellite image.

[0083] Step 4: Calculate the equivalent average background reflectance corresponding to each pixel in the satellite image, and perform proximity effect correction on each pixel of the satellite image.

[0084] Geometric correction: When the geometric position of a remote sensing image changes, resulting in distortions such as uneven rows and columns, inaccurate correspondence between pixel size and ground size, and irregular changes in the shape of objects, it means that the remote sensing image has undergone geometric distortion. The overall deformation of a remote sensing image (relative to the true shape of the ground) is the result of the combined effects of translation, scaling, rotation, torsion, bending and other deformations. Distorted images make it difficult to analyze given quantities and align positions. Therefore, after receiving the remote sensing data, the receiving department will first perform corrections. This correction is often processed based on various parameters of the remote sensing platform, the earth, and the sensor. After users receive this product, further geometric correction is still required due to different purposes of use or different projections and scales. Geometric correction generally includes fine correction and orthorectification.

[0085] Precision correction: Use ground control points to correct the geometric distortion of remote sensing images caused by various factors. Simply put: Correction with topographic maps, with accurate longitude and latitude information after correction. Precision correction is suitable for situations where the ground is flat and elevation information does not need to be considered, or where the ground is undulating and there is no elevation information. Sometimes, a correction has been made based on various parameters of the remote sensing platform, but it still cannot meet the requirements. This method can be used to perform registration correction of remote sensing images relative to ground coordinates, registration correction of remote sensing images relative to map projection coordinate systems, and geometric registration and composite analysis between remote sensing data of different types or phases to obtain more accurate results.

[0086] Orthorectification: Orthophoto production is generally done by selecting some ground control points on the image, and using the digital elevation model (DEM) data that has been acquired within the range of the image to simultaneously perform tilt correction and projection error correction on the image, and resample the image into an orthophoto. Multiple orthophotos are stitched together, and after color balancing, the image cut out within a certain range is an orthophoto. Orthophotos have both topographic map characteristics and image characteristics, and are rich in information. They can be used as a data source for GIS, thereby enriching the presentation of geographic information systems.

[0087] For further optimization, the image registration problem can be divided into two parts: radiation value transformation (i.e. grayscale transformation) and spatial transformation. The radiation value transformation solves the mapping relationship between radiation values ​​between images of different modalities, while the spatial transformation needs to solve the geometric transformation between images. The radiation value transformation is less discussed in feature-based registration methods, but it has an important influence in pixel-based methods. Solving the spatial transformation is the core problem of image registration, which usually includes two parts: one is to determine the model of geometric transformation based on prior knowledge, and the other is to solve the parameters of the transformation model based on the optimization algorithm.

[0088] The image registration method of the present invention includes four types, namely, manual registration, Fourier transform and transform domain method, pixel-based or distribution-based method and feature-based method.

[0089] Image fusion includes three stages: image data preprocessing, image fusion, and image application. The present invention mainly summarizes image data fusion into three levels: pixel level, feature level, and decision level fusion.

[0090] Pixel-level image fusion is to directly process the registered images according to a specific fusion algorithm to generate a new image, and then extract and express the features of the fused image. The process is to directly process the pixel grayscale values ​​of the original image data to retain most of the original information of the scene objects.

[0091] Feature-level image fusion is to process and analyze the target feature information extracted from each source data. It includes: first extracting the obvious feature information of the image object target (such as contour, texture, edge, etc.), synthesizing it into a feature information set, then performing feature vector analysis on the feature information set, and training the analysis results to obtain a training subset, and finally fusing the feature training subset to obtain a fused image. Feature-level image fusion reduces the amount of data calculation and improves the calculation efficiency. Compared with pixel-level fusion, it does not require higher image registration accuracy. Its fusion methods include Bayesian estimation, cluster analysis, entropy method, etc.

[0092] Decision-level image fusion is based on the input extracted feature information and processed through relevant fusion methods to provide a basis for decision-making. Its fusion algorithms mainly include Bayesian estimation, neural network method and expert system.

[0093] Further optimization scheme: During the image shooting process, due to factors such as lens distortion, different reflection characteristics of objects caused by different shooting times, different lighting conditions, shooting conditions and terrain, the image itself has single image quality problems such as uneven lighting, haze, shadows and color cast. These problems greatly affect the interpretation of objects. Therefore, in the process of uniform light and color, in order to make the orthorectified remote sensing images consistent in color and tone, and to achieve seamless splicing of multiple images in a large range, it is necessary to perform uniform light and color on the single orthorectified image results and balance the tone and brightness between multiple orthorectified image results.

[0094] The specific plan is as follows: first, the original image is appropriately resampled, that is, the orthorectification process of a single image is carried out to ensure that the image is corrected to an appropriate resolution; secondly, the quality problem of a single image is processed under the premise that the image itself is not distorted, including removing haze, processing shadows, and color cast processing; then, on the basis of processing the quality problem of a single image, the consistency of multiple images is processed so that all images can maintain a consistent hue and brightness as a whole, and then there will be no obvious difference in hue and brightness after all the single orthorectified images are mosaicked into a picture, thus ensuring the overall visual effect; finally, when mosaicking the edges, the trend and feathering value of the edge lines are adjusted to ensure that there are no obvious traces of mosaic lines on the finished image.

[0095] The main process of image stitching can be summarized as follows:

[0096] 1) Image preprocessing and feature point extraction

[0097] Usually, before extracting image feature points, it is necessary to process the image in some way, such as removing noise, performing some kind of transformation on the image, etc. This step is included in the feature point extraction algorithm. Feature points are the most commonly used and can best represent the attributes and uniqueness of the image, such as corner points, inflection points, points with high curvature, and intersection points.

[0098] 2) Matching of points with the same name

[0099] A certain matching strategy is used to find the corresponding position of the target point in the image to be stitched in the reference image among a large number of feature points. A good matching algorithm should have good accuracy, high precision and fast calculation speed.

[0100] 3) Image registration and transformation model establishment

[0101] That is, a more accurate matching strategy is adopted to eliminate the matching point pairs with low accuracy and the wrong matching point pairs from the matched same-name points. On this basis, the transformation relationship model between the two images is determined, and the parameter values ​​in the model are calculated. This step is usually called precise matching. This step is the most critical step in image stitching. There may be multiple transformations such as translation, rotation, and scaling between the images to be stitched. If the most accurate same-name point pairs cannot be retained and the point pairs with low accuracy cannot be eliminated, a slight error in a point pair may cause various unpredictable deformations in the stitched image, or even fail to form an image.

[0102] 4) Unified coordinate system, integrated reconstruction

[0103] According to the established transformation relationship model, the pixels in the image to be stitched are converted to the coordinate system of the reference plane to complete the unified coordinate transformation. Image fusion mainly smoothes the boundaries of the image stitching and fuses the overlapping areas of the stitched images to obtain a smooth and seamless panoramic image reconstructed by stitching. Since it is impossible for any two adjacent images to be completely identical in acquisition conditions, some image features that should be the same, such as the illumination characteristics of the image, will not be exactly the same in the two images. Therefore, some obvious seams will be generated when the images are stitched together, so fusion technology is needed to make the seams between images less obvious and the stitching more natural.

[0104] It is specifically based on the minimum grayscale difference image stitching algorithm, and its main steps include:

[0105] 1) Each pixel point in the first row of the difference image corresponds to the starting point of each mosaic line, and the energy value of the first row of the difference image is recorded as the starting energy value.

[0106] 2) Starting from the starting point, the grayscale values ​​of the points on each mosaic line are added to the grayscale values ​​of the three nearest neighboring pixels in the corresponding next row. By comparing the sum of the three pixel values, the column with the minimum value is taken as the direction of the next mosaic point of the mosaic line. At the same time, the energy value of this mosaic line is updated to the sum of the energy values ​​of the upper and lower rows.

[0107] 3) Iterate step 2 until the last mosaic line point on the difference image is found. Select the minimum energy value from all mosaic lines as the minimum grayscale difference mosaic line.

[0108] 4) Subsequently, a seam elimination algorithm is used to eliminate the color difference on both sides of the mosaic line.

[0109] The drone aerial images are subjected to pixel normalization, pixel centering, data enhancement, noise reduction, size adjustment and color space conversion in sequence, and the sensor data is filtered and calibrated.

[0110] Furthermore, the process of multi-source data fusion and feature extraction includes:

[0111] Through wireless communication technology, the data collected by the remote sensing platform, drones and ground sensors are transmitted to the central data processing system in real time. The multi-source data fusion of the target remote sensing image, drone aerial image and sensor data is carried out through principal component analysis to obtain fused data. The fused data is then subjected to noise filtering, data cleaning and data enhancement again to extract features related to grape growth and complete feature extraction.

[0112] Furthermore, the process of multi-source data fusion of target remote sensing images, UAV aerial images and sensor data by principal component analysis includes:

[0113] The sensor data with different sampling rates and viewing angles, as well as the target remote sensing images and drone aerial images are synchronously aligned in time and space to obtain time series data;

[0114] The correlation probability distribution of each sensor is synthesized through the multi-Bayesian estimation method to provide a final fusion value. Multi-source data is fused according to the final fusion value to obtain fused data.

[0115] Furthermore, the process of noise filtering, data cleaning and data enhancement of the fused data includes:

[0116] The sensor noise is removed by mean filtering. For missing values ​​in the sensor data, the missing values ​​are replaced with specific values ​​by interpolation through the nan_to_num function. Then, the outliers are identified and removed or corrected by the Z-score method.

[0117] The data is normalized by using the StandardScaler of the Scikit-learn library to convert the data into a form with a mean of 0 and a standard deviation of 1; then the data is standardized to convert the data into a form with zero mean and unit variance;

[0118] Filtering is performed based on a low-pass filter combined with an exponential smoothing method to remove high-frequency noise;

[0119] The target remote sensing images and drone aerial photos are processed by brightness change, random rotation, image flipping and adding motion noise, where the motion noise is Gaussian noise.

[0120] Furthermore, the multi-source data is fused according to the final fusion value, and the process of obtaining fused data includes:

[0121] Assign a prior probability distribution to each sensor, which reflects the understanding of the target variable in the absence of observed data;

[0122] Determine the likelihood function of each sensor, which describes the probability distribution of the target variable given the observed data;

[0123] The posterior probability distribution is calculated using the Bayesian formula in combination with the prior probability distribution and the likelihood function; the posterior probability distribution is used to reflect each sensor's knowledge of the target variable when the observation data is available;

[0124] The posterior probability distributions of all sensors are combined into a joint posterior probability distribution, and the final data fusion result is obtained based on the joint posterior probability distribution; the final fusion value is calculated based on the fused joint posterior probability distribution.

[0125] Furthermore, the process of constructing the initial multimodal network model includes:

[0126] Construct an improved convolutional neural network model and a recurrent neural network model, and train the improved convolutional neural network model and the recurrent neural network model using image data and sensor parameter data respectively;

[0127] The trained convolutional neural network model and the recurrent neural network model are fused to obtain a multimodal neural network model.

[0128] Furthermore, the process of constructing the improved convolutional neural network model and the recurrent neural network model includes:

[0129] Build two identical convolutional neural network models, add a BatchNorm2d layer, a nonlinear ReLU layer, and an attention mechanism module after the two-dimensional convolution layer in the convolutional neural network model to generate a twin adversarial network model; generate a convolutional neural network group based on the twin adversarial network model;

[0130] Construct a recurrent neural network and introduce the attention mechanism into the recurrent neural network for optimization to generate an improved recurrent neural network;

[0131] Among them, the convolutional neural network model is to design a lightweight backbone network based on deep separable convolution to replace the backbone network of the YOLOv8 algorithm;

[0132] Design channel grouping multi-scale fusion convolution embedded into the C2f module;

[0133] Optimize the coordinated attention mechanism and embed it into a lightweight backbone network based on deep separable convolution.

[0134] Design an information sharing decoupling head to obtain the convolutional neural network model.

[0135] The lightweight backbone network based on depthwise separable convolution consists of four streamlined blocks, with a transition layer between the feature input and the streamlined blocks for feature map downsampling and channel number expansion.

[0136] The streamlined block consists of a 3×3 depthwise separable convolutional layer, 2 point-wise convolutions, and an intermediate layer between the point-wise convolutions.

[0137] Channel grouping multi-scale fusion convolution obtains spatial features of multiple scales by introducing basic convolutions with different kernel sizes, and fuses the spatial features through point-by-point convolution.

[0138] Feature fusion methods include:

[0139] The input features are divided into two groups along the channel direction. One group obtains multi-scale feature information through 1×1 and 3×3 basic convolutions, and the multi-scale feature information is concatenated with the features of the other group to obtain fused features.

[0140] Methods for optimizing the coordinated attention mechanism include:

[0141] Based on the coordinated attention mechanism, new paths containing salient information are obtained through global maximum pooling;

[0142] Obtain enhanced information on the original path through convolution and sigmoid operations;

[0143] Add the mean of the enhanced information on the original path to the salient information of the new path to obtain the joint information;

[0144] The enhanced information on the original path is fused with the joint information to obtain the complete enhanced information.

[0145] Methods for designing information sharing decoupling headers include:

[0146] Channel grouping multi-scale fusion convolution replaces all 3×3 convolutions of the two branches in the information sharing decoupling head to perform feature extraction and fusion sharing to obtain the output, which is then sent to different branches for prediction.

[0147] Furthermore, the nutritional growth status of wine grapes is monitored through the target multimodal network model, and the process of obtaining the monitoring results includes:

[0148] The target multimodal network model is used to detect whether there are any flaws on the surface of the grape fruit and the abnormal texture of the fruit surface to obtain the shape and flatness of the grape fruit;

[0149] The geometric size of the grape fruit is measured by the target multimodal network model, and whether the size of the grape fruit is within a preset error threshold is determined to obtain the size of the grape fruit;

[0150] The target multimodal network model is used to analyze the RGB values ​​of the grape fruit to detect color deviation and obtain the color of the grape fruit.

[0151] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for estimating the nutritional growth status of wine grapes using phenotypic characteristics, characterized in that: include: Collect remote sensing image data of grape growth in vineyards based on remote sensing platforms, and obtain environmental parameter data of grape growth through drones and ground sensors; After preprocessing the remote sensing image data and environmental parameter data, multi-source data fusion and feature extraction are performed to obtain feature data; Constructing an initial multimodal network model, and training the initial multimodal network model based on the feature data to obtain a target multimodal network model; The nutritional growth status of wine grapes is monitored by the target multimodal network model to obtain monitoring results.

2. The method for estimating the nutritional growth status of wine grapes using phenotypic characteristics according to claim 1, characterized in that: The process of collecting remote sensing image data of grape growth in the vineyard based on the remote sensing platform and obtaining environmental parameter data of grape growth through drones and ground sensors includes: Using satellite remote sensing technology to regularly obtain macroscopic remote sensing images of the vineyard, and obtaining macroscopic information of the vineyard based on the macroscopic remote sensing images; the macroscopic information includes but is not limited to vegetation coverage and chlorophyll content; Using a drone equipped with a high-definition camera to regularly take low-altitude photos of the vineyard to obtain high-resolution images of the grape plants, and obtaining microscopic information of the grape plants based on the high-resolution images; the microscopic information includes but is not limited to the leaf area and fruit distribution of the grapes; By deploying soil moisture sensors and weather stations in the vineyard, environmental parameter data can be acquired through real-time monitoring; the environmental parameter data include but are not limited to soil moisture, temperature, and light intensity.

3. The method for estimating the nutritional growth status of wine grapes using phenotypic characteristics according to claim 1, characterized in that: The process of preprocessing the remote sensing image data and environmental parameter data includes: The remote sensing images are processed in sequence by radiation correction, geometric correction, image registration, image fusion, light and color uniformity, image stitching and image cropping to obtain the target remote sensing images; The drone aerial images are subjected to pixel normalization, pixel centering, data enhancement, noise reduction, size adjustment and color space conversion in sequence, and the sensor data is filtered and calibrated.

4. The method for estimating the nutritional growth status of wine grapes using phenotypic characteristics according to claim 1, characterized in that: The process of multi-source data fusion and feature extraction includes: Through wireless communication technology, the data collected by the remote sensing platform, UAV and ground sensors are transmitted to the central data processing system in real time. The multi-source data fusion of the target remote sensing image, UAV aerial image and sensor data is carried out through principal component analysis to obtain fused data. The fused data is then subjected to noise filtering, data cleaning and data enhancement again to extract features related to grape growth and complete feature extraction.

5. The method for estimating the nutritional growth status of wine grapes using phenotypic characteristics according to claim 4, characterized in that: The process of multi-source data fusion of target remote sensing images, UAV aerial images and sensor data by principal component analysis includes: The sensor data with different sampling rates and viewing angles, as well as the target remote sensing images and drone aerial images are synchronously aligned in time and space to obtain time series data; The associated probability distribution of each sensor is synthesized by multi-Bayesian estimation method to provide a final fusion value, and multi-source data fusion is performed according to the final fusion value to obtain fused data.

6. The method for estimating the nutritional growth status of wine grapes using phenotypic characteristics according to claim 4, characterized in that: The process of performing noise filtering, data cleaning and data enhancement on the fused data again includes: The sensor noise is removed by mean filtering. For missing values ​​in the sensor data, the missing values ​​are replaced with specific values ​​by interpolation through the nan_to_num function. Then, the outliers are identified and removed or corrected by the Z-score method. The data is normalized by using the StandardScaler of the Scikit-learn library to convert the data into a form with a mean of 0 and a standard deviation of 1; then the data is standardized to convert the data into a form with zero mean and unit variance; Filtering is performed based on a low-pass filter combined with an exponential smoothing method to remove high-frequency noise; The target remote sensing image and the drone aerial image are subjected to brightness change, random rotation, image flipping and motion noise addition processing, wherein the motion noise is Gaussian noise.

7. The method for estimating the nutritional growth status of wine grapes using phenotypic characteristics according to claim 5, characterized in that: The process of fusing multi-source data according to the final fusion value to obtain fused data includes: Assigning a prior probability distribution to each sensor, wherein the prior probability distribution reflects the knowledge of the target variable in the absence of observed data; Determine a likelihood function for each sensor, where the likelihood function describes the probability distribution of the target variable given the observed data; Using the Bayesian formula in combination with the prior probability distribution and the likelihood function, a posterior probability distribution is calculated; the posterior probability distribution is used to reflect each sensor's knowledge of the target variable when the observation data is available; The posterior probability distributions of all sensors are combined into a joint posterior probability distribution, and a final data fusion result is obtained based on the joint posterior probability distribution; and a final fusion value is obtained by calculating the fused joint posterior probability distribution.

8. The method for estimating the nutritional growth status of wine grapes using phenotypic characteristics according to claim 1, characterized in that: The process of constructing the initial multimodal network model includes: Construct an improved convolutional neural network model and a recurrent neural network model, and train the improved convolutional neural network model and the recurrent neural network model using image data and sensor parameter data respectively; The trained convolutional neural network model and the recurrent neural network model are fused to obtain a multimodal neural network model.

9. The method for estimating the nutritional growth status of wine grapes using phenotypic characteristics according to claim 8, characterized in that: The process of building the improved convolutional neural network model and recurrent neural network model includes: Construct two identical convolutional neural network models, add a BatchNorm2d layer, a nonlinear ReLU layer and an attention mechanism module after the two-dimensional convolution layer in the convolutional neural network model to generate a twin adversarial network model; generate a convolutional neural network group based on the twin adversarial network model; Constructing a recurrent neural network, and introducing an attention mechanism into the recurrent neural network for optimization, thereby generating an improved recurrent neural network; The convolutional neural network model is a lightweight backbone network designed based on deep separable convolution to replace the backbone network of the YOLOv8 algorithm; Design channel grouping multi-scale fusion convolution embedded into the C2f module; Optimize the coordinated attention mechanism and embed it into a lightweight backbone network based on deep separable convolution. An information sharing decoupling head is designed to obtain the convolutional neural network model.

10. The method for estimating the nutritional growth status of wine grapes using phenotypic characteristics according to claim 1, characterized in that: The target multimodal network model is used to monitor the nutritional growth status of wine grapes, and the process of obtaining the monitoring results includes: The target multimodal network model is used to detect whether there are flaws on the surface of the grape fruit and abnormal texture on the surface of the fruit to obtain the shape and flatness of the grape fruit; Measuring the geometric size of the grape fruit by using the target multimodal network model, determining whether the size of the grape fruit is within a preset error threshold, and obtaining the size of the grape fruit; The target multimodal network model is used to analyze the RGB values ​​of the grape fruit to detect color deviation and obtain the color of the grape fruit.

Images