A multimodal fruit tree surface area and volume prediction method and system

Abstract

The present invention relates to the technical field of fruit tree planting, and specifically discloses a multimodal fruit tree surface area and volume prediction method and system, wherein the method includes the following contents: S1, extracting the canopy north-south width, canopy east-west width and plant height of each fruit tree point cloud data as text feature values; S2, extracting the contours of the point cloud data of each fruit tree from two angles, front and side, to obtain a contour map as an image feature value; S3, obtaining the volume value; S4, obtaining the surface area value; S5, building a multimodal neural network model architecture; establishing a data set according to the text feature value, image feature value, volume value and surface area value, and training; S6, obtaining the canopy north-south width, canopy east-west width and whole plant height of the fruit tree input by the user; or inputting the front and side photos of the fruit tree as supplementary input, and predicting the volume value and surface area value of the fruit tree. The technical solution of the present invention can simplify the measurement process and improve the prediction accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of fruit tree planting, and in particular to a multi-modal fruit tree surface area and volume prediction method and system. Background Art

[0002] In the cultivation and management of fruit trees, accurate estimation of the volume and surface area of ​​fruit trees is very important, which can provide a scientific basis for management measures such as pruning, fertilization, and pest and disease control. Taking citrus trees as an example, my country's citrus planting area and output are among the highest in the world. According to relevant statistics, the citrus industry contributes about 15% to my country's agricultural output value, making a huge contribution to my country's agricultural economy. Citrus trees are evergreen perennial plants, and their crown characteristics are the key to understanding growth, yield, physiological indicators and growth conditions.

[0003] Common methods for measuring canopy volume include manual measurement and non-contact automatic measurement. Manual methods include measuring the height and width of trees with tools such as tape and height rulers, approximating the crown into a geometric model, and using the model's formula to calculate the volume. Miranda Fuentes et al. found that the elliptical model method gave the best results in predicting fruit tree volume and surface area, R 2 is 0.84. Zheng Yongqiang et al. used an uneven cylindrical shape for Hamlin orange trees, demonstrating that manual volume measurements can be used as growth and yield indicators. Although manual measurements are simple and convenient, they are prone to subjective errors, and there are significant differences between the model and the actual shape of the crown. No simple polyhedron model can accurately represent the actual shape of the fruit tree.

[0004] Non-contact automatic measurement methods such as Dong.X et al. reconstructed the orchard using photos taken by a camera carrying a drone and extracted the area of ​​a single fruit tree in two plots, with root mean square error (RMSE) values ​​of 0.72m and 0.72m, respectively. 2 and 0.48m 2, indicating that the method overestimates the area of ​​fruit trees. This shows that the camera is easily limited by light and climatic conditions, requires high-level equipment and technology, complex data processing, and the necessity of large amounts of data collection. Dong.W et al. collected data from horizontal or top-down perspectives by placing RGB-D depth cameras on poles to collect canopy volume, trunk diameter, tree height, and number of fruits in an orchard, and then used photos from both sides to reconstruct the apple trees in 3D. Subsequently, they used this data to reconstruct a 3D model of the apple tree by using images taken from multiple angles. Based on the reconstruction results, the yield prediction accuracy of the algorithm on the three datasets was 0.91-0.94, with some underestimation problems. Although depth cameras are more precise and accurate in 3D reconstruction and are more sensitive to light changes than standard cameras, they still have disadvantages, including low-quality depth images and a small measurement range. Subsequent researchers tried methods based on ultrasonic sensors, drone oblique photography and lidar, but these high-precision equipment are not only expensive and complicated to use, but also require the coordination of multiple devices and the development of appropriate preprocessing processes and reconstruction algorithms to complete the extraction of volume and surface area.

[0005] After going deep into the fields, we found that most of China's orchards are managed in a decentralized manner. Even in some large-scale orchards, there are many problems that hinder precision agriculture, such as low levels of mechanization and intelligence, and imperfect information construction. Moreover, most orchard managers have not received training in agricultural informatization, and have no idea how to use complex equipment and software, so most of them adopt conservative planting methods.

[0006] Therefore, there is a need for a multimodal fruit tree surface area and volume prediction method and system that can simplify the measurement process and improve prediction accuracy. Summary of the invention

[0007] One of the purposes of the present invention is to provide a multimodal fruit tree surface area and volume prediction system that can simplify the measurement process and improve the prediction accuracy.

[0008] In order to solve the above technical problems, this application provides the following technical solutions:

[0009] A multimodal fruit tree surface area and volume prediction method, comprising the following contents:

[0010] S1. Preparation of text data set: collecting point cloud data of fruit trees in a preset area, extracting the north-south width, east-west width and plant height of each fruit tree point cloud data as text feature values;

[0011] S2. Image data set preparation: For each fruit tree, the point cloud data is extracted from the front and side angles to obtain the contour map as the image feature value;

[0012] S3, volume value acquisition, using dynamic slicing and reconstruction algorithm to reconstruct the point cloud data of a single fruit tree and obtain the volume value;

[0013] S4, obtaining the surface area value, using the AlphaShape algorithm to reconstruct the surface of the fruit tree point cloud data, and obtaining the surface area value based on the reconstructed surface;

[0014] S5. Model training: building a multimodal neural network model architecture based on text features and image features; establishing a data set based on the text feature values, front and side image feature values, volume values, and surface area values ​​of each fruit tree; performing multimodal training based on the data set, and saving the trained multimodal neural network model;

[0015] S6. Model application, obtaining the north-south width of the canopy, the east-west width of the canopy and the height of the whole plant input by the user, and predicting the volume value and surface area value of the fruit tree;

[0016] Alternatively, the user may input photos of the front and side angles of the fruit tree as supplementary input to predict the volume and surface area of ​​the fruit tree.

[0017] Further, the step S2 specifically includes:

[0018] The Grounded-Segment-Anything model is used to extract the mask of the fruit tree point cloud data; the opencv computer vision library is used to extract the mask contour and obtain the contour map;

[0019] Or for the image of a single fruit tree taken in a natural environment, use the Grounded-Segment-Anything model to segment the target fruit tree outline, and use OpenCV to remove the isolated areas in the fruit tree image segmentation in the natural environment to obtain the outline map;

[0020] The extracted contour image is used as the image feature value.

[0021] Further, in step S2, the Grounded-Segment-Anything model is obtained by integrating Grounding DINO and SAM;

[0022] Using the Grounded-Segment-Anything model includes:

[0023] Load the pre-trained Grounding DINO model checkpoint to detect the object bounding box in the image. The size of the target bounding box of the segmented object distinguishes whether the citrus tree is the background. After traversing the bounding boxes of all segmented objects and calculating the area, the largest target detection box area is retained, which is the target to be segmented.

[0024] Initialize the SAM model and use it to perform instance segmentation on the detected target to obtain the target mask image; process the invalid isolated closed areas in the mask image to obtain the contour map of the target tree.

[0025] Further, the step S5 specifically includes:

[0026] S501. Data preprocessing

[0027] The text feature value data of fruit trees were collected and stored in a CSV file; the text feature value data were standardized so that the mean value was 0 and the standard deviation was 1;

[0028] The standardized data is divided into 80% for training and 20% for testing;

[0029] S502. Define model architecture

[0030] Construct a multimodal neural network model architecture, input text feature value data into a two-layer fully connected network of the multimodal neural network model, and input image feature value data into a pre-trained ResNet-50 model to extract features;

[0031] The text and image features are then concatenated and processed through a fully connected network, and finally two predicted values ​​are output, which are the volume and surface area values ​​of the citrus tree.

[0032] Furthermore, the method further includes: S503, setting a loss function

[0033] Using mean square error as the loss function, the mean square error loss function is defined as follows:

[0034]

[0035] Among them, y i Indicates the actual value, represents the predicted value, n is the number of samples;

[0036] S504, Optimization method

[0037] The AdamW optimizer is used with an initial learning rate of 0.001. The ReduceLROnPlateau learning rate scheduler is also used to halve the learning rate when the monitored loss does not improve for 10 consecutive epochs.

[0038] S505, training process

[0039] The training is set to run for a maximum of 100 epochs, with a batch size of 32. The early stopping mechanism is used, and the early stopping criterion is set to monitor the training loss. If the training loss does not improve within 20 consecutive epochs, the training is terminated early.

[0040] Furthermore, the fruit tree is a citrus tree.

[0041] A second object of the present invention is to provide a multimodal fruit tree surface area and volume prediction system using the above method.

[0042] Traditional methods rely on a single type of data, such as using only text data or image data. This limits the accuracy and adaptability of the model because a single data source cannot fully capture the complex characteristics of citrus trees. This solution achieves a more accurate prediction of the volume and surface area of ​​fruit trees by fusing text data such as the canopy width and height of citrus trees with image data taken from different angles. This fusion of multimodal data enables the model to obtain information from multiple dimensions, improving the accuracy and robustness of the prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a schematic flow chart of a multi-modal fruit tree surface area and volume prediction method according to Example 1;

[0044] Figure 2 It is a schematic diagram of segmenting a 3D map of an orchard into individual trees in a multimodal fruit tree surface area and volume prediction method according to Example 1;

[0045] Figure 3 It is a schematic diagram of mask extraction of Grounded-Segment-Anything model in a multimodal fruit tree surface area and volume prediction method according to Example 1;

[0046] Figure 4 A schematic diagram of extracting image contours using OpenCV in a multimodal fruit tree surface area and volume prediction method according to Example 1;

[0047] Figure 5 It is a schematic diagram of segmenting citrus trees using the SAM model before optimization in a multimodal fruit tree surface area and volume prediction method in Example 1;

[0048] Figure 6 It is a schematic diagram of segmenting citrus trees using the optimized SAM model in a multimodal fruit tree surface area and volume prediction method according to Example 1;

[0049] Figure 7 A schematic diagram of removing isolated regions in a multimodal fruit tree surface area and volume prediction method according to Example 1;

[0050] Figure 8This is a schematic diagram of the effect of using dynamic slicing and reconstruction algorithm in a multimodal fruit tree surface area and volume prediction method in Example 1;

[0051] Fig. 9 This is a schematic diagram of the surface reconstruction effect of using the AlphaShape algorithm in a multimodal fruit tree surface area and volume prediction method in Example 1;

[0052] Fig.10 A schematic diagram of a multimodal prediction model architecture in a multimodal fruit tree surface area and volume prediction method according to Embodiment 1;

[0053] Fig.11 It is a schematic diagram of removing isolated points in a multimodal fruit tree surface area and volume prediction method in Example 2;

[0054] Fig.12 This is a schematic diagram of a detailed architecture of a model in a multimodal fruit tree surface area and volume prediction method according to Embodiment 3;

[0055] Fig.13 A comparison diagram of training loss curves in a multimodal fruit tree surface area and volume prediction method according to Embodiment 5;

[0056] Fig.14 This is a schematic diagram of the open-shaped, open-hearted and round-headed shapes of fruit trees in a multimodal fruit tree surface area and volume prediction method in Example 5. DETAILED DESCRIPTION

[0057] The following is further described in detail through specific implementation methods:

[0058] Embodiment 1

[0059] like Figure 1 As shown, a multimodal fruit tree surface area and volume prediction method of this embodiment includes the following steps:

[0060] S1. Text dataset preparation

[0061] Collecting point cloud data of fruit trees in a preset area. As shown in Table 1, this embodiment uses a GOSLAM RS100 laser scanner to collect point cloud data of a total of 336 fruit trees in the main citrus producing areas of Guangdong, Hunan, Sichuan, Chongqing and Jiangxi.

[0062] Table 1 Dataset composition

[0063]

[0064] In this embodiment, a 3D map of the orchard is obtained by scanning, such as Figure 2 As shown, the point cloud data of the orchard is segmented into single trees to obtain the point cloud data of a single fruit tree;

[0065] The process of single tree segmentation is as follows: First, use a lidar scanner to scan the 3D map of the orchard. Then use CloudCompare software to crop the experimental area. Use the CSF algorithm in the lidR package in RStudio software to filter the ground points, and then use the fast Euclidean clustering algorithm written in Python in PyCharm software to segment individual fruit trees. Finally, use the SOR algorithm in CloudCompare software to eliminate noise points, and use the unique() function built into MATLAB to eliminate duplicate points. Thus, the point cloud data of a single fruit tree is obtained.

[0066] The north-south width of the canopy, the east-west width of the canopy and the plant height of each fruit tree point cloud data were extracted as text feature values.

[0067] S2. Image dataset preparation

[0068] For each fruit tree point cloud data, two angles, front and side, are selected to extract the contour using the SAM model to obtain the contour map; the specific steps are as follows:

[0069] like Figure 3 As shown, the Grounded-Segment-Anything model is used to extract the mask of the fruit tree point cloud data;

[0070] like Figure 4 As shown, the mask contour is extracted using the opencv computer vision library to obtain a contour map;

[0071] For the image of a single citrus tree taken in a natural environment, the Grounded-Segment-Anything model based on SAM is used for segmentation, such as Figure 5 As shown, this embodiment optimizes the parameters of the SAM model for citrus trees so that it can accurately segment the outline of the target fruit tree from a real complex environment, such as Figure 6 shown.

[0072] like Figure 7 As shown, OpenCV is used to remove isolated areas in the image segmentation of citrus trees in natural environments and obtain a contour map.

[0073] The extracted contour image is used as the image feature value.

[0074] S3. Volume value acquisition

[0075] The point cloud of a single fruit tree is reconstructed and the volume value is calculated using a dynamic slicing and reconstruction algorithm. This algorithm dynamically merges slices by considering the area change ratio and density difference of adjacent slices. It has been proven in experiments that it can effectively remove holes and gaps in the citrus tree model. It is the reconstruction algorithm that is closest to the real volume of the citrus tree. The reconstruction effect is as follows: Figure 8 shown.

[0076] In this embodiment, a canopy volume calculation method proposed by the applicant is specifically adopted, with patent application number: 202410583001.7. By inputting point cloud data, the crown point cloud data is indirectly sliced ​​to obtain the area of ​​the outer contour polygon of the crown slice; the area change percentage of each adjacent slice is calculated; the overall density of each slice is calculated, and the difference between the slice density and the average density is calculated; if the area change percentage R of two adjacent slices is greater than the average area percentage of the adjacent slices, or the density difference between the previous slice and the current slice is less than the overall average difference, the slice is marked as 1, otherwise it is marked as 0; if the marks of two adjacent slices are the same, the two slices are merged; the slice point cloud is reconstructed, and the volume of the reconstructed area is calculated; the volumes of all slices are added together to obtain the crown volume.

[0077] S4. Obtaining surface area value

[0078] The AlphaShape algorithm is used to reconstruct the surface of the fruit tree point cloud data. The reconstruction effect is as follows Fig. 9 As shown, when using the AlphaShape algorithm to construct the contour, there is a threshold α b , when α≥α b When , the boundary polygon formed can contain all points in the point set P. According to the reconstruction effect of the citrus tree, it is found that when α = 0.05, the reconstruction effect is the best, so 0.05 is used as the α value to perform AlphaShape reconstruction, and the surface area is obtained based on the reconstructed surface.

[0079] S5. Model training

[0080] Build a multimodal neural network model architecture based on text features and image features; establish a data set based on the text feature values, front and side image feature values, volume values ​​and surface area values ​​of each fruit tree; randomly divide the data set into 80% training set and 20% test set, define the loss function, introduce the optimization method and early stopping mechanism, and start multimodal training, and save the best training model (.pth). The model architecture is as follows: Fig.10 shown.

[0081] S6. Model application

[0082] The east-west canopy width, north-south canopy width, and whole plant height of the citrus tree input by the user are obtained, and the volume and surface area values ​​of the citrus tree are predicted.

[0083] Alternatively, in addition to inputting the east-west and north-south canopy widths and the height of the entire plant, the user can also take photos of the citrus tree from the front and side angles as supplementary input to predict the volume and surface area values ​​of the citrus tree.

[0084] This embodiment also provides a multimodal fruit tree surface area and volume prediction system using the above method.

[0085] With this solution, there is no need to classify citrus trees according to tree shape or crown. Fruit farmers can choose to directly give the east-west and north-south canopy widths and plant heights to accurately predict the volume and surface area of ​​the fruit trees. At the same time, fruit farmers can also choose to take two more pictures of citrus trees, which will automatically complete the steps of target segmentation-obtaining contours-extracting features, making the prediction results more robust.

[0086] In this solution, the accuracy of volume prediction reached 98%, and the accuracy of surface area prediction reached 97%, which solved the defects of traditional methods such as expensive equipment, difficult operation, and insufficient technology, and provided important technical support for the precision and intelligence of China's citrus industry.

[0087] Embodiment 2

[0088] The difference between this embodiment and the first embodiment is that in step S2 of this embodiment, corresponding parameters are set and the Grounded-Segment-Anything model is used for segmentation.

[0089] Grounded-Segment-Anything is a model that integrates Grounding DINO and SAM. Compared with the original sam, Grounded-Segment-Anything has the function of inputting text as a prompt. The setting of Grounded-Segment-Anything parameters has a significant impact on the segmentation effect, especially box_threshold, text_threshold, and text_prompt. Box_threshold represents the threshold parameter for object detection. If the confidence score of a bounding box is lower than this threshold, the bounding box will be filtered out. Text_threshold represents the threshold parameter for text matching. If the matching degree is lower than this threshold, the detected object will not be considered as a result of matching the text prompt. Text_prompt represents the description word of the object to be segmented. Accurate description words will make the model segmentation more accurate, such as Figure 5 and Figure 6 As shown, Figure 5 The text_prompt is "a tree in the middle", Figure 6 The text_prompt is "singlelargestareatree in the middle".

[0090] Specifically, the Grounded-Segment-Anything model includes:

[0091] Load the pre-trained Grounding DINO model checkpoint to detect the object bounding box in the image. After obtaining the object detection box, you also need to perform instance segmentation on the tree within the box.

[0092] Initialize the SAM model and use it to perform instance segmentation on the detected target to obtain the mask image of the target.

[0093] Although Grounded-Segment-Anything has strong generalization ability, the segmentation effect is unstable due to the complex background of trees and overlapping occlusion between trees in natural environments. Therefore, the SAM model needs to be fine-tuned to enable it to segment citrus trees with complex backgrounds. The fine-tuning process is as follows: 500 pictures of citrus trees with complex backgrounds were collected from 8 production areas, and the pictures were enhanced after manually annotating the outlines of citrus trees with complex backgrounds. A total of 1,000 pictures were obtained for fine-tuning the SAM model. Compared with the model before fine-tuning, the fine-tuned model has stronger general ability and ability to cope with complex backgrounds, such as Figure 5 .

[0094] Since there are multiple trees in the photo taken in the natural environment, the SAM model will segment multiple citrus trees. Therefore, it is necessary to remove the citrus trees in the background and retain the target citrus tree. Because the target citrus tree is in the center and occupies the largest photo area when shooting, this embodiment distinguishes whether the citrus tree is the background according to the size of the target bounding box of the segmented object. After traversing the bounding boxes of all segmented objects and calculating the area, the largest target detection box area is retained, which is the target to be segmented.

[0095] The mask image of the target object can be obtained by segmenting the citrus tree from the target detection frame using SAM. Due to the similarity of the tree colors in the image and the overlap between the citrus trees, the obtained mask image usually has multiple invalid isolated closed small areas, such as Fig.11 As shown, these invalid isolated closed small areas need to be processed. In this embodiment, after converting the image into a grayscale image using opencv, the image is binarized to obtain a clear boundary of the tree. The contour map of the target tree can be obtained by using the opencv built-in contour search function findContours.

[0096] Embodiment 3

[0097] The difference between this embodiment and the first embodiment is that step S5 of this embodiment specifically includes:

[0098] S501. Data preprocessing

[0099] First, three columns of data of citrus trees were collected, two columns of canopy width and one column of canopy height, and the data were stored in a CSV file. These data were standardized. This embodiment uses StandardScaler to standardize the data so that its mean is 0 and its standard deviation is 1 to ensure that the input data has a similar scale during the training process. The standardized data is divided into 80% for training and 20% for testing. This embodiment uses a fixed random seed when dividing to ensure the repeatability of the experiment and facilitate comparison with other models.

[0100] S502. Define model architecture

[0101] like Fig.12 As shown in the figure, a multimodal neural network model architecture is constructed to effectively fuse text data and image data. In the figure, image_fcl,weight(512,2048) is the fully connected layer weight matrix (image input), text_fcl,weight(128,3) is the text input, image_fcl,bias(512) is the bias vector, and (1,2) is the predicted output: volume and surface area.

[0102] The text data (two columns of canopy width and one column of canopy height) were input into a two-layer fully connected network of the multimodal neural network model, and the image data (outline images of citrus trees) were input into a pre-trained ResNet-50 model to extract features.

[0103] The text and image features are then concatenated and further processed through a fully connected network, which outputs two predicted values, namely the volume and surface area of ​​the citrus tree. Each fully connected layer is followed by a ReLU activation function to introduce nonlinearity, and a Dropout layer is used after the fully connected layer to improve the generalization ability of the model.

[0104] S503. Setting loss function

[0105] Use Mean Squared Error (MSE) as the loss function. The mean squared error loss function is defined as follows:

[0106]

[0107] Among them, y i Indicates the actual value, Represents the predicted value, and n is the number of samples. This loss function has a higher penalty for larger errors, thereby encouraging the model to make more accurate predictions.

[0108] S504, Optimization method

[0109] The AdamW optimizer is used, which combines the advantages of the Adam optimizer and weight decay regularization to prevent overfitting. This example sets the initial learning rate to 0.001. In addition, the ReduceLROnPlateau learning rate scheduler is used to halve the learning rate when the monitored loss does not improve within 10 consecutive epochs.

[0110] S505, training process

[0111] The training is set to run for a maximum of 100 epochs, with a batch size of 32. To further prevent overfitting, an early stopping mechanism is used, with the early stopping criterion set to monitor the training loss. If the training loss does not improve within 20 consecutive epochs, the training is terminated early. This method ensures that the model does not overfit the training data and can better generalize to unseen data.

[0112] This multimodal neural network model uses the pre-trained ResNet-50 model to efficiently extract deep features of the image. These features include important information such as the edge and texture of the image, so that the model can better capture the detailed features of the fruit tree, thereby improving the prediction accuracy. By increasing the number of fully connected layers and nodes, the expressiveness of the model can be enhanced, enabling it to learn more complex features. At the same time, the use of Dropout technology can effectively prevent overfitting and improve the generalization ability of the model. The learning rate scheduler can dynamically adjust the learning rate according to the change in loss during training, so that the model can converge quickly in the early stage of training, and carefully adjust the parameters in the later stage of training to avoid falling into the local optimal solution. By monitoring the loss on the validation set, when the loss of the model on the validation set no longer decreases, stop training in time, you can prevent the model from overfitting and save training time.

[0113] Embodiment 4

[0114] The difference between this embodiment and the first embodiment is that in step S6 of this embodiment, the trained multimodal neural network model is stored in the ".pth" format. According to the application scenario, it is mounted on the web page, mobile application and / or unmanned inspection vehicle. Users can choose the following three ways to input data:

[0115] 1. The user only inputs the east-west, north-south canopy width and the height of the entire plant, and the system predicts and outputs the volume and surface area values ​​of the citrus tree.

[0116] 2. In addition to inputting the east-west and north-south canopy widths and the height of the entire plant, users can take photos of the citrus tree from the front and side angles as supplementary input. This supplement enables the model to learn more about the characteristics of the tree shape from the image, making the predicted volume surface area more accurate, especially for the two irregular and complex fruit trees, Kaixinxing and Libixing.

[0117] 3. For agricultural researchers, the model can be mounted on unmanned inspection vehicles, inspection robots and other equipment. After taking pictures of citrus trees with a binocular camera, the canopy width and height of the tree can be automatically extracted to predict the volume surface area.

[0118] Traditional methods rely on a single type of data, such as using only text data or image data. This limits the accuracy and adaptability of the model because a single data source cannot fully capture the complex characteristics of citrus trees. This solution achieves a more accurate prediction of the volume and surface area of ​​fruit trees by fusing text data such as the canopy width and height of citrus trees with image data taken from different angles. This fusion of multimodal data enables the model to obtain information from multiple dimensions, improving the accuracy and robustness of the prediction.

[0119] Prediction accuracy and error control: Manual measurement and non-contact automatic measurement methods are prone to errors, such as the impact of light and climate conditions on the camera, the measurement range limitation of the depth camera, the high cost and operation complexity of the lidar, etc. The multimodal model of this scheme performs best in all indicators of volume and surface area prediction, R 2 The values ​​reached 0.98 and 0.97, the mean absolute error (MAE) was significantly reduced, and the error variance (VRD) was also effectively controlled, reflecting higher accuracy and stability.

[0120] Usage scenarios and equipment requirements: Existing technologies use equipment such as lidar, depth cameras, and high-resolution cameras, which are costly and complex to operate, and require professional technicians to operate and process data. In this solution, fruit farmers only need to take photos of fruit trees or input canopy width and height to make predictions. The equipment requirements are greatly reduced, and the operation is simple. Even fruit farmers without professional technical backgrounds can use it easily. In addition, this solution effectively improves the accuracy of image feature extraction and prediction by introducing a pre-trained ResNet model.

[0121] Data processing and model training: The existing technology data processing is complex and often requires a lot of preprocessing work, such as removing background, correcting lighting conditions, and segmenting overlapping trees. This solution uses the Grounded-Segment-Anything model and dynamic slice reconstruction algorithm to achieve efficient point cloud data segmentation and 3D reconstruction, simplifying the data processing process. At the same time, the multimodal training method is adopted, combining text features and image features, so that the model can capture the characteristics of citrus trees faster and more accurately, improving training efficiency and prediction accuracy.

[0122] Robustness and adaptability: Existing technologies are highly dependent on tree shapes or crown types and are difficult to adapt to the diverse forms of fruit trees, especially in domestic orchards where tree shapes are diverse. This solution no longer requires citrus trees to be classified by tree shape or crown. The prediction can be completed directly by inputting the canopy width and height or taking two photos. It is more adaptable and can handle diverse tree shapes, improving the robustness of the model in different application scenarios.

[0123] Embodiment 5

[0124] Traditional methods usually rely on a single type of data, which limits the accuracy and adaptability of the model. This solution achieves a more accurate prediction of the volume and surface area of ​​citrus trees by integrating text data such as the canopy width and height of citrus trees with image data taken from different angles.

[0125] When the random seed is fixed to ensure that the validation set and test set of each model are the same, a variety of machine learning and deep learning methods are compared, including linear regression, gradient boosting regression, support vector machine, K nearest neighbor regression and XGBoost and other machine learning methods and DNN (deep neural network method) without image modality and DNN+CNN (convolutional neural network feature extraction) methods. The comparison results in Table 2 show that the multimodal model (DNN+ResNet) of this scheme performs best in all indicators of volume and surface area prediction.

[0126] Table 2 Comparison of the accuracy of each model, v- represents volume prediction, s- represents surface area prediction

[0127]

[0128] The specific advantages of this solution are: Strong interpretation ability: DNN+ResNet's R 2The values ​​are the highest in both volume and surface area predictions, indicating that the model can explain the changes in the data well; high prediction accuracy: the MSE and RMSE values ​​of DNN+ResNet are the lowest in both volume and surface area predictions, showing extremely high prediction accuracy; minimum error: the MAE value of DNN+ResNet is the lowest in both volume and surface area predictions, indicating that the gap between the predicted value and the true value is the smallest.

[0129] like Fig.13 The training loss of the multimodal model (with image features) of this scheme is significantly lower than that of DNN (text features only), and the convergence speed is faster. This shows that the introduction of image features enables the model to capture the characteristics of citrus trees more quickly, thereby improving the training efficiency. By introducing the shape contours of citrus trees from two angles, this model provides additional spatial information, which helps to more accurately estimate the volume and surface area of ​​citrus trees. This shows that when dealing with data with complex spatial structures, combining image features can significantly improve the performance of multimodal models.

[0130] We collected Libi-type citrus trees at the Ganzhou Citrus Research Institute, collected standardized natural round-headed citrus trees at the Lvmeng Company, and collected naturally grown round-headed and open-hearted citrus trees at the National Citrus Germplasm Resource Garden and Luzhou Orchard. A total of 40 trees (crown width range 0.53-4.29m, plant height range 1.25-2.95m) were collected. Point cloud and photo information were collected for each tree. These trees were not in the training set. We put these trees into the trained multimodal model to judge the model efficiency in real scenarios.

[0131] Table 3 Comparison of citrus tree volume prediction accuracy

[0132]

[0133]

[0134] In Table 3, v- represents volume, s- represents surface area, d- represents DNN model, dr- represents the DNN+ResNet model we trained, MAE represents mean absolute error, MRD represents mean relative error, and VRD represents relative error variance.

[0135] It can be seen from Table 3 that the DNN+ResNet multimodal model of this scheme has improved the R2 of predicted volume and surface area by 0.02 compared with the DNN method without image modality, the error is greatly reduced and the error variance distribution is more uniform, which means that the DNN+ResNet model has higher prediction accuracy and smaller error.

[0136] Further analysis found that the R2 values ​​of the DNN+ResNet multimodal model in the volume surface area prediction of all tree shapes (Li Bi shape, Kai Xin shape and round head shape) were significantly higher than those of the traditional DNN model (the R2 of the Li Bi shape increased by 0.10 and 0.14 respectively, and the Kai Xin shape increased by 0.07 and 0.08 respectively), the mean absolute error was reduced by more than half, and the mean relative error and relative error variance were significantly reduced, which indicates that the model is more applicable under different tree shapes and has better generalization ability.

[0137] like Fig.14 As shown in the figure, whether it is the more regular open-heart shape and round-head shape, or the more complex Li Bi shape, the DNN+ResNet multimodal model shows high prediction accuracy and stability, reflecting its strong adaptability under diversified tree shapes. This adaptability enables the model to better handle different types and shapes of citrus trees in practical applications, and improves the comprehensiveness and accuracy of the prediction of citrus tree volume and surface area.

[0138] The above are only embodiments of the present invention. The invention is not limited to the fields involved in this implementation case. The common sense such as the known specific structures and characteristics in the scheme is not described in detail here. The ordinary technicians in the relevant field know all the common technical knowledge in the technical field to which the invention belongs before the application date or the priority date, can obtain all the existing technologies in the field, and have the ability to apply the conventional experimental means before that date. The ordinary technicians in the relevant field can improve and implement this scheme in combination with their own abilities under the enlightenment given by this application. Some typical known structures or known methods should not become obstacles for ordinary technicians in the relevant field to implement this application. It should be pointed out that for those skilled in the art, without departing from the structure of the present invention, several deformations and improvements can be made, which should also be regarded as the protection scope of the present invention, which will not affect the effect of the implementation of the present invention and the practicality of the patent. The protection scope required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to explain the content of the claims.

Claims

1. A multimodal fruit tree surface area and volume prediction method, characterized in that: It includes the following: S1. Preparation of text data set: collecting point cloud data of fruit trees in a preset area, extracting the north-south width, east-west width and plant height of each fruit tree point cloud data as text feature values; S2. Image data set preparation: For each fruit tree, the point cloud data is extracted from the front and side angles to obtain the contour map as the image feature value; Specifically include: The Grounded-Segment-Anything model is used to extract the mask of the fruit tree point cloud data; the opencv computer vision library is used to extract the mask contour and obtain the contour map; Or for the image of a single fruit tree taken in a natural environment, use the Grounded-Segment-Anything model to segment the target fruit tree outline, and use OpenCV to remove the isolated areas in the fruit tree image segmentation in the natural environment to obtain the outline map; The extracted contour image is used as the image feature value; S3, volume value acquisition, using dynamic slicing and reconstruction algorithm to reconstruct the point cloud data of a single fruit tree and obtain the volume value; S4, obtaining the surface area value, using the AlphaShape algorithm to reconstruct the surface of the fruit tree point cloud data, and obtaining the surface area value based on the reconstructed surface; S5. Model training: building a multimodal neural network model architecture based on text features and image features; establishing a data set based on the text feature values, front and side image feature values, volume values ​​and surface area values ​​of each fruit tree; performing multimodal training based on the data set and saving the trained multimodal neural network model; specifically including: S501. Data preprocessing The text feature value data of fruit trees were collected and stored in a CSV file; the text feature value data were standardized so that the mean value was 0 and the standard deviation was 1; The standardized data is divided into 80% for training and 20% for testing; S502. Define model architecture Build a multimodal neural network model architecture, input text feature value data into a two-layer fully connected network of the multimodal neural network model, and input image feature value data into a pre-trained ResNet-50 model to extract features; Then the text and image features are concatenated and processed through a fully connected network, and finally two predicted values ​​are output, which are the volume value and surface area value of the citrus tree; S503. Setting loss function Using mean square error as the loss function, the mean square error loss function is defined as follows: Among them, y i Indicates the actual value, represents the predicted value, n is the number of samples; S504, Optimization method The AdamW optimizer is used with an initial learning rate of 0.

001. The ReduceLROnPlateau learning rate scheduler is also used to halve the learning rate when the monitored loss does not improve for 10 consecutive epochs. S505, training process The training was set to run for a maximum of 100 epochs, with a batch size of 32. The early stopping mechanism was used, and the early stopping criterion was set to monitor the training loss. If the training loss did not improve within 20 consecutive epochs, the training was terminated early. S6. Model application, obtaining the north-south width of the canopy, the east-west width of the canopy and the height of the whole plant input by the user, and predicting the volume value and surface area value of the fruit tree; Alternatively, the user may input photos of the front and side angles of the fruit tree as supplementary input to predict the volume and surface area of ​​the fruit tree.

2. The multimodal fruit tree surface area and volume prediction method according to claim 1, characterized in that: In step S2, the Grounded-Segment-Anything model is obtained by integrating Grounding DINO and SAM; Using the Grounded-Segment-Anything model includes: Load the pre-trained Grounding DINO model checkpoint for detecting object bounding boxes in images; The size of the target bounding box of the segmented object distinguishes whether the citrus tree is the background. After traversing the bounding boxes of all segmented objects and calculating the area, the largest target detection box area is retained, which is the target to be segmented; Initialize the SAM model and use it to perform instance segmentation on the detected target to obtain the target mask image; process the invalid isolated closed areas in the mask image to obtain the contour map of the target tree.

3. The multimodal fruit tree surface area and volume prediction method according to claim 1, characterized in that: The fruit trees are citrus trees.

4. A multimodal fruit tree surface area and volume prediction system, characterized in that: Use the method according to any one of claims 1 to 3.

Images