An orchard single fruit picking optimization method based on multi-modal data fusion

Abstract

This invention relates to an optimized method for single-fruit harvesting in orchards based on multimodal data fusion. It involves simultaneously acquiring image information of the fruit, local environmental data, and tree physiological indicators using multimodal sensors, and preprocessing these data. Taking into account the heterogeneous nature of different modalities, independent feature extraction networks are used to obtain deep features for each modality, constructing a heterogeneous modal deep fusion network. A cross-domain attention mechanism dynamically integrates these features, effectively capturing the complex correlations affecting fruit maturity and generating fused features. Based on these features, the model can predict the real-time maturity of individual fruits. Combining external conditions, a multi-objective optimization algorithm is used to calculate the harvesting priority score for each fruit, generating an executable harvesting sequence to guide robots or manual harvesting in an orderly manner. This invention significantly improves the accuracy of maturity prediction and the overall efficiency of harvesting operations, providing effective support for refined management and intelligent operation in orchards.

Description

Technical Field

[0001] This invention belongs to the field of agricultural information intelligence technology, specifically relating to an optimization method for single fruit harvesting in orchards based on multimodal data fusion. Background Technology

[0002] Traditional orchard fruit harvesting decisions rely primarily on human experience or automated methods based on a single visual modality. These methods typically assess ripeness roughly by analyzing external characteristics such as fruit color, size, or shape, making it difficult to accurately evaluate internal quality and optimal harvesting time. Due to a lack of comprehensive consideration of the tree's physiological state and growing environment, the predictive results of such methods often exhibit significant biases, failing to meet the higher demands of modern precision agriculture for consistent fruit quality and efficient harvesting.

[0003] Currently, orchard harvesting decisions mainly rely on human experience or rough predictions of maturity based on single-modal data such as RGB images or single-spectral data. Existing technologies have the following shortcomings:

[0004] Relying solely on visual or environmental data fails to capture physiological information reflecting the fruit's internal quality, resulting in inaccurate maturity predictions, particularly regarding internal quality indicators.

[0005] The decision-making dimension is singular. Existing automated detection typically only outputs maturity level, but lacks dynamic, globally optimized picking priority output, failing to guide picking robots or teams to achieve the logistics goals of optimal efficiency and consistent quality. Heterogeneous data fusion efficiency is low. Current technologies lack efficient and adaptive fusion mechanisms for three highly heterogeneous modalities: high-dimensional image spatial data, low-dimensional environmental time-series data, and sparse physiological indicator data.

[0006] Furthermore, most existing harvesting optimization solutions focus on single aspects such as maturity prediction or route planning, lacking a comprehensive decision-making mechanism that integrates prediction results with external constraints such as harvesting efficiency, market supply and demand, and logistical factors. This results in the inability to maximize the overall efficiency of harvesting operations, making it difficult to dynamically adapt to the complex orchard production environment and changing market demands, thus hindering further improvements in the level of intelligent orchard management. Summary of the Invention

[0007] According to a first aspect of the present invention, the present invention claims protection for an orchard single-fruit harvesting optimization method based on multimodal data fusion, comprising:

[0008] S1, Multimodal sensors are used to collect and preprocess multimodal data of the fruits to be harvested in the orchard to obtain multimodal fruit data;

[0009] S2, perform heterogeneous modal feature extraction independently on the multimodal fruit data to obtain heterogeneous multimodal fruit features;

[0010] S3, construct a heterogeneous modal deep fusion network, and input the heterogeneous multimodal fruit features into the heterogeneous modal deep fusion network to obtain deep fused fruit features;

[0011] S4. Construct a fully connected prediction network, input the deeply fused fruit features into the fully connected prediction network, and obtain the single fruit maturity prediction value of the fruit to be picked.

[0012] S5. Based on the predicted maturity and external constraints, a multi-objective optimization algorithm is used to calculate the single-fruit picking priority score of the fruit to be picked, and the picking operation is performed according to the single-fruit picking priority score.

[0013] Furthermore, S1 also includes:

[0014] The multimodal data includes at least image modal data, environmental modal data, and tree physiological modal data;

[0015] The image modal data includes RGB images of the fruit to be picked, RGB-D depth images, and multispectral or hyperspectral images;

[0016] The environmental modal data is local microenvironmental data, including at least the temperature, humidity, and light intensity around a single branch, and the environmental modal data is formed into a time-series feature sequence.

[0017] The tree physiological modal data are physiological indicators reflecting the health of fruit trees and water stress, including at least leaf temperature, leaf area index, and trunk microdeformation data.

[0018] Furthermore, the preprocessing in S1 further includes:

[0019] Target segmentation, 6D pose fruit localization, and background denoising.

[0020] Furthermore, S2 also includes:

[0021] Heterogeneous modal features are independently extracted from the image modal data, environmental modal data, and tree physiological modal data to obtain image features, environmental features, and physiological features.

[0022] Furthermore, the method also includes:

[0023] When extracting heterogeneous modal features independently from the image modal data, CNN or VisionTransformer network is used to extract spatial features of fruit color, shape, and surface defects as image features;

[0024] When independently extracting heterogeneous modal features from the environmental modal data, RNN or Transformer is used to extract the long-term dependency and trend features of the environmental modal data as environmental features;

[0025] When extracting heterogeneous modal features independently from the tree physiological modality data, a sparse coding network or a fully connected network is used to extract the potential features of physiological indicators as physiological features.

[0026] Furthermore, S3 also includes:

[0027] Using the image features as the dominant features, auxiliary information most relevant to fruit ripeness is extracted from the environmental and physiological features through gating mechanisms or multi-head attention.

[0028] A physiological-temporal perception module is constructed, which uses the environmental features to modulate the weights of the physiological features to reflect the real-time impact of the environment on the physiological state, and generates unified, high-dimensional deep fusion fruit features.

[0029] Furthermore, the method also includes:

[0030] The extracted image features, environmental features, and physiological features are input into the cross-domain attention fusion module;

[0031] Construct a physiological-temporal awareness module: use the environmental features to modulate the weights of the physiological features to correct the maturity prediction bias based on image features;

[0032] A task-aware weighted subnetwork is constructed to receive maturity detection task objectives, dynamically learn and output attention weights for each modality, and generate unified, high-dimensional deep fusion fruit features.

[0033] Furthermore, S4 also includes:

[0034] Based on the deep fusion of fruit characteristics, predict the internal quality indicators of the fruit and its harvesting window period; input the deep fusion of fruit characteristics into the prediction head of the fully connected prediction network, and output the single-fruit maturity prediction value of the fruit to be harvested.

[0035] Furthermore, S5 also includes:

[0036] Based on the predicted maturity value of a single fruit, and in conjunction with external constraints, a multi-objective optimization algorithm is used to calculate the priority score P:

[0037] P = W M · S Maturity + W E · E Efficiency + W Q Q Quality+ C

[0038] Among them, S Maturity E is the predicted maturity value for a single fruit. Efficiency It is a parameter related to harvesting efficiency, Q Quality C is the quality uniformity parameter, and W is the market logistics constraint coefficient. M W E W Q The corresponding weights for each element.

[0039] Furthermore, S5 also includes:

[0040] Based on the multiple single-fruit picking priority scores, output a picking priority list containing all fruits on a single branch;

[0041] According to the picking priority list, the picking robot or manual picking team will prioritize picking the fruits or branches with the highest P score.

[0042] Compared with the prior art, the beneficial results of the present invention are:

[0043] High accuracy in maturity prediction: It deeply integrates three heterogeneous modal data, namely image, environment and tree physiology, to comprehensively reflect the fruit status and significantly improve the accuracy of predicting internal quality indicators.

[0044] Achieving refined dynamic decision-making: It realizes the output of harvesting priorities at the single fruit / single branch level, transforming maturity prediction results into directly executable business optimization decisions, filling the gap in the decision-making level of existing technologies.

[0045] High robustness: The cross-domain attention fusion mechanism can effectively overcome the interference caused by single-modal data (such as image occlusion or environmental sensor drift), making the system more stable and reliable.

[0046] Optimize harvesting efficiency and quality: Prioritized harvesting strategies can effectively improve harvesting efficiency and ensure the uniformity of product quality across harvested batches. Attached Figure Description

[0047] Figure 1 The flowchart illustrates a method for optimizing single-fruit harvesting in orchards based on multimodal data fusion, as claimed in this embodiment of the invention. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0049] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0050] According to the first embodiment of the present invention, referring to Figure 1 This invention claims protection for a method for optimizing single-fruit harvesting in orchards based on multimodal data fusion, comprising:

[0051] S1, Multimodal sensors are used to collect and preprocess multimodal data of the fruits to be harvested in the orchard to obtain multimodal fruit data;

[0052] S2, perform heterogeneous modal feature extraction independently on the multimodal fruit data to obtain heterogeneous multimodal fruit features;

[0053] S3, construct a heterogeneous modal deep fusion network, and input the heterogeneous multimodal fruit features into the heterogeneous modal deep fusion network to obtain deep fused fruit features;

[0054] S4. Construct a fully connected prediction network, input the deeply fused fruit features into the fully connected prediction network, and obtain the single fruit maturity prediction value of the fruit to be picked.

[0055] S5. Based on the predicted maturity and external constraints, a multi-objective optimization algorithm is used to calculate the single-fruit picking priority score of the fruit to be picked, and the picking operation is performed according to the single-fruit picking priority score.

[0056] Furthermore, S1 also includes:

[0057] The multimodal data includes at least image modal data, environmental modal data, and tree physiological modal data;

[0058] The image modal data includes RGB images of the fruit to be picked, RGB-D depth images, and multispectral or hyperspectral images;

[0059] The environmental modal data is local microenvironmental data, including at least the temperature, humidity, and light intensity around a single branch, and the environmental modal data is formed into a time-series feature sequence.

[0060] The tree physiological modal data are physiological indicators reflecting the health of fruit trees and water stress, including at least leaf temperature, leaf area index, and trunk microdeformation data.

[0061] In this embodiment, the image modality (M) Image Acquire RGB images, depth images (RGB-D), and multispectral / hyperspectral images of the fruit. Preprocessing includes target segmentation, fruit localization (6D pose), and background denoising.

[0062] Environmental Modality (M) Env ): Collect local microenvironment data, such as temperature, humidity, and light intensity around a single branch, to form a time-series characteristic sequence.

[0063] Tree physiological modes (M Phys ): Collect physiological indicators reflecting the health of fruit trees and water stress, such as leaf temperature, leaf area index, and trunk microdeformation data.

[0064] Furthermore, the preprocessing in S1 further includes:

[0065] Target segmentation, 6D pose fruit localization, and background denoising.

[0066] Data collection platforms, such as picking robots or inspection vehicles equipped with sensors, collect data on individual branches of peach trees. The data is shown in Table 1.

[0067] Table 1 Single Branch Data Collection Table

[0068]

[0069] The target variables for harvesting include the predicted value of the soluble solids content (SSC) of peaches as an indicator of maturity and the expected market price.

[0070] Furthermore, S2 also includes:

[0071] Heterogeneous modal features are independently extracted from the image modal data, environmental modal data, and tree physiological modal data to obtain image features, environmental features, and physiological features.

[0072] Furthermore, the method also includes:

[0073] When extracting heterogeneous modal features independently from the image modal data, CNN or VisionTransformer network is used to extract spatial features of fruit color, shape, and surface defects as image features;

[0074] When independently extracting heterogeneous modal features from the environmental modal data, RNN or Transformer is used to extract the long-term dependency and trend features of the environmental modal data as environmental features;

[0075] When extracting heterogeneous modal features independently from the tree physiological modality data, a sparse coding network or a fully connected network is used to extract the potential features of physiological indicators as physiological features.

[0076] In this embodiment, image features (F) Image Spatial features such as fruit color, shape, and surface defects are extracted using networks such as CNN or Vision Transformer.

[0077] Environmental characteristics (F) Env ): Use RNN or Transformer to extract long-term dependencies and trend features from environmental time series data.

[0078] Physiological characteristics (F Phys ): Employing sparse coding networks or fully connected networks to extract latent features of physiological indicators.

[0079] Furthermore, S3 also includes:

[0080] Using the image features as the dominant features, auxiliary information most relevant to fruit ripeness is extracted from the environmental and physiological features through gating mechanisms or multi-head attention.

[0081] A physiological-temporal perception module is constructed, which uses the environmental features to modulate the weights of the physiological features to reflect the real-time impact of the environment on the physiological state, and generates unified, high-dimensional deep fusion fruit features.

[0082] In this embodiment, a cross-domain attention fusion mechanism is employed, specifically designed to unify and fuse spatial features (images), temporal features (environment), and sparse features (physiology).

[0083] Fusion process: F Image As a dominant feature, through gating mechanisms or multi-head attention, from F Env and F Phys Extract the auxiliary information most relevant to fruit ripeness. For example, design a "physiological-temporal sensing module" to use environmental temporal features F. Env To modulate physiological characteristics F Phys The weights are assigned to reflect the real-time impact of the environment on physiological states. Ultimately, a unified, high-dimensional deep fusion feature F is generated. fusion .

[0084] Furthermore, the method also includes:

[0085] The extracted image features, environmental features, and physiological features are input into the cross-domain attention fusion module;

[0086] Construct a physiological-temporal awareness module: use the environmental features to modulate the weights of the physiological features to correct the maturity prediction bias based on image features;

[0087] A task-aware weighted subnetwork is constructed to receive maturity detection task objectives, dynamically learn and output attention weights for each modality, and generate unified, high-dimensional deep fusion fruit features.

[0088] In this embodiment, F Image We used a deep convolutional network (ResNet50) to extract the color, shape, and occlusion features of individual fruits.

[0089] F Env : Use a gated recurrent unit (GRU) network to extract environmental temporal features over the past 24 hours.

[0090] F Phys : Use a fully connected network (FCN) to encode features of physiological indicators.

[0091] F will be extracted during deep fusion Image (Dimension D) I ), F Env (Dimension D) E ), F Phys (Dimension D) P Input the cross-domain attention fusion module.

[0092] Physiological-temporal awareness module: utilizing F Env To modulate F Phys The weight of physiological characteristics F is increased when high temperature and humidity in the environment lead to increased water stress in the tree. Phys The contribution weights are used to correct the maturity prediction bias based on image features.

[0093] The task-aware weighted subnetwork receives the task objective of "maximizing SSC uniformity", dynamically learns and outputs the attention weight α for each modality. I ,α E ,α P For example, in the pursuit of high sugar content, α I The weight of (color / appearance) was slightly reduced, while α P The weight of (water / physiological state) was increased.

[0094] Output: Unified deep fusion feature F fusion .

[0095] Furthermore, S4 also includes:

[0096] Based on the deep fusion of fruit features, predict the internal quality indicators of the fruit and its harvesting window period; input the deep fusion of fruit features into the constructed and trained fully connected prediction network, and through the nonlinear mapping of the network, directly output the predicted maturity value of a single fruit to be harvested, such as the predicted value of soluble solids content (SSC). The prediction network is trained based on a large amount of sample data with known true maturity labels, and is completed by minimizing the loss function between the predicted value and the true value.

[0097] In this embodiment, the fully connected prediction network is a neural network structure known to those skilled in the art, typically composed of one or more fully connected layers connected sequentially, with non-linear activation functions such as ReLU applied between layers. The network's input dimension matches the dimension of the deeply fused fruit features, and the output dimension corresponds to the dimension of the prediction target, such as a maturity index. The specific structure of the network, such as the number of layers and the number of neurons per layer, can be determined through conventional supervised learning training on a labeled training dataset.

[0098] The fusion is input into the FCN prediction head, which outputs the real-time SSC prediction value for a single fruit and the SSC growth rate for the next 3 days.

[0099] Furthermore, S5 also includes:

[0100] Based on the predicted maturity value of a single fruit, and in conjunction with external constraints, a multi-objective optimization algorithm is used to calculate the priority score P:

[0101] P = WM · SMaturity + WE · EEfficiency + WQ · QQuality + C

[0102] Among them, SMaturity is the predicted maturity value of a single fruit, EEfficiency is a parameter related to harvesting efficiency, which may include factors such as the spatial aggregation density of fruits, the density of adjacent fruits on the same branch, the expected path length or time for the harvesting robot arm to move from the current position to the target fruit, and the degree of shading of the fruit in the canopy, which affect the time or difficulty of single fruit harvesting operations, QQuality is a quality uniformity parameter, C is a market logistics constraint coefficient, and WM, WE, and WQ are the corresponding weights of each element.

[0103] Furthermore, S5 also includes:

[0104] Based on the multiple single-fruit picking priority scores, output a picking priority list containing all fruits on a single branch;

[0105] According to the picking priority list, the picking robot or manual picking team will prioritize picking the fruits or branches with the highest P score.

[0106] In this embodiment, the picking priority score P is dynamically calculated using the following formula:

[0107] P = W SSC · SSC Predict + W Rate · R Growth + W Density · D Branch - C Risk

[0108] in:

[0109] SSC Predict : Predicted SSC value.

[0110] R Growth SSC growth rate in the next 3 days (if the growth rate is fast, the priority is reduced and we will wait).

[0111] D Branch Fruit density per branch (higher density means higher harvesting efficiency and higher priority).

[0112] C Risk : Risk of pests or diseases or cracking (higher risk means higher priority and earlier harvesting is required).

[0113] W weight: dynamically adjusted based on market objectives. For example, if the market urgently needs high-sugar fruits, then W... SSC Increase; if there is insufficient labor for harvesting, W Density Increase.

[0114] Decision Output: The algorithm outputs a picking priority list containing all fruits {P1, P2, P3, ...} on a single branch. The picking robot or manual picking team uses this list to prioritize picking the fruits or branches with the highest P score.

[0115] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0116] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0117] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A method for optimizing single-fruit harvesting in orchards based on multimodal data fusion, characterized in that, include: S1, Multimodal sensors are used to collect and preprocess multimodal data of the fruits to be harvested in the orchard to obtain multimodal fruit data; S2, perform heterogeneous modal feature extraction independently on the multimodal fruit data to obtain heterogeneous multimodal fruit features; S3, construct a heterogeneous modal deep fusion network, and input the heterogeneous multimodal fruit features into the heterogeneous modal deep fusion network to obtain deep fused fruit features; S4. Construct a fully connected prediction network, input the deeply fused fruit features into the fully connected prediction network, and obtain the single fruit maturity prediction value of the fruit to be picked. S5. Based on the predicted maturity and external constraints, a multi-objective optimization algorithm is used to calculate the single-fruit picking priority score of the fruit to be picked and the picking operation is performed according to the single-fruit picking priority score. The S1 further includes: The multimodal data includes at least image modal data, environmental modal data, and tree physiological modal data; The image modal data includes RGB images of the fruit to be picked, RGB-D depth images, and multispectral or hyperspectral images; The environmental modal data is local microenvironmental data, including at least the temperature, humidity, and light intensity around a single branch, and the environmental modal data is formed into a time-series feature sequence. The tree physiological modal data are physiological indicators reflecting the health of fruit trees and water stress, including at least leaf temperature, leaf area index, and trunk microdeformation data; The S2 further includes: Heterogeneous modal features are independently extracted from the image modal data, environmental modal data, and tree physiological modal data to obtain image features, environmental features, and physiological features; Also includes: The extracted image features, environmental features, and physiological features are input into the cross-domain attention fusion module; Construct a physiological-temporal awareness module: use the environmental features to modulate the weights of the physiological features to correct the maturity prediction bias based on image features; A task-aware weighted subnetwork is constructed to receive maturity detection task objectives, dynamically learn and output attention weights for each modality, and generate unified, high-dimensional deep fusion fruit features.

2. The orchard single-fruit harvesting optimization method based on multimodal data fusion according to claim 1, characterized in that, The preprocessing described in S1 further includes: Target segmentation, 6D pose fruit localization, and background denoising.

3. The orchard single-fruit harvesting optimization method based on multimodal data fusion according to claim 1, characterized in that, Also includes: When extracting heterogeneous modal features independently from the image modal data, CNN or Vision Transformer networks are used to extract spatial features of fruit color, shape, and surface defects as image features. When independently extracting heterogeneous modal features from the environmental modal data, RNN or Transformer is used to extract the long-term dependency and trend features of the environmental modal data as environmental features; When extracting heterogeneous modal features independently from the tree physiological modality data, a sparse coding network or a fully connected network is used to extract the potential features of physiological indicators as physiological features.

4. The orchard single-fruit harvesting optimization method based on multimodal data fusion according to claim 1, characterized in that, The S3 further includes: Using the image features as the dominant features, auxiliary information most relevant to fruit ripeness is extracted from the environmental and physiological features through gating mechanisms or multi-head attention. A physiological-temporal perception module is constructed, which uses the environmental features to modulate the weights of the physiological features to reflect the real-time impact of the environment on the physiological state, and generates unified, high-dimensional deep fusion fruit features.

5. The orchard single-fruit harvesting optimization method based on multimodal data fusion according to claim 4, characterized in that, The S4 further includes: Based on the deep fusion of fruit characteristics, predict the internal quality indicators of the fruit and its harvesting window period; input the deep fusion of fruit characteristics into the prediction head of the fully connected prediction network, and output the single-fruit maturity prediction value of the fruit to be harvested.

6. The orchard single-fruit harvesting optimization method based on multimodal data fusion according to claim 4, characterized in that, The S5 also includes: Based on the predicted maturity value of a single fruit, and in conjunction with external constraints, a multi-objective optimization algorithm is used to calculate the picking priority score for a single fruit: P=W M ·S Maturity +W E ·E Efficiency +W Q ·Q Quality +C Where P is the priority score for picking a single fruit, and S Maturity E is the predicted maturity value for a single fruit. Efficiency It is a parameter related to harvesting efficiency, Q Quality C is the quality uniformity parameter, and W is the market logistics constraint coefficient. M W E W Q The corresponding weights for each element.

7. The orchard single-fruit harvesting optimization method based on multimodal data fusion according to claim 4, characterized in that, The S5 also includes: Based on the multiple single-fruit picking priority scores, output a picking priority list containing all fruits on a single branch; According to the picking priority list, the picking robot or manual picking team will prioritize picking the fruit or branch with the highest picking priority score.

Images