Camellia oleifera fruit maturity discrimination and precise picking system and method fusing multi-sensor perception

Abstract

The application provides a camellia oleifera fruit maturity discrimination and accurate picking system and method fusing multi-sensor perception, comprising: a partition acquisition module, which performs data acquisition, verification and structure unification on each sampling partition to obtain a multi-source unified data set; a robust preprocessing module, which performs fruit body positioning, geometric normalization, illumination normalization and reflectivity ratio feature extraction to obtain a structured robust representation; a maturity discrimination module, which constructs a maturity discrimination model to output maturity latent variable representation, maturity estimation and uncertainty estimation; a maturity prediction module, which constructs a graph space-time model to obtain a maturity probability distribution; a picking decision module, which formulates an executable picking plan; and a picking execution module, which executes the picking plan using a picking robot and performs model optimization. The application can deeply mine fruit physical representation, realize accurate fruit discrimination and picking, improve picking quality and efficiency, and reduce waste.

Description

Technical Field

[0001] This invention relates to the fields of image recognition and fruit harvesting technology, and in particular to a system and method for determining the maturity of camellia oleifera fruit and for precise harvesting by integrating multi-sensor perception. Background Technology

[0002] Camellia oleifera fruit, known as the "Oriental Golden Fruit," is a complete crop with no waste. It boasts extremely high economic, nutritional, medicinal, ecological, and industrial value, making it a core resource of distinctive woody oil crops in southern my country. Within the appropriate maturity window, priority should be given to harvesting Camellia oleifera fruits that have reached the target maturity standard, reducing the mixing of immature and overripe fruits and ensuring more consistent raw material quality across harvested batches. Current methods for judging and harvesting the maturity of camellia fruit mainly rely on sensory and traditional experience, visual and image recognition, and sensory perception. However, these methods have the following problems: 1) Data acquisition is multimodal, isolated, and uncalibrated; visual algorithms struggle to cope with complex lighting and shadows in the field, resulting in features containing a large amount of environmental noise rather than the essence of the fruit; 2) Maturity judgment is often a simple binary or tri-class classification, lacking a measure of the confidence level of the output results and failing to integrate and align intrinsic physical indicators such as spectral density and hardness; 3) Judgments are static, ignoring spatiotemporal evolution, and can only judge the fruit at the present moment. They cannot integrate environmental data such as meteorology and accumulated temperature to make spatiotemporal dynamic predictions of the entire orchard's ripening process, leading to passive decision-making; 4) Harvesting strategies are simple, harvesting once the threshold is exceeded, making it difficult to handle the trade-off between the optimal window, risk, and multiple objectives. Summary of the Invention

[0003] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a system and method for judging the maturity of camellia fruit and for precise harvesting by integrating multi-sensor perception. By deeply mining the physical characteristics of the fruit through multi-sensor fusion perception and combining it with a spatiotemporal graph model for forward-looking prediction, the system can achieve precise fruit judgment and harvesting, improve harvesting quality and efficiency, and reduce waste.

[0004] To achieve the above objectives, the present invention provides the following solution: a system for determining the maturity of camellia fruit and for precise harvesting that integrates multi-sensor perception, comprising: The partitioned data acquisition module is used to divide the sampling area into multiple sampling partitions, and to collect, verify and unify the data structure of each sampling partition to obtain a multi-source unified dataset. The robust preprocessing module is used to perform fruit body localization, geometric normalization and illumination normalization based on the multi-source unified dataset to obtain the normalized corrected fruit body contour, and calculate the reflectance ratio feature of the normalized corrected fruit body contour to construct a structured robust characterization. The maturity discrimination module is used to construct an encoder structure using the structured robust representation, perform self-supervised cross-domain learning on the encoder structure, and introduce MC Dropout and total loss function to obtain a maturity discrimination model, outputting a maturity latent variable representation, maturity estimate and uncertainty estimate. The maturity prediction module is used to construct a graph spatiotemporal model based on the sampling partitions, integrate the maturity discrimination results into the graph spatiotemporal model to perform partition prediction, and obtain the maturity probability distribution. The harvesting decision module is used to define executable objectives, execution constraints, and objective functions based on the maturity probability distribution, and to divide risk batches to obtain an executable harvesting plan; The harvesting execution module is used to execute the harvesting plan using a harvesting robot and collect harvesting evidence in real time for model optimization. The partitioned data collection module, the robust preprocessing module, the maturity discrimination module, the maturity prediction module, the harvesting decision module, and the harvesting execution module are interconnected.

[0005] Optionally, the partition acquisition module includes: Spatial hierarchical units are used to grid the sampling area to obtain multiple sampling zones. For each sampling zone, different sampling intensities, sampling times, and sampling point IDs are set. The data sampling unit is used to acquire near-field RGB images, multispectral reflectance maps, thermal infrared maps, depth maps, fruit physical indicators, spatial locations, and climate data and timestamps based on the sampling partition using calibrated sensors to obtain a multi-source dataset; The data verification unit is used to perform integrity checks, time consistency checks, spatial consistency checks, and calibration information checks based on the multi-source dataset to obtain a multi-source synchronized dataset. The unified structural unit is used to generate a unique sample ID, partition ID, fruit tree ID, collection timestamp, weather tag, temperature in degrees Celsius, relative humidity, illuminance, RGB image path, multispectral path, thermal infrared path, depth path, multispectral exposure, occlusion level, fruit physical indicators, dielectric indicators, and hardness for each sample based on the multi-source synchronous dataset, thereby obtaining a multi-source unified dataset including sample metadata.

[0006] Optionally, the robust preprocessing module includes: The fruit body localization unit is used to obtain the coarse outline of the fruit body based on the multi-source unified dataset by using threshold segmentation. For the coarse outline of the fruit body, it retains the regions in the largest connected region or the foreground connected region whose size matches the scale of the fruit body, and removes the long strip and thin sheet regions to obtain the refined outline of the fruit body. Then, it identifies the bounding box and principal axis direction of the refined outline of the fruit body and generates the transformation parameters of the normalized coordinate system of the fruit body. The normalization unit is used to perform fruit body cropping and unified view standardization based on the refined fruit body outline to obtain a geometrically normalized fruit body outline. Then, shadow division and color consistency pull-back are performed on the geometrically normalized fruit body outline to obtain a normalized corrected fruit body outline. The reflectance conversion unit is used to estimate the reflectance based on the normalized corrected fruit body contour by using the white field and dark field to count the digital output of the multispectral sensor, to obtain the uniform band reflectance vector of each fruit body region, and to construct the reflectance ratio feature based on the uniform band reflectance vector. The alignment and fusion unit is used to align and fuse the normalized corrected fruit body contour and the reflectance ratio feature to obtain a structured robust characterization.

[0007] Optionally, the normalization unit includes: The geometric normalization subunit is used to perform fixed boundary cropping of the fruit body based on the refined fruit body outline using the bounding box to obtain a fruit body cropping image, and to perform radial or perspective transformation on the fruit body cropping image using the transformation parameters of the fruit body normalized coordinate system to obtain a fruit body image with a unified standard view. Then, based on the fruit body image with a unified standard view, the main axis direction of the fruit body is aligned and the rotation angle is obtained to obtain the geometric normalized fruit body outline. The illumination normalization subunit is used to estimate the local brightness field within the fruit body region using the geometrically normalized fruit body contour, and to mark pixels with significantly low brightness and a correlation decrease with the surrounding neighborhood as shadow candidates to obtain shadow regions. Based on the shadow regions, the fruit body region is divided into shadow pixels and non-shadow pixels. The non-shadow pixels are used as the color statistical benchmark to perform color consistency pull-back of shadow pixels, and to regenerate the normalized color map of the fruit body region to obtain the normalized corrected fruit body contour.

[0008] Optionally, the maturity determination module includes: The encoder unit is used to select a CNN-style feature extraction network as the RGB encoder, select an MLP convolutional encoder as the spectral encoder, and then weight and fuse the RGB encoder and the spectral encoder according to the occlusion level to obtain the encoder structure. The self-supervised learning unit is used to construct positive samples based on the structured robust representation by using a spatiotemporal nearest neighbor construction method and a physical labeling guidance method, and to construct negative samples based on samples in different sampling partitions. Based on the positive and negative samples, cross-domain invariant features are learned through contrastive loss, and modality consistency loss is introduced to complete the self-supervised cross-domain contrastive learning of the encoder structure. The physical alignment unit is used to select the regression network for the physical indicators of the fruit to define the physical mapping head, construct the physical mapping function using the physical mapping head, and introduce the reverse consistency constraint into the physical mapping function to obtain the physical regression loss. The total loss unit is used to weight and fuse the contrastive learning loss, multimodal consistency loss, physical regression loss, and maturity supervision loss to obtain the total loss function; The discrimination estimation unit is used to introduce MC Dropout and the total loss function into the encoder structure to obtain a maturity discrimination model and output maturity discrimination results including maturity latent variable representation, maturity estimation and uncertainty estimation.

[0009] Optionally, the maturity prediction module includes: The garden map unit is used to construct a node set and an edge set based on the sampling partition, taking each partition as a node, setting the edge weight of each edge, and then constructing node features for each node to obtain an initial graph model; wherein, the edge weight is used to express the probability that the maturity evolution between nodes is affected by the microenvironment. The probability prediction unit is used to define the upper and lower limits of maturity to obtain a maturity window. Based on the maturity window, the maturity discrimination result is integrated into the initial graph model for regression training and variance learning to maximize the hit probability of the maturity window and obtain the graph spatiotemporal model. The prediction distribution unit is used to input real-time node features and currently available maturity discrimination results into the graph spatiotemporal model to predict the mean maturity, uncertainty, maturity hit probability, and maturity window arrival time interval of each sampling partition in the future, thereby obtaining the maturity probability distribution.

[0010] Optionally, the node features include the original climate sequence and cumulative effect features. The original climate sequence includes temperature, humidity, light intensity, rainfall, and wind speed. The cumulative effect features include the cumulative effective temperature, cumulative rainfall, and dry-wet cycle intensity over the past several days.

[0011] Optionally, the harvesting decision module includes: An executable target unit is used to define maturity levels and threshold intervals according to the maturity probability distribution and the maturity window, define the probability of unripeness in each sampling partition as the risk of mixed harvesting, and set a harvestable time threshold to obtain an executable target; wherein, the maturity level and threshold interval include a suitable ripe interval, an unripe interval, and an overripe interval; A dual-objective optimization unit is used to define the execution constraints and objective functions of a single harvesting task based on the executable objectives, thereby obtaining an optimized harvesting task; the objective functions include a quality benefit objective function and a waste penalty objective function. The picking plan unit is used to divide each of the sampling partitions into a high-confidence mature main batch, a window-edge observation secondary batch, and a high-risk reserved batch based on the optimized picking task and the maturity probability distribution, and output an executable picking plan.

[0012] Optionally, the picking execution module includes: An executable entry unit is used to convert the harvesting plan into a minimum executable field to obtain a field entry; the minimum executable field includes task number, partition ID, tree number, execution time window, risk batch, maturity level, confidence level, and harvesting action; The picking and recording unit is used to execute the field entries using a picking robot and to collect photos of the fruit before picking, photos of the fruit during picking, and photos of the fruit after it is packed into barrels in real time, so as to obtain picking evidence. The picking detection unit is used to perform quality detection on the picking results based on the picking evidence, obtain actual maturity statistics, and update and optimize the maturity discrimination model and the graph spatiotemporal model based on the actual maturity statistics.

[0013] This invention also provides a method for determining the maturity of camellia fruit and for precise harvesting by integrating multi-sensor perception, including: The sampling area is divided into multiple sampling zones. Data is collected, verified, and structured for each sampling zone to obtain a multi-source unified dataset. Based on the multi-source unified dataset, fruit body localization, geometric normalization and illumination normalization are performed to obtain the normalized corrected fruit body contour. The reflectance ratio feature of the normalized corrected fruit body contour is calculated to construct a structured robust characterization. Using the structured robust representation, an encoder structure is constructed, and self-supervised cross-domain learning is performed on the encoder structure. MC Dropout and the total loss function are introduced to obtain a maturity discrimination model, which outputs a maturity latent variable representation, maturity estimation, and uncertainty estimation. Based on the sampling partitions, a graph spatiotemporal model is constructed, and the maturity discrimination results are integrated into the graph spatiotemporal model to perform partition prediction, thereby obtaining the maturity probability distribution. Based on the maturity probability distribution, executable objectives, execution constraints, and objective functions are defined, and risk batches are divided to obtain an executable harvesting plan; The harvesting plan is executed using a harvesting robot, and harvesting evidence is collected in real time for model optimization.

[0014] This invention discloses the following technical effects by providing a system and method for determining the maturity of camellia fruit and for precise harvesting by integrating multi-sensor perception: 1. A spatiotemporally layered, multimodal unified data acquisition and organization system was constructed. This system not only gridded the park and implemented differentiated sampling strategies, but also rigorously validated diverse and heterogeneous data, integrating it into a unified multi-source dataset with standardized metadata. This solved the problems of fragmented, format-chaotic, and spatiotemporally misaligned agricultural data acquisition, providing complete and traceable data raw materials for all subsequent algorithms. 2. By designing a physical robust characterization pipeline from geometric normalization to illumination normalization and then to reflectance characteristics, the local brightness field is estimated, shadow pixels are classified into a class, and color consistency is pulled back based on non-shadow areas. Then, it is converted into the material's inherent reflectance ratio characteristics, which greatly eliminates the interference of ambient light. A structured robust characterization that is insensitive to strong field interference factors such as illumination, viewing angle, and shadow is obtained, so that the characteristics can truly reflect the fruit's internal physiological state, rather than the external environment.

[0015] 3. A multimodal discriminant model was established that integrates self-supervised cross-domain contrastive learning, explicit alignment of physical indicators, and uncertainty estimation. A multimodal encoder was designed and injected with MC Dropout, which not only outputs maturity but also quantifies uncertainty. The model learned cross-modal invariant features of multispectral and RGB spectra and aligned them with physical ground truths such as hardness, which greatly improved the discrimination accuracy and cross-scene generalization ability. Moreover, the uncertainty output provides risk perception for subsequent prediction and decision-making.

[0016] 4. By introducing a graph spatiotemporal model that integrates the cumulative effect of climate and the current status of fruit, the park is modeled as a graph network. The edge weights express the probability of microenvironmental influence, and the node features integrate the cumulative effects of temperature and rainfall. The model learns and outputs the maturity probability distribution and window interval of each zone. It can move from static judgment at a single point to dynamic spatiotemporal extrapolation of the entire park's ripening process. It can accurately predict where, when, and with what degree of certainty, the optimal harvesting period can be reached.

[0017] 5. Based on probabilistic prediction, by defining a risk batch segmentation strategy and a bi-objective optimization function that includes quality benefits and waste penalties, the predicted probability is transformed into high-confidence primary batches, observation secondary batches, and high-risk reserved batches. The risk of mixed harvesting is explicitly modeled, which can transform fuzzy agronomic decisions into quantifiable and executable optimization problems, balancing the core contradiction of ensuring harvest quality and reducing overripe waste.

[0018] 6. By designing a closed-loop self-learning and quality traceability mechanism encompassing planning, execution, evidence, and optimization, the plan is broken down into the smallest executable items. During harvesting, image evidence is forcibly collected throughout the entire process—before harvesting, during harvesting, and after harvesting—for objective statistics of true maturity and model updates. This enables the system to achieve online self-evolution, automatically detect and correct biases in the judgment model, and form a complete data evidence chain, making every harvesting action traceable and reviewable.

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the system architecture provided in an embodiment of the present invention; Figure 2 This is a flowchart of maturity discrimination and prediction provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the method flow provided in an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] like Figure 1 As shown, this invention provides a system for determining the maturity of camellia fruit and for precise harvesting that integrates multi-sensor perception, including: 1. For example Figure 2 As shown, the partitioned data acquisition module is used to divide the sampling area into multiple sampling partitions, and to collect, verify, and unify the data structure for each sampling partition to obtain a multi-source unified dataset; the partitioned data acquisition module includes: 1.1 Spatial hierarchical unit is used to grid the sampling area to obtain multiple sampling partitions. For each sampling partition, different sampling intensity, sampling time and sampling point ID are set.

[0025] 1.2 Data sampling unit, used to acquire near-field RGB images, multispectral reflectance maps, thermal infrared maps, depth maps, fruit physical indicators, spatial location, and climate data and timestamps based on the sampling partition using calibrated sensors, to obtain a multi-source dataset; fruit physical indicators include moisture content, dielectric constant, or hardness. Climate data includes temperature, humidity, light intensity, and rainfall proxy indicators; timestamps must be aligned with image acquisition events.

[0026] 1.3 Data verification unit, used to perform integrity checks, time consistency checks, spatial consistency checks and calibration information checks based on the multi-source dataset to obtain a multi-source synchronized dataset.

[0027] 1.4 A unified structural unit is used to generate a unique sample ID, partition ID, fruit tree ID, collection timestamp, weather tag, temperature in degrees Celsius, relative humidity, illuminance, RGB image path, multispectral path, thermal infrared path, depth path, multispectral exposure, occlusion level (low, medium, high), fruit physical indicators, dielectric indicators, and hardness for each sample based on the multi-source synchronous dataset, thereby obtaining a multi-source unified dataset including sample metadata.

[0028] 2. For example Figure 2 As shown, the robust preprocessing module is used to perform fruit body localization, geometric normalization, and illumination normalization based on the multi-source unified dataset to obtain the normalized corrected fruit body contour, and calculate the reflectance ratio feature of the normalized corrected fruit body contour to construct a structured robust characterization; the robust preprocessing module includes: 2.1 Fruit body localization unit, used to obtain the coarse outline of the fruit body based on the multi-source unified dataset by threshold segmentation, retain the region whose size conforms to the fruit body scale in the largest connected region or foreground connected region, and remove the long strip and thin sheet regions to obtain the refined fruit body outline, and then identify the bounding box and principal axis direction of the refined fruit body outline to generate the transformation parameters of the normalized coordinate system of the fruit body.

[0029] 2.2 Normalization unit, used to perform fruit body cropping and unified viewpoint standardization based on the refined fruit body outline to obtain a geometrically normalized fruit body outline, and then perform shadow division and color consistency adjustment on the geometrically normalized fruit body outline to obtain a normalized corrected fruit body outline; the normalization unit includes: 2.2.1 Geometric normalization subunit, used to perform fixed boundary cropping of the fruit body based on the refined fruit body outline using the bounding box, for example, 224×224 or 256×256, to obtain a cropped fruit body image, and to perform radial or perspective transformation on the cropped fruit body image using the transformation parameters of the normalized coordinate system of the fruit body to obtain a fruit body image with a unified standard viewpoint, and then to perform alignment of the main axis direction of the fruit body and obtain the rotation angle based on the unified standard viewpoint fruit body image to obtain the geometric normalized fruit body outline.

[0030] 2.2.2 Illumination normalization subunit, used to estimate the local brightness field within the fruit body region using the geometrically normalized fruit body contour, and mark pixels with significantly low brightness and a correlation decrease with the surrounding neighborhood as shadow candidates to obtain shadow regions. Based on the shadow regions, the fruit body region is divided into shadow pixels and non-shadow pixels. The non-shadow pixels are used as the color statistical benchmark to perform color consistency pullback of shadow pixels, and the normalized color map of the fruit body region is regenerated to obtain the normalized corrected fruit body contour.

[0031] 2.3 Reflectance conversion unit, used to estimate reflectance based on the normalized corrected fruit body contour, by using the white field and dark field to count the digital output of the multispectral sensor, to obtain a unified band reflectance vector for each fruit body region, and to construct a reflectance ratio feature, i.e., a maturity-related feature, based on the unified band reflectance vector.

[0032] 2.4 Alignment and fusion unit, used to align and fuse the normalized corrected fruit body contour and the reflectance ratio feature to obtain a structured robust characterization.

[0033] 3. For example Figure 2 As shown, the maturity discrimination module is used to construct an encoder structure using the structured robust representation, perform self-supervised cross-domain learning on the encoder structure, and introduce MC Dropout and a total loss function to obtain a maturity discrimination model, outputting a maturity latent variable representation, maturity estimate, and uncertainty estimate; the maturity discrimination module includes: 3.1 Encoder unit, used to select a CNN-style feature extraction network as the RGB encoder, select an MLP convolutional encoder as the spectral encoder, and then weight and fuse the RGB encoder and the spectral encoder according to the occlusion level to obtain the encoder structure.

[0034] 3.2 Self-supervised learning unit, used to construct positive samples based on the structured robust representation by using spatiotemporal nearest neighbor construction and physical labeling guidance, and construct negative samples by using samples in different sampling partitions. Based on the positive and negative samples, cross-domain invariant features are learned through contrastive loss, and modality consistency loss is introduced to complete the self-supervised cross-domain contrastive learning of the encoder structure, so that samples near the maturity state are closer in the representation space.

[0035] Spatiotemporal nearest neighbor construction: Samples collected from the same partition within a short period of time are likely to have similar maturity levels; Physical labeling guidance: If there are a small number of physical truth values ​​or artificial maturity levels, samples of the same or similar levels can be used as the correct counterparts.

[0036] Modal consistency: Constrain RGB characterization, spectral characterization, and thermal characterization to maintain a consistent orientation on the same maturity sample.

[0037] 3.3 Physical alignment unit, used to select the regression network of fruit physical indicators to define the physical mapping head, construct the physical mapping function using the physical mapping head, and introduce the reverse consistency constraint into the physical mapping function to obtain the physical regression loss.

[0038] Reverse consistency constraint: mapping the predicted physical values ​​back to maturity.

[0039] 3.4 Total Loss Unit, used to weight and fuse contrastive learning loss, multimodal consistency loss, physical regression loss, and maturity supervision loss (cross-entropy or regression loss) to obtain the total loss function.

[0040] 3.5 The discriminant estimation unit is used to introduce MC Dropout and the total loss function into the encoder structure to obtain a maturity discrimination model, and output maturity discrimination results including maturity latent variable representation, maturity estimation, and uncertainty estimation. MC Dropout is used to estimate the prediction variance.

[0041] 4. For example Figure 2 As shown, the maturity prediction module is used to construct a graph spatiotemporal model based on the sampling partitions, and to integrate the maturity discrimination results into the graph spatiotemporal model for partition prediction to obtain the maturity probability distribution; the maturity prediction module includes: 4.1 A garden map unit is used to construct a set of nodes and a set of edges based on the sampling partitions, treating each partition as a node, and setting the edge weight for each edge. Then, for each node, node features are constructed to obtain an initial graph model. The edge weights are used to express the probability that the maturity evolution between nodes is affected by the microenvironment. The node features include the original climate sequence and cumulative effect features. The original climate sequence includes temperature, humidity, light intensity, rainfall, and wind speed. The cumulative effect features include the cumulative effective temperature, cumulative rainfall, and dry-wet cycle intensity over the past several days.

[0042] 4.2 Probability prediction unit, used to define the upper and lower limits of maturity to obtain a maturity window. Based on the maturity window, the maturity discrimination result is integrated into the initial graph model for regression training and variance learning to maximize the hit probability of the maturity window and obtain the graph spatiotemporal model.

[0043] 4.3 Prediction distribution unit, used to input real-time node features and currently available maturity discrimination results into the graph spatiotemporal model, predict the maturity mean, uncertainty, maturity hit probability and maturity window arrival time interval of each sampling partition in the future, and obtain the maturity probability distribution: prediction mean and prediction standard deviation.

[0044] The maturity window arrival time interval can be defined as, for example, the time when the probability of first entry exceeds a threshold, or the highest peak region in the future sequence can be taken as the arrival window.

[0045] 5. A harvesting decision module, used to define executable objectives, execution constraints, and objective functions based on the maturity probability distribution, and to perform risk batch division to obtain an executable harvesting plan; the harvesting decision module includes: 5.1 An executable target unit, used to define maturity levels and threshold intervals according to the maturity probability distribution and the maturity window, define the probability of unripeness in each sampling partition as the risk of mixed sampling, and set a harvestable time threshold to obtain an executable target; wherein, the maturity level and threshold interval include a moderately ripe interval, an unripe interval, and an overripe interval; the probability of unripeness includes unripe and overripe. Harvesting time threshold: harvesting is allowed only when the window hit probability exceeds the threshold.

[0046] 5.2 A dual-objective optimization unit is used to define the execution constraints and objective functions of a single harvesting task based on the executable objectives, so as to obtain an optimized harvesting task; the objective functions include a quality benefit objective function and a waste penalty objective function.

[0047] Execution constraints, for example: The daily harvesting hours are fixed. What is the maximum number of trees and buckets that can be harvested per hour? Limited personnel or equipment resources meant that only certain zones could be sampled first.

[0048] 5.3 The picking plan unit is used to divide each of the sampling partitions into a high-confidence mature main batch, a window-edge observation secondary batch, and a high-risk reserved batch based on the optimized picking task and the maturity probability distribution, and output an executable picking plan.

[0049] 6. A harvesting execution module, used to execute the harvesting plan using a harvesting robot and collect harvesting evidence in real time for model optimization; the harvesting execution module includes: 6.1 Executable entry unit, used to convert the harvesting plan into the minimum executable field to obtain field entries; the minimum executable field includes task number, partition ID, tree number, execution time window, risk batch, maturity level, confidence level, and harvesting action.

[0050] 6.2 The picking and recording unit is used to execute the field entries using the picking robot and collect photos of the fruit before picking, photos of the confirmation during picking, and photos of the fruit after packing in real time to obtain picking evidence.

[0051] 6.3 The picking and detection unit is used to perform quality detection on the picking results based on the picking evidence, obtain actual maturity statistics, and update and optimize the maturity discrimination model and the graph spatiotemporal model based on the actual maturity statistics.

[0052] like Figure 3 As shown, a method for determining the maturity of camellia oleifera fruit and for precise harvesting that integrates multi-sensor perception includes: The sampling area is divided into multiple sampling zones. Data is collected, verified, and structured for each sampling zone to obtain a multi-source unified dataset. Based on the multi-source unified dataset, fruit body localization, geometric normalization and illumination normalization are performed to obtain the normalized corrected fruit body contour. The reflectance ratio feature of the normalized corrected fruit body contour is calculated to construct a structured robust characterization. Using the structured robust representation, an encoder structure is constructed, and self-supervised cross-domain learning is performed on the encoder structure. MC Dropout and the total loss function are introduced to obtain a maturity discrimination model, which outputs a maturity latent variable representation, maturity estimation, and uncertainty estimation. Based on the sampling partitions, a graph spatiotemporal model is constructed, and the maturity discrimination results are integrated into the graph spatiotemporal model to perform partition prediction, thereby obtaining the maturity probability distribution. Based on the maturity probability distribution, executable objectives, execution constraints, and objective functions are defined, and risk batches are divided to obtain an executable harvesting plan; The harvesting plan is executed using a harvesting robot, and harvesting evidence is collected in real time for model optimization.

[0053] Therefore, this invention provides a system and method for determining the maturity of camellia fruit and for precise harvesting by integrating multi-sensor perception. By deeply mining the physical characteristics of the fruit through multi-sensor fusion perception and combining it with a spatiotemporal graph model for forward-looking prediction, the system achieves precise fruit identification and harvesting, thereby improving harvesting quality and efficiency and reducing waste.

[0054] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0055] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A system for Camellia oleifera maturity discrimination and precision picking by fusing multi-sensor perception, characterized in that, include: The partitioned data acquisition module is used to divide the sampling area into multiple sampling partitions, and to collect, verify and unify the data structure of each sampling partition to obtain a multi-source unified dataset. The robust preprocessing module is used to perform fruit body localization, geometric normalization and illumination normalization based on the multi-source unified dataset to obtain the normalized corrected fruit body contour, and calculate the reflectance ratio feature of the normalized corrected fruit body contour to construct a structured robust characterization. The maturity discrimination module is used to construct an encoder structure using the structured robust representation, perform self-supervised cross-domain learning on the encoder structure, and introduce MC Dropout and total loss function to obtain a maturity discrimination model, outputting a maturity latent variable representation, maturity estimate and uncertainty estimate. The maturity prediction module is used to construct a graph spatiotemporal model based on the sampling partitions, integrate the maturity discrimination results into the graph spatiotemporal model to perform partition prediction, and obtain the maturity probability distribution. The harvesting decision module is used to define executable objectives, execution constraints, and objective functions based on the maturity probability distribution, and to divide risk batches to obtain an executable harvesting plan; The harvesting execution module is used to execute the harvesting plan using a harvesting robot and collect harvesting evidence in real time for model optimization. The partitioned data collection module, the robust preprocessing module, the maturity discrimination module, the maturity prediction module, the harvesting decision module, and the harvesting execution module are interconnected.

2. The system according to claim 1, wherein, The partition acquisition module includes: Spatial hierarchical units are used to grid the sampling area to obtain multiple sampling zones. For each sampling zone, different sampling intensities, sampling times, and sampling point IDs are set. The data sampling unit is used to acquire near-field RGB images, multispectral reflectance maps, thermal infrared maps, depth maps, fruit physical indicators, spatial locations, and climate data and timestamps based on the sampling partition using calibrated sensors to obtain a multi-source dataset; The data verification unit is used to perform integrity checks, time consistency checks, spatial consistency checks, and calibration information checks based on the multi-source dataset to obtain a multi-source synchronized dataset. The unified structural unit is used to generate a unique sample ID, partition ID, fruit tree ID, collection timestamp, weather tag, temperature in degrees Celsius, relative humidity, illuminance, RGB image path, multispectral path, thermal infrared path, depth path, multispectral exposure, occlusion level, fruit physical indicators, dielectric indicators, and hardness for each sample based on the multi-source synchronous dataset, thereby obtaining a multi-source unified dataset including sample metadata.

3. The system according to claim 2, wherein the system comprises a plurality of sensors, and the system is capable of sensing the maturity of the tea fruits and the system is capable of picking the tea fruits. The robust preprocessing module includes: The fruit body localization unit is used to obtain the coarse outline of the fruit body based on the multi-source unified dataset by using threshold segmentation. For the coarse outline of the fruit body, it retains the regions in the largest connected region or the foreground connected region whose size matches the scale of the fruit body, and removes the long strip and thin sheet regions to obtain the refined outline of the fruit body. Then, it identifies the bounding box and principal axis direction of the refined outline of the fruit body and generates the transformation parameters of the normalized coordinate system of the fruit body. The normalization unit is used to perform fruit body cropping and unified view standardization based on the refined fruit body outline to obtain a geometrically normalized fruit body outline. Then, shadow division and color consistency pull-back are performed on the geometrically normalized fruit body outline to obtain a normalized corrected fruit body outline. The reflectance conversion unit is used to estimate the reflectance based on the normalized corrected fruit body contour by using the white field and dark field to count the digital output of the multispectral sensor, to obtain the uniform band reflectance vector of each fruit body region, and to construct the reflectance ratio feature based on the uniform band reflectance vector. The alignment and fusion unit is used to align and fuse the normalized corrected fruit body contour and the reflectance ratio feature to obtain a structured robust characterization.

4. The system according to claim 3, wherein the system comprises a plurality of sensors, and the system is capable of sensing the maturity of the tea fruits and the system is capable of picking the tea fruits. The normalization unit includes: The geometric normalization subunit is used to perform fixed boundary cropping of the fruit body based on the refined fruit body outline using the bounding box to obtain a fruit body cropping image, and to perform radial or perspective transformation on the fruit body cropping image using the transformation parameters of the fruit body normalized coordinate system to obtain a fruit body image with a unified standard view. Then, based on the fruit body image with a unified standard view, the main axis direction of the fruit body is aligned and the rotation angle is obtained to obtain the geometric normalized fruit body outline. The illumination normalization subunit is used to estimate the local brightness field within the fruit body region using the geometrically normalized fruit body contour, and to mark pixels with significantly low brightness and a correlation decrease with the surrounding neighborhood as shadow candidates to obtain shadow regions. Based on the shadow regions, the fruit body region is divided into shadow pixels and non-shadow pixels. The non-shadow pixels are used as the color statistical benchmark to perform color consistency pull-back of shadow pixels, and to regenerate the normalized color map of the fruit body region to obtain the normalized corrected fruit body contour.

5. The system for determining the maturity of camellia fruit and for precise harvesting by integrating multi-sensor perception as described in claim 4, characterized in that, The maturity determination module includes: The encoder unit is used to select a CNN-style feature extraction network as the RGB encoder, select an MLP convolutional encoder as the spectral encoder, and then weight and fuse the RGB encoder and the spectral encoder according to the occlusion level to obtain the encoder structure. The self-supervised learning unit is used to construct positive samples based on the structured robust representation by using a spatiotemporal nearest neighbor construction method and a physical labeling guidance method, and to construct negative samples based on samples in different sampling partitions. Based on the positive and negative samples, cross-domain invariant features are learned through contrastive loss, and modality consistency loss is introduced to complete the self-supervised cross-domain contrastive learning of the encoder structure. The physical alignment unit is used to select the regression network for the physical indicators of the fruit to define the physical mapping head, construct the physical mapping function using the physical mapping head, and introduce the reverse consistency constraint into the physical mapping function to obtain the physical regression loss. The total loss unit is used to weight and fuse the contrastive learning loss, multimodal consistency loss, physical regression loss, and maturity supervision loss to obtain the total loss function; The discrimination estimation unit is used to introduce MC Dropout and the total loss function into the encoder structure to obtain a maturity discrimination model and output maturity discrimination results including maturity latent variable representation, maturity estimation and uncertainty estimation.

6. The system for determining the maturity of camellia fruit and for precise harvesting by integrating multi-sensor perception as described in claim 5, characterized in that, The maturity prediction module includes: The garden map unit is used to construct a node set and an edge set based on the sampling partition, taking each partition as a node, setting the edge weight of each edge, and then constructing node features for each node to obtain an initial graph model; wherein, the edge weight is used to express the probability that the maturity evolution between nodes is affected by the microenvironment. The probability prediction unit is used to define the upper and lower limits of maturity to obtain a maturity window. Based on the maturity window, the maturity discrimination result is integrated into the initial graph model for regression training and variance learning to maximize the hit probability of the maturity window and obtain the graph spatiotemporal model. The prediction distribution unit is used to input real-time node features and currently available maturity discrimination results into the graph spatiotemporal model to predict the mean maturity, uncertainty, maturity hit probability, and maturity window arrival time interval of each sampling partition in the future, thereby obtaining the maturity probability distribution.

7. The system for determining the maturity of camellia fruit and for precise harvesting by integrating multi-sensor perception as described in claim 6, characterized in that, The node features include the original climate sequence and cumulative effect features. The original climate sequence includes temperature, humidity, light intensity, rainfall, and wind speed. The cumulative effect features include the cumulative effective temperature, cumulative rainfall, and dry-wet cycle intensity over the past several days.

8. The system for determining the maturity of camellia fruit and for precise harvesting by integrating multi-sensor perception as described in claim 7, characterized in that, The harvesting decision module includes: An executable target unit is used to define maturity levels and threshold intervals according to the maturity probability distribution and the maturity window, define the probability of unripeness in each sampling partition as the risk of mixed harvesting, and set a harvestable time threshold to obtain an executable target; wherein, the maturity level and threshold interval include a suitable ripe interval, an unripe interval, and an overripe interval; A dual-objective optimization unit is used to define the execution constraints and objective functions of a single harvesting task based on the executable objectives, thereby obtaining an optimized harvesting task; the objective functions include a quality benefit objective function and a waste penalty objective function. The picking plan unit is used to divide each of the sampling partitions into a high-confidence mature main batch, a window-edge observation secondary batch, and a high-risk reserved batch based on the optimized picking task and the maturity probability distribution, and output an executable picking plan.

9. A system for determining the maturity of camellia fruit and for precise harvesting that integrates multi-sensor perception, as described in claim 8, is characterized in that... The harvesting execution module includes: An executable entry unit is used to convert the harvesting plan into a minimum executable field to obtain a field entry; the minimum executable field includes task number, partition ID, tree number, execution time window, risk batch, maturity level, confidence level, and harvesting action; The picking and recording unit is used to execute the field entries using a picking robot and to collect photos of the fruit before picking, photos of the fruit during picking, and photos of the fruit after it is packed into barrels in real time, so as to obtain picking evidence. The picking detection unit is used to perform quality detection on the picking results based on the picking evidence, obtain actual maturity statistics, and update and optimize the maturity discrimination model and the graph spatiotemporal model based on the actual maturity statistics.

10. The method for determining the maturity of camellia fruit and for precise harvesting based on multi-sensor sensing according to claim 9, characterized in that, include: The sampling area is divided into multiple sampling zones. Data is collected, verified, and structured for each sampling zone to obtain a multi-source unified dataset. Based on the multi-source unified dataset, fruit body localization, geometric normalization and illumination normalization are performed to obtain the normalized corrected fruit body contour. The reflectance ratio feature of the normalized corrected fruit body contour is calculated to construct a structured robust characterization. Using the structured robust representation, an encoder structure is constructed, and self-supervised cross-domain learning is performed on the encoder structure. MC Dropout and the total loss function are introduced to obtain a maturity discrimination model, which outputs a maturity latent variable representation, maturity estimation, and uncertainty estimation. Based on the sampling partitions, a graph spatiotemporal model is constructed, and the maturity discrimination results are integrated into the graph spatiotemporal model to perform partition prediction, thereby obtaining the maturity probability distribution. Based on the maturity probability distribution, executable objectives, execution constraints, and objective functions are defined, and risk batches are divided to obtain an executable harvesting plan; The harvesting plan is executed using a harvesting robot, and harvesting evidence is collected in real time for model optimization.

Images