Tomato internode length three-dimensional measurement method, system and medium

Abstract

The application discloses a kind of tomato internode length three-dimensional measurement method, for greenhouse shelter, stem bending and direct two-dimensional pixel ranging is easy to produce geometric error, obtains plant single frame color image and depth data;Color image is input into feature-enhanced multi-task perception model, and the two-dimensional coordinates of internode instance segmentation mask and two end nodes key points are output;Based on pixel correspondence relationship, candidate three-dimensional point cloud is extracted, and three-dimensional coordinates of internode end point are generated by key point neighborhood depth;Candidate point cloud is voxel down-sampled, and outlier is filtered out, and depth band-pass filtering and spatial density clustering are carried out, and target main stem point cloud is obtained in combination with spatial distance and depth consistency evaluation;Estimate main shaft by principal component analysis and axial slice, center line extraction is carried out, and is smoothly fitted using cubic B-spline curve;End point three-dimensional coordinates are projected to fitting curve, and end point outside end of curve is linearly extrapolated along corresponding tangent to compensate, and the three-dimensional measurement result of internode length is obtained by calculating spatial curve arc length.

Description

Technical Field

[0001] This invention belongs to the fields of smart agriculture phenotyping, computer vision and 3D point cloud processing technology, specifically involving a 3D measurement method, system and medium for tomato internode length. Background Technology

[0002] Tomatoes are an important economic crop widely cultivated in greenhouse environments. Internode length directly reflects stem elongation and plant structure, serving as a crucial phenotypic indicator for assessing plant growth and development, guiding cultivation management, and predicting potential yield. Currently, methods for obtaining this phenotypic parameter face the following main technical bottlenecks:

[0003] (1) Traditional contact-based manual measurement has limitations:

[0004] Currently, internode measurements in actual production mainly rely on manual handheld measuring tools (such as vernier calipers). This method is not only time-consuming, labor-intensive, and inefficient, but the measurement results are also highly susceptible to subjective influences from human operating experience and natural plant posture changes. Furthermore, contact-based operations can easily cause mechanical damage to the plants, and cannot meet the needs of continuous, large-scale, and objective online monitoring of crop growth in modern greenhouse settings.

[0005] (2) Traditional two-dimensional vision methods lack real physical scale constraints:

[0006] Existing measurement methods based on monocular RGB 2D images mainly rely on pixel scale or projection geometry for estimation. Due to the inherent lack of depth information, their measurement accuracy is highly susceptible to interference from camera shooting distance, changes in viewing angle and posture, and leaf occlusion. Especially under the complex canopy and dense planting conditions in the middle and late stages of greenhouse tomato growth, 2D images often suffer from severe perspective projection distortion, making it impossible for algorithms to extract the true physical length with spatial geometric consistency.

[0007] (3) Multi-view 3D reconstruction methods are difficult to acquire and have high computational costs:

[0008] To address the scale deficiency issue in 2D measurements, some existing technologies have introduced multi-view / multi-frame fusion approaches for 3D reconstruction. However, these methods require complex multi-angle data acquisition and cross-view point cloud registration and fusion processes, resulting in high computational costs and expensive hardware. More critically, in densely planted greenhouse environments, the camera's viewing angle is severely limited, and even slight swaying of plants due to airflow or mechanical influences can easily disrupt the temporal and spatial consistency of cross-frame data, leading to 3D reconstruction failures or highly unstable extracted phenotypic features.

[0009] (4) Conventional single-task model architecture leads to computational redundancy and alignment problems:

[0010] In rapid measurement scenarios based on single-frame vision, accurate internode measurement requires simultaneously acquiring the segmentation mask and keypoints of the stem region. Existing conventional vision algorithms often employ independent segmentation and keypoint networks for separate inference. This multi-model serial invocation approach not only causes severe waste of computational resources in the low-level feature extraction stage, greatly increasing the system's forward inference latency, but also lacks unified prior constraints between the heterogeneous results output by different models, necessitating the introduction of complex feature matching and post-processing alignment operations. This highly complex solution is difficult to meet the practical needs of real-time, high-throughput edge deployment of agricultural equipment.

[0011] (5) The original local 3D point cloud has unstructured and high-noise characteristics, and cannot be directly used for accurate measurement:

[0012] Even after obtaining the two-dimensional semantic region of the target through a visual model and mapping it to a three-dimensional point cloud, the original local point cloud extracted from a single frame of RGB-D still possesses highly unstructured characteristics. On the one hand, in the complex background of a greenhouse, the extracted local point cloud often contains a large amount of outlier noise such as overlapping leaves and supports, and is prone to depth loss. The lack of an effective spatial clustering and denoising screening mechanism will lead to the failure of extracting the true target point cloud. On the other hand, tomato internodes typically exhibit a slender and curved spatial morphology during natural growth. Existing point cloud measurement methods often crudely calculate the Euclidean distance between the two endpoints, completely ignoring the true geometric topology of the curved stem. Without targeted continuous spatial geometric fitting processing, this simple calculation of "using straight lines to represent curves" will produce serious geometric measurement errors, making it impossible to achieve high-precision measurements with geometric consistency.

[0013] In summary, existing technologies still face significant challenges in achieving efficient, robust, and accurate three-dimensional non-contact measurement of tomato internode length in complex greenhouse environments. Summary of the Invention

[0014] The purpose of this invention is to overcome the shortcomings of existing technologies, such as poor robustness of single visual models, lack of three-dimensional spatial constraints, and susceptibility of point cloud processing to noise interference. This invention provides a three-dimensional measurement method, system, and medium for tomato internode length based on RGB-D fusion and a multi-task perception model. This method improves the deep learning object detection network and introduces a parallel dual-head architecture to jointly extract two-dimensional semantic and key point features of slender internodes. At the same time, it combines RGB-D depth information and introduces a point cloud optimization mechanism based on a multi-dimensional cost function to perform three-dimensional spatial verification of visual features.

[0015] The technical solution adopted in this invention includes the following steps:

[0016] A three-dimensional measurement method for tomato internode length, characterized by the following steps:

[0017] Step 1: Multimodal Data Acquisition and Spatial Alignment

[0018] RGB-D depth imaging equipment was used to acquire pixel-aligned RGB color and depth images of the target tomato plant, and the camera intrinsic and extrinsic parameter matrices were collected. During the acquisition process, the working distance between the equipment and the main stem of the plant was kept within the range of 26-45cm, and the acquisition area was focused on the fully developed internodes in the subapical mature zone of the tomato plant.

[0019] Step 2: Joint extraction of 2D visual features based on multi-task parallel perception:

[0020] The RGB color image is input into a pre-trained multi-task parallel perceptual vision model, and a single end-to-end inference is performed to synchronously output the instance segmentation mask and the two-dimensional coordinates of key points between the target tomato segments; and based on the output results of the same target instance, a one-to-one correspondence is formed between the instance segmentation mask and the two-dimensional coordinates of key points.

[0021] The visual model includes a backbone feature extraction network, a feature fusion network, and a multi-task prediction network:

[0022] The backbone feature extraction network includes a deformation-adaptive feature extraction module, which uses deformable convolutional units to dynamically adjust the feature sampling grid to adapt to the curved shape of the tomato stem. Preferably, the deformation-adaptive feature extraction module adopts a C2f-DCNv3 structure, which achieves adaptive feature alignment for curved targets by dynamically learning the two-dimensional spatial offset and modulation scalar of the convolutional kernel sampling points.

[0023] The feature fusion network includes a multi-scale context aggregation module, which extracts and fuses multi-scale contextual features through multi-branch convolutional structures with different receptive fields to enhance the model's robustness to occlusion and complex backgrounds. Preferably, the multi-scale context aggregation module adopts an MSBlock structure, extracting multi-scale features through multiple convolutional branches with different dilation rates and fusing them along the channel dimension.

[0024] The multi-task prediction network includes an instance segmentation branch and a keypoint localization branch. The instance segmentation branch employs a generation mechanism based on "prototype-coefficient" decoupling, comprising a prototype generation module and a mask coefficient prediction module. It generates a pixel-level segmentation mask by linearly combining and decoding the mask coefficients with the mask prototype. The keypoint localization branch uses a heatmap-free direct regression prediction mechanism, directly outputting the two-dimensional normalized coordinate offset of the keypoint relative to a preset reference point and its visibility confidence. In the tomato internode detection scenario, the keypoint is defined as the physiological nodes at both ends of the internode, with the number of keypoints N=2.

[0025] The method for determining instance-level correspondences: During a single forward propagation, the joint prediction network assigns a unified instance identifier to each retained candidate target instance, and its joint output set is represented as follows: ,in It is a two-dimensional bounding box. For instance segmentation mask, The key point set is defined as follows: by sharing the same instance identifier, an instance-level one-to-one correspondence is established between the segmentation mask and the key point set during the output stage.

[0026] Model Training: The visual model is based on the single-stage object detection network YOLOv8 and is trained end-to-end using a multi-task joint loss function. The joint loss function includes detection loss, segmentation loss, keypoint localization loss, and keypoint visibility loss, and uses preset weight coefficients to balance the gradient contributions of each sub-task during training. The detection loss includes category classification loss, bounding box regression loss, and localization distribution loss; the segmentation loss includes mask binary cross-entropy loss; the keypoint localization loss uses OKS loss; and the keypoint visibility loss uses binary cross-entropy loss.

[0027] Step 3: 3D spatial mapping and cross-modal constraint generation based on 2D visual features:

[0028] Based on the camera intrinsic parameter matrix and depth image obtained in step one, the instance segmentation mask and keypoint coordinates output in step two are mapped to three-dimensional space respectively:

[0029] - Extract the 3D point cloud corresponding to the area covered by the instance segmentation mask, and use it as the initial region of interest point cloud between target segments to provide spatial region constraints;

[0030] -Based on the two-dimensional coordinates of the key points and the effective depth information of the neighborhood in the aligned point cloud, generate the three-dimensional coordinates of the inter-segment endpoints of the inter-segment start and end points to provide structural endpoint constraints;

[0031] - The spatial region constraints and structural endpoint constraints together constitute three-dimensional prior constraint information, providing joint input for subsequent point cloud optimization and curve modeling.

[0032] Step 4: Point cloud optimization, spatial curve modeling, and physical length calculation based on 3D prior constraints:

[0033] 4.1 The main structure point cloud is subjected to voxel downsampling and statistical outlier filtering for local noise suppression to obtain a continuous and pure stem internode point cloud;

[0034] 4.2 Spatial density clustering algorithm is used to cluster the point cloud of the region of interest after depth bandpass filtering, and a multi-feature evaluation function including spatial distance cost and depth consistency cost is constructed. The three-dimensional coordinates of the internode endpoints are used as reference benchmarks to evaluate and filter each point cloud cluster. The point cloud cluster with the best evaluation value is selected as the target stem main structure point cloud, and the interference point cloud clusters composed of background leaves or supports are removed.

[0035] 4.3 Based on the main structure point cloud, extract the central skeleton point sequence and construct a three-dimensional spatial parameterized curve to fit the natural bending shape of the tomato stem;

[0036] 4.4 Project the three-dimensional coordinates of the intersegmental endpoints onto the fitted curve. If the three-dimensional coordinates of the intersegmental endpoints are located outside the endpoints of the fitted curve, perform linear extrapolation compensation along the tangent vector direction of the curve endpoints.

[0037] 4.5 Calculate the arc length of the fitted curve and add it to the extrapolated compensation length to output the final three-dimensional physical length of the tomato internode.

[0038] Furthermore, the one-to-one correspondence between the instance segmentation mask and the two-dimensional coordinates of the key points is as follows:

[0039] The joint prediction network assigns a uniform instance identifier to each retained candidate target instance during a single forward propagation; its joint output set is represented as follows. ,in, For the first Two-dimensional bounding boxes of each target instance. For the first Pixel-level instance segmentation mask for each target instance For the first A set of key points for each target instance;

[0040] By sharing the same instance identifier, an instance-level one-to-one correspondence is established between the segmentation mask and the key point set during the output stage.

[0041] Furthermore, the multi-feature evaluation function constructed in step four is expressed as:

[0042]

[0043] in, The cost is the spatial distance between the point cloud cluster and the three-dimensional coordinates of the internode endpoints. This is a depth consistency penalty term for point cloud clusters relative to the 3D coordinates of internode endpoints. and These are preset weighting coefficients used to adjust the relative contributions of spatial distance cost and depth consistency penalty.

[0044] Furthermore, the specific method for extracting the central skeleton points and fitting the three-dimensional spatial parameterized curve in step 4.3 is as follows:

[0045] Principal component analysis is used to determine the principal axis direction of the point cloud. The point cloud is divided into several slices along the principal axis direction. The median coordinates of the point cloud in each slice are calculated as skeleton points to form an ordered skeleton point sequence.

[0046] The center point sequence is smoothly fitted using a cubic B-spline curve to obtain the spatial parameterized curve C(u), where u∈[0,1].

[0047] Furthermore, the specific method for calculating the arc length of the fitted curve in step 4.5 is as follows:

[0048] Project the three-dimensional coordinates of the intersegmental endpoints generated in step three onto the spatial parameterized curve. Determine the boundary parameters of the curve corresponding to the start and end points of the tomato internodes. and ;

[0049] In the parameter range Uniform sampling is performed within the curve, and the arc length is calculated by accumulating the Euclidean distance between adjacent sampling points. ;

[0050] Calculate the physical length of the tomato internode as L total = + extstart +L extend ,in extstar and L extend These are the extrapolated compensation lengths for the starting and ending points obtained in step 4.4, respectively.

[0051] On the other hand, the present invention also provides a three-dimensional measurement system for tomato internode length based on RGB-D fusion and a multi-task perception model, characterized in that it includes:

[0052] The multimodal data acquisition module is used to acquire RGB color images and depth images of the target tomato plant, and perform pixel-level spatiotemporal alignment on the two, while extracting the camera intrinsic and extrinsic parameter matrices; it outputs the aligned RGB color image, depth image and camera parameters, and connects to the first input end of the multi-task parallel perception vision model module and the first input end of the three-dimensional space mapping module, respectively.

[0053] The multi-task parallel perception vision model module internally stores a pre-trained multi-task parallel perception vision model, which is used to receive the RGB color image output by the multimodal data acquisition module, perform a single end-to-end inference, synchronously output the instance segmentation mask and key point two-dimensional coordinates of each tomato internode target instance, and based on the output results of the same target instance, make the instance segmentation mask and key point two-dimensional coordinates form a one-to-one correspondence.

[0054] The 3D spatial mapping module receives the depth image and camera parameters output by the multimodal data acquisition module, as well as the instance segmentation mask and key point 2D coordinates with instance-level correspondence output by the multi-task parallel perception vision model module; based on the camera parameters and depth image, it maps the instance segmentation mask to an initial region of interest point cloud in 3D space, and simultaneously maps the key point 2D coordinates to intersegment endpoint 3D coordinates in 3D space; the initial region of interest point cloud is used as 3D processing input, and the intersegment endpoint 3D coordinates are used as 3D prior information, and both are output to the point cloud processing and curve modeling module;

[0055] The point cloud processing and curve modeling module is used to receive the initial region of interest point cloud and the three-dimensional coordinates of the inter-segment endpoints output by the three-dimensional space mapping module.

[0056] The output module is used to receive the arc length of the internode centerline curve and the extrapolation compensation length of the endpoints output by the point cloud processing and curve modeling module, and output the final three-dimensional physical length measurement result of the tomato internode.

[0057] Third, the present invention also provides a computer-readable storage medium storing executable instructions thereon, which, when executed by one or more processors, implement the three-dimensional measurement method for tomato internode length based on RGB-D fusion and multi-task perception model as described above.

[0058] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0059] 1) By deeply coupling visual semantics and spatial geometry, and combining parallel dual-head networks with multi-dimensional cost function point cloud optimization, the false negative and false positive rates under complex lighting and dense foliage occlusion in greenhouses are reduced. By fusing two-dimensional visual priors (segmentation masks, key points) with three-dimensional depth information to construct an evaluation function, joint quality assessment of point cloud clusters is achieved, reducing misselection of point clouds and deviation in key point localization caused by complex backgrounds.

[0060] 2) By using RGB-D pixel-level alignment and reprojection, two-dimensional features are accurately mapped to three-dimensional point cloud space, eliminating measurement errors caused by changes in camera shooting angle and perspective projection distortion ("nearer objects appear larger, farther objects smaller"). Combining B-spline space curve fitting and a three-dimensional arc length integral algorithm, not only is the three-dimensional curved topological morphology of tomato internodes under natural growth conditions restored, but also the geometric measurement error caused by "substituting straight lines for curves" is reduced by replacing traditional straight-line Euclidean distance estimation with arc length integral, achieving high-precision absolute physical length measurement.

[0061] 3) By employing non-contact machine vision and point cloud computing, irreversible physical damage to tomato plant stems is avoided during the measurement process. Simultaneously, the measurement process is fully automated, and the core algorithm has the potential for lightweight deployment, allowing it to be integrated into the visual servo system of automated greenhouse harvesting robots or phenotypic monitoring and inspection platforms, providing core technical support for large-scale, high-throughput data collection and improved variety breeding in modern smart agriculture.

[0062] 4) This invention introduces deformable convolutional units into the backbone network, enabling the feature sampling positions to adaptively adjust according to the bending shape and irregular boundaries of tomato stems; and embeds a multi-receptive-field parallel aggregation module in the feature fusion network, effectively enhancing the model's semantic integrity and anti-interference ability under conditions of branch and leaf intersection occlusion. These two innovations jointly improve the model's robustness in perceiving slender, curved, and partially occluded targets in complex greenhouse scenarios. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the overall process of the three-dimensional measurement method for tomato internode length provided in an embodiment of the present invention.

[0064] Figure 2 This is a schematic diagram of the network structure of the YOLO-ISPNet multi-task parallel sensing model provided in an embodiment of the present invention.

[0065] Figure 3 This is a schematic diagram of tomato internode instance segmentation mask and key point detection results provided in an embodiment of the present invention.

[0066] Figure 4 This is a schematic diagram of the deformation adaptive feature extraction module (DCNv3 module) provided in an embodiment of the present invention.

[0067] Figure 5 A schematic diagram of the multi-scale context aggregation module (MSBlock structure) provided in an embodiment of the present invention.

[0068] Figure 6 This invention provides a schematic diagram of the process for generating ROI point clouds and inter-segment endpoint three-dimensional coordinates based on instance segmentation masks and two-dimensional coordinates of key points.

[0069] Figure 7 This is a schematic diagram of the ROI point cloud preprocessing and target stem main structure point cloud screening process provided in an embodiment of the present invention.

[0070] Figure 8 This is a schematic diagram illustrating the calculation of tomato internode length based on spatial curve fitting, endpoint projection, and tangential extrapolation compensation, provided for an embodiment of the present invention. Detailed Implementation

[0071] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0072] Example 1: A method for measuring internode length in tomatoes based on multi-task parallel sensing and 3D mapping

[0073] like Figure 1 As shown, this embodiment follows the process of "multimodal data acquisition and spatial alignment → multi-task parallel perception visual model construction and inference → two-dimensional feature three-dimensional spatial mapping → point cloud optimization, spatial curve modeling and physical length calculation" to automatically detect and measure the physical length of tomato internodes in a greenhouse setting.

[0074] Step 1: Multimodal Data Acquisition and Spatial Alignment

[0075] (a) Equipment deployment and calibration parameter acquisition

[0076] The target tomato plants were photographed using an RGB-D depth imaging device to acquire multimodal raw data and construct a dataset. To ensure the robustness of the model in an unstructured greenhouse environment and the accuracy of subsequent 3D measurements, the specific acquisition process and parameter settings are as follows:

[0077] 1. Target area location:

[0078] The subapical maturation zone below the apical elongation zone of the tomato plant was selected as the target sampling area, specifically 1-2 internodes that have recently fully extended and whose morphology is basically finalized. Internode elongation in this section has essentially stopped, avoiding interference from morphological changes caused by rapid plant growth. This invention defines "internode length" as the distance between two adjacent nodes along the main stem direction and focuses on the aforementioned fully developed internodes for three-dimensional feature extraction.

[0079] 2. Equipment configuration and viewing angle control:

[0080] In this embodiment, a ZED Mini binocular stereo camera is preferably used as the depth imaging device, with a resolution set to 1920×1080 pixels. To simulate the actual working perspective of a ground-based mobile robot in a greenhouse environment, a continuous viewpoint acquisition strategy is adopted: the device is kept at approximately the same height as the plant and moves laterally around the plant in the horizontal direction, with the acquisition viewpoint covering a range of ±90° from the frontal view. During the acquisition process, the device height is dynamically adjusted to ensure that the field of view always covers the subapical maturity zone, and the working distance between the device and the main stem of the plant is controlled within the range of 26–45 cm (preferably 30–40 cm) to obtain high-quality near-side multi-view image data.

[0081] 3. Data synchronization and underlying interface calls:

[0082] The depth imaging device and its software development kit (SDK) are initialized. By calling the device's underlying synchronization and alignment interfaces, RGB color and depth images that have undergone spatiotemporal synchronization and achieved pixel-level precise alignment are directly acquired. Simultaneously, the camera intrinsic and extrinsic parameter matrices of the current imaging device are obtained in real time through the SDK. These strictly aligned data and camera parameters together provide a reference input for the subsequent accurate mapping of two-dimensional semantic masks to three-dimensional point cloud space.

[0083] 4. Enlarging and partitioning datasets for complex environments:

[0084] To enhance the adaptability of the subsequent recognition model to unstructured environments, this step involves data collection under various lighting conditions (including direct sunlight on sunny days, diffused light on cloudy days, front lighting, and backlighting) and at different time periods to ensure the diversity of sample morphology and lighting. After collection, images containing multiple varieties (such as large-fruited tomatoes and cherry tomatoes) are manually cleaned to remove blurry and invalid data, ultimately constructing a multimodal dataset containing high-quality images. Furthermore, this dataset is randomly divided into training, validation, and test sets according to a preset ratio (e.g., 8:1:1) for subsequent training and evaluation of the neural network model.

[0085] Step 2: Constructing and inferring a multi-task parallel perceptual visual model for segmentation and keypoint fusion.

[0086] The RGB color image obtained in the first step is input into the visual model for end-to-end inference to achieve synchronous perception of tomato target region instance segmentation and node key point localization.

[0087] like Figure 2As shown, in this embodiment, the visual model uses a YOLOv8 single-stage network as the basic framework for reconstruction and improvement. The multi-task parallel perceptual visual model comprises a three-part basic architecture: a backbone feature extraction network, a neck feature fusion network, and a head multi-task prediction network. The backbone feature extraction network extracts multi-scale basic features from the input RGB color image; the neck feature fusion network performs contextual semantic aggregation and bidirectional fusion of features at different scales; and the head multi-task prediction network outputs instance segmentation results and keypoint localization results synchronously and with strict alignment based on the fused shared features. In one embodiment, the single-stage network is used as the basic framework for reconstruction and improvement.

[0088] (1) Constructing a deformation-adaptive backbone feature extraction network

[0089] To enhance the model's geometric adaptability to the natural bending shape, irregular boundaries, and local deformation of tomato stems, this invention incorporates a deformation-adaptive feature extraction module into the backbone feature extraction network. This module dynamically adjusts the feature sampling grid by introducing a learnable spatial sampling offset and weight modulation mechanism, enabling it to adaptively conform to the physical edges of the tomato's non-rigid organs. In a preferred embodiment, as shown... Figure 4 As shown, the deformation-adaptive feature extraction module specifically adopts an improved C2f-DCNv3 structure. The internal processing mechanism of this module is as follows: the input feature map first undergoes channel mapping through an initial convolutional layer, and is then divided into a first branch and a second branch in the channel dimension. The first branch acts as a skip connection, directly mapping to the tail. The second branch is then serially input into several cascaded bottleneck structures. Specifically, the bottleneck structures use deformable convolutional units (DCNv3) instead of standard convolutions, dynamically calculating the two-dimensional spatial offset of the convolution kernel sampling points to achieve adaptive feature alignment for curved targets. Finally, the features of the first branch and the output features of the second branch are concatenated in the channel dimension, and then fused and dimensionality reduced by the terminal convolutional layer.

[0090] (2) To construct a multi-scale contextual feature fusion network that balances local spatial details of the tomato target (such as tiny key nodes) with high-level global semantic information (to overcome complex branch occlusion), the feature fusion network performs bidirectional aggregation and fusion of feature maps extracted from the backbone at different scales, ultimately outputting shared feature maps at multiple scales for use by subsequent multi-task prediction networks. To expand the effective receptive field without significantly increasing computational overhead, this invention innovatively embeds a multi-scale contextual aggregation module into the feature fusion network. In a preferred embodiment, the multi-scale contextual aggregation module specifically adopts a lightweight MSBlock structure, such as... Figure 5 As shown, the internal processing mechanism of this module is as follows: After the input feature map enters the module, it is grouped by channel or processed by a multi-branch parallel structure, and then fed into multiple feature extraction branches with different receptive field sizes (e.g., convolutional layers with different dilation rates). Each branch extracts local details and global context features at its respective receptive field scale, and then performs step-by-step fusion and aggregation of multi-scale features along the channel dimension, thereby greatly enhancing the model's anti-occlusion expression capability in mid-to-high-level feature spaces. Furthermore, the feature fusion network finally outputs multiple feature layers with different resolutions (specifically P3, P4, and P5 feature layers in the YOLO architecture). Among them, the high-resolution feature layer (P3) is used to retain rich fine-grained spatial geometric information, the medium-resolution feature layer (P4) is used to achieve scale transition, and the low-resolution feature layer (P5) is used to enhance the high-level semantic coherence across occlusion regions. Together, these three layers provide high-quality input for the multi-task prediction head.

[0091] (3) Construct a decoupled multi-task parallel prediction structure

[0092] Based on the multi-scale shared features output by the feature fusion network (Neck), a decoupled multi-task parallel prediction network (Head) is constructed. To avoid feature competition and gradient interference between perceptual tasks of different physical dimensions (i.e., region-level pixel segmentation and structure-level topology localization) in the shared feature space, this invention uses independent convolutional channels at the prediction front end of each scale feature map for feature decoupling. This structure enables the model to synchronously and strictly align the output of target bounding boxes, pixel-level instance segmentation masks, and keypoint 2D coordinates for the same tomato internode target during a single forward propagation. Specifically, this multi-task parallel prediction structure includes the following two core branches at the micro-feature decoding level:

[0093] 3.1 Instance-based branching: Decoupled generation based on prototype and coefficients

[0094] In a preferred embodiment of the present invention, to obtain high-precision geometric contours of tomato internodes while ensuring real-time inference, the instance segmentation branch adopts a generation mechanism based on "prototype-coefficient" decoupling (preferably referring to the YOLOv8-seg architecture). Specifically, the internal data flow of this branch is completed collaboratively by two sub-networks: one is a prototype generation module (Proto-Net), which receives high-resolution feature maps from the feature fusion network and generates a preset number of globally shared and category-independent mask prototypes through a series of convolution operations to characterize the underlying spatial structure in the image; the other is a mask coefficient prediction module, which shares some features with the target detection task and is responsible for predicting a set of low-dimensional mask coefficients for each detected candidate target instance (i.e., tomato internode). In the inference output stage, the model linearly combines the predicted specific instance mask coefficients with the global mask prototypes (e.g., matrix multiplication) and processes them through an activation function (e.g., Sigmoid) to decode and generate a pixel-level segmentation mask corresponding to the specific tomato internode instance. This decoupled generation mechanism effectively avoids the cumbersome candidate box pruning (RoI Align) operation in traditional segmentation networks, and significantly improves the mask generation efficiency in complex greenhouse backgrounds.

[0095] 3.2 Key Point Localization Branch: Direct Regression Prediction Without Heatmap

[0096] In a preferred embodiment of the present invention, considering that tomato internodes are often obstructed by interbranchs and leaves in a greenhouse environment, the key point localization branch abandons the traditional heatmap prediction paradigm which has high computational overhead and is susceptible to quantization errors, and instead adopts an anchor-free direct regression prediction mechanism (preferably referring to the YOLOv8-pose architecture).

[0097] Specifically, this branch, in the decoupled feature map space, directly treats the spatial coordinates of target keypoints as an inherent attribute of the target instance for numerical regression. For each valid target candidate instance, the network directly predicts and outputs a vector of length 3N (where N is the number of keypoints in a single instance). In the specific scenario of tomato internode detection, the keypoints are defined as the two physiological nodes at the two ends of the internode, corresponding to N=2. This predicted vector not only includes the two-dimensional normalized coordinate offsets (x, y) of these two keypoints relative to the grid or the target center, but also simultaneously outputs a visibility confidence score characterizing whether the node is occluded.

[0098] Through this direct regression mechanism, the network can quickly locate the physiological endpoint in a single inference.

[0099] (4) Establish the instance-level correspondence between instance segmentation mask and key point 2D coordinates.

[0100] Compared to traditional serial segmentation and keypoint detection schemes, multi-model cascaded approaches are prone to error accumulation. Furthermore, in situations where greenhouse foliage obstructs the view and targets are adjacent to each other, mismatches may occur between the segmentation masks and keypoint results for different targets, necessitating additional post-processing alignment steps. To mitigate these issues, this invention enables the target recognition model to output instance segmentation results and keypoint results for the same target tomato internode instance, and establishes an instance-level correspondence between the instance segmentation results and keypoint results based on the same instance output.

[0101] 1. Methods for determining instance-level correspondences

[0102] The core of the instance-level correspondence is that the multi-task joint prediction network assigns a unified instance identifier to each retained candidate target instance during a single forward propagation process, and outputs bounding boxes, segmentation results and key point results synchronously based on the instance identifier, thereby establishing a one-to-one correspondence between the three in the output stage.

[0103] Specifically, suppose the joint prediction network for the first... The joint output set of n target instances is denoted as . Then it can be expressed as:

[0104] in: Indicates the first Two-dimensional bounding boxes of each target instance; Indicates the first Pixel-level instance segmentation mask for each target instance; Indicates the first A set of key points for each target instance.

[0105] In the tomato internode detection scenario addressed by this invention, This includes the two-dimensional coordinates of the physiological nodes at both ends of the segment and their visibility confidence.

[0106] The segmentation mask is obtained by sharing the same instance identifier. With key point set They are identified as belonging to the same target instance, thus realizing an instance-level correspondence of "one instance corresponds to one segmentation result and a set of key point results", reducing cross-instance feature mismatch.

[0107] 2. The joint constraint effect of heterogeneous visual features

[0108] Through the above instance-level correspondence, the segmentation results and key point results together constitute the visual constraint input between target segments when subsequently mapped to three-dimensional space.

[0109] Wherein: Spatial region constraint: The instance segmentation mask This mask is used to define the pixel-level region range of the target instance in a two-dimensional image. During mapping to three-dimensional space, the mask serves as a spatial filtering condition, used to extract depth point cloud information of the corresponding intersegmental regions from the depth image obtained in step one, thereby reducing background noise and interference from adjacent structures.

[0110] Structural endpoint constraints: the set of key points This is used to define the structural endpoint positions and topological positions at both ends of the target instance. After completing the 3D mapping, the key point results are converted into endpoint constraint information in spatial coordinates to assist in subsequent steps such as inter-segment curve fitting, centerline extraction, and physical length calculation.

[0111] By combining regional-level visual features with structural-level visual features, this invention improves the consistency and stability of mapping two-dimensional results to three-dimensional geometric information.

[0112] 3. Equivalent extensions of implementation methods

[0113] In a preferred embodiment, the instance-level correspondence can be achieved by sharing the same candidate instance index. In other embodiments, the instance-level correspondence between the segmentation results and the keypoint results can also be achieved by sharing the same candidate detection box, sharing the same output position code, sharing a unified instance identifier, or other association benchmarks that can characterize the same target instance.

[0114] It should be noted that the instance-level correspondence is not limited to the specific implementation described above. Any technical solution that establishes a clear correspondence between the segmentation results and keypoint results based on the same target benchmark at the output of a multi-task network, and reduces additional post-processing alignment steps and improves the consistency of 2D-to-3D mapping, can be applied to this invention.

[0115] (5) Model training and multi-task joint optimization

[0116] To achieve end-to-end training of the multi-task parallel perception visual model, this step uses a multi-task joint loss function to uniformly optimize the visual model based on the constructed tomato plant scene dataset.

[0117] 5.1 Construction of Training Dataset

[0118] During the model training phase, a multi-task training dataset of tomato plants was constructed. The training dataset includes RGB images of tomato plants collected under different growth stages, different degrees of occlusion, and different lighting conditions. The target bounding boxes, instance segmentation masks, keypoint 2D coordinates, and node visibility labels were annotated to support joint training of target detection, instance segmentation, and keypoint localization tasks.

[0119] In one embodiment, the key points in the training dataset are preferably defined as the physiological nodes at both ends of the tomato internodes, so as to be consistent with the key point prediction task in step two and the three-dimensional endpoint mapping process in step three.

[0120] 5.2 Construction of Joint Loss Function for Multiple Tasks

[0121] To coordinate the optimization directions of different tasks during backpropagation, a unified multi-task joint loss function is used to train the visual model. This joint loss function includes detection loss, segmentation loss, keypoint localization loss, and keypoint visibility loss. Different loss terms are weighted using preset weighting coefficients to balance the gradient contributions of each task during training.

[0122] In a preferred embodiment, the multi-task joint loss function It can be represented as:

[0123] in, These are the weighting coefficients for the losses of each subtask, used to adjust the relative contributions of different tasks in the joint optimization. These weighting coefficients can be set and adjusted according to specific application scenarios, training data distribution, or validation set performance.

[0124] 5.3 Composition and Function of Loss Items

[0125] In one embodiment, the loss of each subtask in the joint loss function may include the following components:

[0126] (1) Detection branch loss: The detection branch loss includes category classification loss, bounding box regression loss and localization distribution loss, which are used to constrain the category discrimination and spatial localization accuracy of target instances.

[0127] (2) Segmentation branch loss: The segmentation branch loss is used to constrain the difference between the predicted segmentation mask and the true segmentation mask in order to improve the accuracy and boundary integrity of the segmentation of the internode region of tomato.

[0128] (3) Key point localization branch loss: The key point localization branch loss includes key point position regression loss and key point visibility loss, which are used to improve the accuracy of key point position prediction and enhance the model's ability to judge key point visibility in the case of occlusion.

[0129] In a preferred embodiment, the losses for each of the above sub-tasks can be implemented using existing loss function design methods. For example, the category classification loss can use binary cross-entropy loss or focal loss, the bounding box regression loss can use CIoU, GIoU, DIoU or other bounding box regression losses, the localization distribution loss can use distribution focal loss, the segmentation loss can use pixel-level binary cross-entropy loss, the keypoint localization loss can use coordinate regression loss or OKS-based scale-normalized coordinate regression loss, and the visibility loss can use binary cross-entropy loss or other classification losses.

[0130] By employing the aforementioned joint optimization method, the model can improve the regional accuracy of instance segmentation results and the stability of key point localization results while ensuring target detection performance, thereby providing higher quality visual input for subsequent steps such as 3D point cloud extraction, endpoint constraints, and curve modeling.

[0131] 5.4 Equivalent Extensions of Implementation Methods

[0132] It should be noted that the specific loss function types and their combinations described above are merely preferred implementations of this invention in the tomato internode detection scenario and do not constitute a limitation of this invention. Any training method capable of joint gradient backpropagation and collaborative optimization for object detection, instance segmentation, and keypoint localization tasks can be applied to this invention.

[0133] (6) Model output results and technical effects

[0134] The multi-task parallel perception vision model, trained as described above, can be deployed in actual tomato plant detection scenarios for end-to-end inference. The RGB color image of the tomato plant to be detected, obtained in step one, is input into the vision model. During a single forward propagation, the model synchronously outputs the following results for each detected tomato internode target instance:

[0135] 1. Object detection results: Two-dimensional bounding box and class confidence score for each object instance;

[0136] 2. Region segmentation results: pixel-level instance segmentation mask for each target instance;

[0137] 3. Topology localization results: Two-dimensional coordinates of a preset number of key points within each target instance and their visibility confidence scores.

[0138] In the tomato internode detection scenario, the key points are preferably physiological nodes at both ends of the internode, corresponding to the number of key points. .

[0139] The segmentation results and keypoint results output synchronously above share the same target instance identifier or candidate instance benchmark, thus establishing a correspondence at the instance level and forming a joint two-dimensional visual feature output. This joint two-dimensional visual feature output can serve as input constraints for subsequent steps such as 3D point cloud extraction, cross-modal space mapping, topology analysis, and inter-segment physical length measurement.

[0140] Through the above end-to-end inference process, the visual model can simultaneously complete target detection, instance segmentation and key point localization in a single inference process, reducing the repetitive feature extraction process and additional result alignment steps in the traditional serial "segmentation-key point" cascade scheme, thereby improving processing efficiency and result consistency.

[0141] Compared to existing single-stage detection networks or serial cascaded visual processing schemes, the multi-task parallel perception mechanism constructed in this invention has the following technical advantages:

[0142] Firstly, by introducing deformation-adaptive feature enhancement and multi-scale context aggregation structure during feature extraction and feature fusion, the model's ability to perceive the slender, curved, and shading features of tomato stems in complex greenhouse environments is improved.

[0143] Secondly, by constructing a parallel prediction structure and instance-level correspondence in a unified shared feature space, the instance segmentation results and key point results are established in the output stage, which reduces the error accumulation and additional post-processing complexity caused by serial processing of multiple models.

[0144] Third, it provides a unified and stable visual constraint input for the subsequent mapping of two-dimensional visual features to three-dimensional point clouds and geometric models, thus providing a reliable basis for the measurement of physical parameters of tomato internodes.

[0145] Step 3: Mapping 2D features to 3D space based on the camera model

[0146] To achieve cross-modal conversion from two-dimensional image pixel space to three-dimensional physical space, this step, based on the camera intrinsic parameter model, maps the two-dimensional visual features extracted in step two, including instance segmentation masks and key point coordinates, into three-dimensional point cloud data, and forms the three-dimensional spatial constraint information corresponding to the target segments.

[0147] 3.1 Three-dimensional spatial back projection based on depth camera model

[0148] Using a depth imaging device, such as a ZED stereo camera or other RGB-D sensor, depth data aligned with a two-dimensional color image is acquired. By calling the point cloud acquisition module in the device's application programming interface or software development kit, a back-projection transformation from two-dimensional pixel coordinates to a three-dimensional camera coordinate system is performed based on the pinhole camera's intrinsic parameter model, thereby outputting the three-dimensional spatial coordinates corresponding to each pixel within the effective field of view. and color information .

[0149] In one embodiment, the point cloud results output by the depth camera SDK can be used directly; in another embodiment, the depth image can be manually back-projected based on the camera intrinsic parameters to obtain the corresponding 3D point cloud data.

[0150] 3.2 Three-dimensional Region of Interest Extraction Based on Two-Dimensional Features

[0151] like Figure 6 As shown, based on the two-dimensional features jointly output in step two, the three-dimensional information of the corresponding target tomato internodes is extracted from the panoramic three-dimensional point cloud data.

[0152] in:

[0153] 1. Point cloud extraction of masked regions: For the pixels within the masked area of ​​the instance segmentation... Using it as a spatial index, the three-dimensional coordinates of the corresponding position in the point cloud data are directly read or back-projected to form the initial three-dimensional point cloud of the region of interest corresponding to the target segment;

[0154] 2. Keypoint 3D Anchoring: For the inter-node endpoints of the keypoint prediction branch output, anchoring is based on their 2D pixel coordinates. Effective depth information of the neighborhood is extracted from the 3D point cloud aligned with the color image to generate 3D coordinates of the intersegment endpoints, which are used to characterize the start and end positions of the target intersegment in 3D space.

[0155] Through the above 3D extraction process, the instance segmentation mask is used to limit the spatial range of the target segment, and the key point results are used to provide structural endpoint constraints at both ends of the segment. Together, they constitute the 3D prior input for subsequent point cloud optimization, spatial curve modeling, and physical length calculation.

[0156] Step 4: Point cloud optimization, spatial curve modeling, and physical length calculation

[0157] The initial 3D point cloud of the region of interest obtained in the third step is subjected to spatial refinement and geometric modeling to solve the problems of noise interference, structural breaks, and measurement errors of curved structures caused by using straight-line ranging in the original point cloud.

[0158] 4.1 Point Cloud Local Noise Suppression Processing

[0159] Technical issue: Redundancy and local outliers exist in the initial point cloud of the region of interest, affecting the stability of subsequent clustering screening and curve fitting.

[0160] Technical solution: Perform statistical feature analysis on the main structure point cloud to filter out local outliers. In one embodiment, a statistical outlier filtering method is used: calculate each point and its nearest neighbor. The average distance between each point is calculated, and the mean of the overall average distance is also calculated. and standard deviation If the average distance of a point is greater than If it is an outlier, it will be identified as an outlier and removed. This is the preset threshold coefficient.

[0161] In other embodiments, moving least squares, radius filtering, or other point cloud smoothing methods may also be used for local noise suppression.

[0162] Technical effect: By suppressing point cloud noise, cleaner and more continuous stem internode point cloud data can be obtained, providing a more stable geometric input for subsequent 3D curve fitting.

[0163] 4.2 Point Cloud Clustering and Principal Structure Optimization Based on Multi-Feature Evaluation Function

[0164] Technical issue: The initial region of interest point cloud often contains background leaves, supports, and floating noise, which can cause the geometric modeling to deviate from the real stem structure if used directly.

[0165] Technical Solution: First, based on the three-dimensional coordinates of the internode endpoints, a depth constraint range is determined, and depth bandpass filtering is performed on the denoised region of interest point cloud to suppress interference from foreground blades and background structures. Then, a spatial density-based clustering algorithm is used to cluster the filtered three-dimensional point cloud. In one embodiment, the DBSCAN algorithm is used for clustering, by setting a neighborhood radius. The minimum number of contained points is used to group points with similar spatial densities into the same cluster, and low-density isolated points are marked as noise.

[0166] Subsequently, a multi-feature fusion evaluation function is constructed to assess the quality of each cluster. The evaluation function considers at least the following factors:

[0167] 1. Spatial distance characteristics: Calculate the spatial Euclidean distance between the point cloud cluster and the three-dimensional coordinates of the internode endpoints, for example, by taking the quantile of the distance as the spatial distance cost. ;

[0168] 2. Depth Consistency Feature: Compare the depth distribution of point cloud clusters with the 3D coordinates of internode endpoints, and apply a depth penalty term to clusters whose depth is significantly smaller than that of the internode endpoints. To identify interference clusters that may be located in foreground occlusion positions.

[0169] In one embodiment, the evaluation function can be expressed as:

[0170] in, and These are preset weighting coefficients.

[0171] Based on the calculation results of the evaluation function, one or more point cloud clusters with the smallest evaluation value are selected as the true main structural point cloud of the stem. When there are two internode endpoint three-dimensional coordinates, the clusters with the best evaluation value relative to the starting and ending anchors can be selected and merged.

[0172] Technical effect: Through the multi-feature evaluation mechanism, the accuracy and robustness of main stem point cloud extraction can be improved, and the impact of complex background and occlusion structure on subsequent modeling can be reduced.

[0173] 4.3 Spatial Curve Reconstruction and Endpoint Extrapolation

[0174] Technical issues: Real plant stems are usually naturally curved, and using straight-line fitting will produce systematic truncation errors; at the same time, due to local occlusion or lack of depth, the point cloud at the endpoints may be incomplete.

[0175] Technical solution:

[0176] 1. Central skeleton point extraction: The slice projection method is used to determine the principal axis direction of the point cloud based on principal component analysis. The point cloud is divided into several slices along the principal axis direction, and the median coordinates or center coordinates of the point cloud in each slice are calculated as skeleton points, thus forming an ordered skeleton point sequence.

[0177] 2. Three-dimensional smooth curve fitting: A parametric spatial curve model is constructed to fit the skeleton point sequence. In a preferred embodiment, a cubic B-spline curve is used to smoothly fit the center point sequence to obtain a spatial parametric curve. ,in .

[0178] 3. Endpoint extrapolation compensation: Calculate the tangent vectors at both ends of the fitted curve and project the three-dimensional coordinates of the internode endpoints onto the curve. When the three-dimensional coordinates of the internode endpoints are located outside the end of the fitted curve, linear extrapolation compensation is performed along the tangential direction of the corresponding curve endpoint. The extrapolation length is taken as the projection distance from the curve endpoint to the three-dimensional coordinates of the internode endpoint along the tangential direction.

[0179] Technical effects: By modeling with smooth spatial curves, the bending geometry of tomato stems can be represented more realistically; by extrapolating the endpoints, the length truncation error caused by missing local point clouds can be reduced.

[0180] 4.4 Calculation of Curved Path Length and Output of Physical Quantities

[0181] Technical issue: It is necessary to accurately convert discrete three-dimensional points and fitted curves into the physical length of the segments to reduce the systematic underestimation error caused by straight-line distance measurement.

[0182] Technical solution:

[0183] 1. Using the sampling projection method, the three-dimensional coordinates of the intersegmental endpoints are projected onto the fitted parametric curve. Above, determine the boundary parameters of the curve corresponding to the start and end points of the tomato internodes. and ;

[0184] 2. Within the parameter range Uniform sampling is performed within the curve, the Euclidean distance between adjacent sampling points is calculated and accumulated to obtain the arc length of the corresponding part of the fitted curve. ;

[0185] 3. The arc length of the curve is compared with the extrapolated lengths at both ends obtained in step 4.3. and Adding them together, we get the final total physical length of the tomato internodes:

[0186] Output the This is a three-dimensional measurement result of the internode length of tomatoes.

[0187] Technical effect: By calculating the path length along a spatial curve, the systematic error generated when using linear Euclidean distance estimation can be reduced, thereby improving the morphological measurement accuracy of non-rigid curved organs of crops.

[0188] To further illustrate the feasibility and effectiveness of the method of the present invention, this embodiment provides an experimental scheme for measuring tomato internodes based on multi-task parallel sensing and three-dimensional mapping. This embodiment automatically detects and measures the physical length of tomato internodes in a greenhouse setting, following the processing flow of steps one to four described above.

[0189] 1. Experimental Environment and Hardware / Software Configuration

[0190] In this embodiment, model training, 2D visual inference, and 3D point cloud processing are all performed on a unified computing platform. In one embodiment, the computing platform includes a high-performance central processing unit, large-capacity memory, and a graphics processing unit that supports deep learning acceleration. In a preferred embodiment, a computing device using an Intel Xeon series processor, an NVIDIA GeForce RTX 4090 graphics processor, and 256GB of memory may be used.

[0191] Regarding the software environment, Windows or Linux operating systems can be used, and model training and inference programs can be built based on the Python language and the deep learning framework PyTorch; at the same time, CUDA and cuDNN are combined to achieve underlying GPU acceleration. It should be noted that the above-mentioned hardware and software environment is only a preferred embodiment of the present invention and does not constitute a limitation of the present invention.

[0192] 2. Visual perception model construction and training parameter settings

[0193] In this embodiment, a multi-task parallel perception architecture based on a single-stage detection network modification is used as the visual model.

[0194] The backbone feature extraction network includes a deformation adaptive feature enhancement module, the feature fusion network includes a multi-scale context aggregation module, and the prediction head is set to a decoupled parallel output structure to synchronously output two-dimensional target bounding boxes, instance segmentation masks, and key point coordinates.

[0195] In the tomato internode detection scenario, key points are preferentially defined as the physiological nodes at both ends of the internode, corresponding to the number of key points. .

[0196] During the model training phase, the input image can be uniformly scaled and filled to a preset size. In a preferred embodiment, the input image size can be set to... Pixels. The batch size, number of data loading threads, and number of training epochs for model training can be set according to the data scale and hardware resources. In a preferred embodiment, the training batch size can be set to 16-32, the number of data loading threads can be set to 4-8, and the number of training epochs can be set to 100-300; for example, the batch size can be set to 32, the number of data loading threads can be set to 8, and the total number of training epochs can be set to 300.

[0197] 3. Setting Parameters for 3D Point Cloud Processing and Geometric Fitting

[0198] After completing the 2D visual inference, the depth camera SDK is called to obtain the 3D point cloud of the scene, and the 3D region of interest point cloud between the target segments is extracted based on the instance segmentation mask and key point results.

[0199] When performing principal structure optimization, noise suppression, and curve fitting on the extracted 3D point cloud, the following parameter settings can be used in this embodiment:

[0200] (1) Point cloud denoising employs a statistical outlier filtering method to purify the main structure point cloud. In one embodiment, the number of nearest neighbor points... The standard deviation multiplier threshold can be set from 10 to 50. It can be set to 1 to 3. In a preferred embodiment, it can be set to... , .

[0201] (2) Centerline Extraction and Curve Fitting: PCA principal axis extraction is performed on the denoised point cloud, and slices are made along the principal axis direction. The number of slices can be adaptively set according to the point cloud density. In a preferred embodiment, the number of slices can be set to 20-120. Subsequently, global spatial curve fitting is performed on the median points of each slice. Preferably, a cubic B-spline curve can be used to fit the natural curvature of the tomato stem.

[0202] (3) Endpoint extrapolation compensation projects the three-dimensional key points output by the network onto the fitted curve. When the physiological nodes exceed the coverage of the curve, linear extrapolation compensation can be performed according to the direction of the tangent vector at the endpoint of the curve to obtain a more complete intersegmental curve path length.

[0203] It should be noted that the point cloud processing parameters and curve fitting parameters described above are only example settings in the preferred embodiment and can be adjusted according to the point cloud density, target scale, and measurement accuracy requirements.

[0204] 4. Experimental Results and Effect Verification

[0205] After testing on a self-built greenhouse tomato multimodal dataset, the method described in this embodiment can still stably complete the automatic detection and length measurement of tomato internodes under conditions of shading by branches and leaves and bending of stems.

[0206] In one embodiment, the deviation between the measurement results of the present invention and the actual measured values ​​can be evaluated using indicators such as mean absolute error (MAE), mean square error (RMSE), or relative error. In a preferred embodiment, the mean absolute error between the physical length of tomato internodes measured by the method of the present invention and the actual measured values ​​can be controlled within a small range. Compared with traditional measurement methods based on single-model detection and linear Euclidean distance estimation, the present invention can achieve higher measurement accuracy and better robustness under complex shading and natural curvature conditions.

[0207] The average absolute error between the physical length of tomato internodes measured by the method of the present invention and the actual measured value is 1.07 mm; compared with the control measurement method that does not introduce depth bandpass constraint and point cloud cluster optimization, the average absolute error is reduced by about 74.2%.

[0208] 5. Conclusion of the Example

[0209] This embodiment demonstrates that the present invention, through its overall technical process of "multi-task parallel visual perception—two-dimensional feature three-dimensional mapping—point cloud main structure optimization—spatial curve arc length calculation," can achieve automated and robust physical length measurement of tomato internodes in unstructured greenhouse scenarios. Compared to traditional schemes that rely on a single detection result or straight-line distance measurement, the present invention exhibits better geometric characterization capabilities and measurement accuracy under conditions of bent stems and shading.

Claims

1. A three-dimensional measurement method for tomato internode length, characterized in that, Includes the following steps: Step 1: Multimodal Data Acquisition and Spatial Alignment RGB-D depth imaging equipment was used to acquire pixel-aligned RGB color and depth images of the target tomato plant, and the camera intrinsic and extrinsic parameter matrices were collected. During the acquisition process, the working distance between the equipment and the main stem of the plant was kept within the range of 26-45cm, and the acquisition area was focused on the fully developed internodes in the subapical mature zone of the tomato plant. Step 2: Joint extraction of 2D visual features based on multi-task parallel perception; The RGB color image is input into a pre-trained multi-task parallel perception visual model, and a single end-to-end inference is performed to synchronously output the instance segmentation mask and the 2D coordinates of key points between the target tomato nodes; wherein, the instance segmentation mask and the 2D coordinates of key points form a one-to-one correspondence based on the output results of the same target instance; Step 3: 3D spatial mapping and cross-modal constraint generation based on 2D visual features: Based on the camera intrinsic parameter matrix and depth image obtained in step one, the instance segmentation mask and keypoint 2D coordinates output in step two are mapped to 3D space respectively: - Extract the 3D point cloud corresponding to the area covered by the instance segmentation mask, and use it as the initial region of interest point cloud between target segments; - Generate the three-dimensional coordinates of inter-segment endpoints based on the two-dimensional coordinates of key points and the effective depth information of the neighborhood in the aligned point cloud; Step 4: Point cloud filtering, spatial curve modeling, and physical length calculation; Step four: point cloud filtering, spatial curve modeling, and physical length calculation specifically includes: 4.1 The initial region of interest point cloud is subjected to voxel downsampling and statistical outlier filtering for local noise suppression to obtain a continuous and clean stem internode point cloud; 4.2 Spatial density clustering algorithm is used to cluster the point cloud of the region of interest after depth bandpass filtering, and a multi-feature evaluation function including spatial distance cost and depth consistency cost is constructed. The three-dimensional coordinates of the internode endpoints are used as reference benchmarks to evaluate and filter each point cloud cluster. The point cloud cluster with the best evaluation value is selected as the target stem main structure point cloud, and the interference point cloud clusters composed of background leaves or supports are removed. 4.3 Based on the point cloud of the main structure of the target stem, extract the central skeleton point sequence and construct a three-dimensional spatial parameterized curve to fit the natural bending shape of the tomato stem; 4.4 Project the three-dimensional coordinates of the intersegment endpoints onto the fitted curve. If the three-dimensional coordinates of the intersegment endpoints are located outside the endpoints of the fitted curve, perform linear extrapolation compensation along the tangent vector direction of the curve endpoints. 4.5 Calculate the arc length of the fitted curve and add it to the extrapolated compensation length to output the final three-dimensional physical length of the tomato internode.

2. The three-dimensional measurement method for tomato internode length according to claim 1, characterized in that, Step two, based on the joint extraction of two-dimensional visual features using multi-task parallel perception, specifically includes: The visual model includes a backbone feature extraction network, a feature fusion network, and a multi-task prediction network; the backbone feature extraction network is equipped with a deformation adaptive feature extraction module, which is used to dynamically adjust the feature sampling grid to adapt to the bending shape of the tomato stem; The feature fusion network includes a multi-scale context aggregation module for extracting and fusing local details and global semantic features from different receptive fields. The multi-task prediction network includes an instance segmentation branch and a key point localization branch, which respectively output pixel-level instance segmentation masks and two-dimensional coordinates and visibility confidence of key points.

3. The three-dimensional measurement method for tomato internode length according to claim 2, characterized in that, The deformation adaptive feature extraction module in step two adopts a C2f-DCNv3 structure; Its internal processing mechanism is as follows: the input feature map is divided into a first branch and a second branch after channel mapping by the initial convolutional layer. The first branch serves as a skip connection, and the second branch is serially input into several cascaded bottleneck structures. The bottleneck structure uses deformable convolutional units DCNv3 to replace standard convolutions. By dynamically learning the two-dimensional spatial offset and modulation scalar of the convolution kernel sampling points, adaptive feature alignment for curved targets is achieved. Finally, the features of the first branch and the output features of the second branch are spliced ​​and fused.

4. The three-dimensional measurement method for tomato internode length according to claim 1, characterized in that, The multi-scale context aggregation module adopts the MSBlock structure; Its internal processing mechanism is as follows: the input feature map is grouped by channel or a multi-branch parallel structure, and then enters multiple convolutional branches with different dilation rates. The multi-scale features extracted by each branch are fused and aggregated step by step in the channel dimension, and the output is an enhanced feature map with rich contextual information.

5. The three-dimensional measurement method for tomato internode length according to claim 2, characterized in that, The instance segmentation branch adopts a generation mechanism based on "prototype-coefficient" decoupling, including: a prototype generation module for generating a preset number of globally shared mask prototypes; a mask coefficient prediction module for predicting a set of mask coefficients for each detected candidate target instance; and linearly combining the mask coefficients with the mask prototypes and processing them through an activation function to decode and generate a pixel-level segmentation mask corresponding to the specific instance.

6. The three-dimensional measurement method for tomato internode length according to claim 2, characterized in that, The key point localization branch adopts a direct regression prediction mechanism without heatmaps, directly outputting the two-dimensional normalized coordinate offset of the key point relative to the preset reference point and the visibility confidence. In the tomato internode detection scenario, the key point is defined as the physiological nodes at both ends of the internode, and the number of key points N=2.

7. The three-dimensional measurement method for tomato internode length according to claim 2, characterized in that, The one-to-one correspondence between the instance segmentation mask and the two-dimensional coordinates of the key points is as follows: The joint prediction network assigns a uniform instance identifier to each retained candidate target instance during a single forward propagation; its joint output set is represented as follows. ,in, For the first Two-dimensional bounding boxes of each target instance. For the first Pixel-level instance segmentation mask for each target instance For the first A set of key points for each target instance; By sharing the same instance identifier, an instance-level one-to-one correspondence is established between the segmentation mask and the key point set during the output stage.

8. The three-dimensional measurement method for tomato internode length according to claim 3, characterized in that, The multi-feature evaluation function constructed in step four is expressed as follows: in, The cost is the spatial distance between the point cloud cluster and the three-dimensional coordinates of the internode endpoints. This is a depth consistency penalty term for point cloud clusters relative to the 3D coordinates of internode endpoints. and These are preset weighting coefficients used to adjust the relative contributions of spatial distance cost and depth consistency penalty.

9. The three-dimensional measurement method for tomato internode length according to claim 3, characterized in that, The specific method for extracting the central skeleton points and fitting the three-dimensional parametric curve in step 4.3 is as follows: Principal component analysis is used to determine the principal axis direction of the point cloud. The point cloud is divided into several slices along the principal axis direction. The median coordinates of the point cloud in each slice are calculated as skeleton points to form an ordered skeleton point sequence. The skeleton point sequence is fitted with a cubic B-spline curve using least squares to obtain a spatially parameterized curve. ,in .

10. The three-dimensional measurement method for tomato internode length according to claim 3, characterized in that, The specific method for calculating the arc length of the fitted curve in step 4.5 is as follows: Project the three-dimensional coordinates of the intersegmental endpoints generated in step three onto the spatial parameterized curve. Determine the boundary of curve parameters corresponding to the three-dimensional coordinates of the tomato internode endpoints. and ; In the parameter range Uniform sampling is performed within the curve, and the arc length is calculated by accumulating the Euclidean distance between adjacent sampling points. ; Calculate the physical length of the tomato internode as L total = + extstart +L extend ,in extstar and L extend These are the extrapolated compensation lengths for the starting and ending points obtained in step 4.4, respectively.

11. The three-dimensional measurement method for tomato internode length according to claim 2, characterized in that, The visual model is based on the single-stage object detection network YOLOv8 and is trained end-to-end using a multi-task joint loss function. The joint loss function includes detection loss, segmentation loss, keypoint localization loss, and keypoint visibility loss, and uses preset weight coefficients to balance the gradient contributions of each subtask during training. The detection loss includes category classification loss, bounding box regression loss, and localization distribution loss. The segmentation loss includes mask binary cross-entropy loss. The keypoint localization loss uses OKS loss. The keypoint visibility loss uses binary cross-entropy loss.

12. A three-dimensional tomato internode length measurement system based on the RGB-D fusion and multi-task perception model implemented by the three-dimensional tomato internode length measurement method of claim 1, characterized in that, include: The multimodal data acquisition module is used to acquire RGB color images and depth images of the target tomato plant, and perform pixel-level spatiotemporal alignment on the two, while extracting the camera intrinsic and extrinsic parameter matrices; it outputs the aligned RGB color image, depth image and camera parameters. The output of the multimodal data acquisition module is connected to the input of the multi-task parallel perception vision model module and the three-dimensional space mapping module, respectively. The multi-task parallel perception vision model module internally stores a pre-trained multi-task parallel perception vision model, which is used to receive the RGB color image output by the multimodal data acquisition module, perform a single end-to-end inference, synchronously output the instance segmentation mask and key point two-dimensional coordinates of each tomato internode target instance, and based on the output results of the same target instance, make the instance segmentation mask and key point two-dimensional coordinates form a one-to-one correspondence. The 3D spatial mapping module receives the depth image and camera parameters output by the multimodal data acquisition module, as well as the instance segmentation mask and key point 2D coordinates with instance-level correspondence output by the multi-task parallel perception vision model module; based on the camera parameters and depth image, it maps the instance segmentation mask to an initial region of interest point cloud in 3D space, and simultaneously maps the key point 2D coordinates to intersegment endpoint 3D coordinates in 3D space; the initial region of interest point cloud is used as 3D processing input, and the intersegment endpoint 3D coordinates are used as 3D prior information, and both are output to the point cloud processing and curve modeling module; The point cloud processing and curve modeling module is used to receive the initial region of interest point cloud and the three-dimensional coordinates of the inter-segment endpoints output by the three-dimensional space mapping module. The output module is used to receive the arc length of the internode centerline curve and the extrapolation compensation length of the endpoints output by the point cloud processing and curve modeling module, and output the final three-dimensional physical length measurement result of the tomato internode.

13. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by one or more processors, implement the three-dimensional measurement method for tomato internode length as described in any one of claims 1 to 11.

Images