Tomato axillary bud identification and two-dimensional inactivation positioning method

Abstract

The application discloses a tomato axillary bud recognition and two-dimensional inactivation positioning method and belongs to the field of tomato cultivation. In order to solve the problems of insufficient precision of existing pure two-dimensional detection lacking geometric supervision and easy increase of reasoning overhead and hardware cost caused by introduction of depth estimation, a joint network model containing a posture estimation detection head and a parallel light-weight depth auxiliary head is constructed; a depth label is generated offline, geometric supervision is exerted on a shared network through four-channel synchronous enhancement and weighted depth loss; during reasoning, the depth auxiliary head is skipped to calculate, and during derivation, the auxiliary head is removed through dead code elimination or physical pruning; and inactivation is completed based on detection key points, pixel-physical mapping conversion and laser execution coordinates. The application improves detection precision, realizes zero additional reasoning overhead, is light in deployment, low in hardware cost and suitable for the field of tomato cultivation.

Description

Technical Field

[0001] This invention relates to the field of optimizing tomato cultivation using image understanding technology, specifically to a method for identifying and locating tomato axillary buds in two dimensions. Background Technology

[0002] Real-time identification and precise inactivation of axillary buds (lateral buds) in tomatoes (also known as tomatoes, Solanum lycopersicum L.) are key agronomic measures in greenhouse tomato cultivation. The growth of axillary buds consumes plant nutrients and competes with the main stem and fruit for resources, leading to a decline in fruit yield and quality. Traditional manual inspection and removal methods are labor-intensive and inefficient, making it difficult to meet the uniform management requirements of large-scale greenhouse production. With the development of computer vision and laser technology, automatic identification based on visual detection and laser-based targeted inactivation have become important technological development directions for replacing manual labor.

[0003] In existing technologies, deep learning-based target detection and pose estimation models, such as YOLOv11-Pose keypoint detection models, have been attempted for target recognition and localization in agricultural scenarios. However, purely two-dimensional detection methods generally face insufficient accuracy in complex greenhouse agricultural scenarios. In greenhouse environments, tomato axillary buds are small in scale and face challenges such as severe shading by branches and leaves and drastic changes in light conditions. Existing methods rely solely on RGB texture and color features, lacking the ability to perceive and learn spatial geometric structures. They cannot use spatial hierarchical relationships to help distinguish axillary buds from surrounding stem and leaf tissues, resulting in keypoint localization accuracy often failing to meet the pixel-level accuracy requirements of subsequent laser focusing. This can easily lead to inactivation point offsets or even accidental inactivation of the main stem and fruit.

[0004] To compensate for the limitations of pure 2D detection in perceiving spatial geometric information, existing technologies have introduced monocular depth estimation models to assist in 3D spatial understanding, such as MiDaS, DPT, and Depth Anything V2. However, these independent depth estimation models typically have between 50M and 300M parameters. If deployed in conjunction with detection models, they incur significant inference overhead, making it difficult to meet the real-time requirements of greenhouse edge computing devices. More importantly, independent depth models only optimize for depth estimation during training, and the geometric features they extract differ significantly from the feature distribution required for detection, failing to provide effective geometric supervision for the detection network. Existing technologies also include solutions that utilize depth cameras to directly acquire 3D point cloud data of plants. However, depth cameras not only significantly increase system hardware costs but are also prone to imaging interference in the complex environment of greenhouses with high humidity, high temperature, and dust, resulting in insufficient long-term operational stability.

[0005] Some existing solutions employ a multi-task learning framework, adding a depth prediction branch at the end of the detection network. This allows for joint training of the detection and depth estimation tasks by sharing a backbone feature network, aiming to enhance the network's spatial feature learning capabilities. However, such solutions still have several shortcomings in practical applications: most solutions still perform forward computation of the depth branch during the inference phase. Even if the final localization does not depend on depth information, this generates unnecessary floating-point operations and memory usage. For agricultural edge devices with limited computing power, even a small amount of additional computation can significantly impact system real-time performance. Existing multi-task solutions lack a smooth transition mechanism from training to deployment. The multi-branch model in the training phase is inconsistent with the single-task model in the deployment phase, increasing the cost of model maintenance and version management, and easily leading to weight compatibility issues. Obtaining pixel-level depth ground truth is costly. In agricultural scenarios, acquiring large-scale labeled depth data requires specialized acquisition equipment or manual point-by-point labeling, with labeling costs far exceeding those of ordinary object detection tasks, further limiting the large-scale application of depth-supervised solutions in agricultural vision.

[0006] At the end of target detection applications, existing laser inactivation systems mostly adopt a three-dimensional spatial positioning approach. That is, after obtaining the target's three-dimensional coordinates through depth information, they are converted into laser execution coordinates by combining the hand-eye calibration matrix. This approach requires completing multiple complex processes such as camera intrinsic parameter calibration, hand-eye calibration, and depth sensor calibration. The system integration and debugging are difficult and the maintenance cost is high. For greenhouse scenarios with fixed cameras and fixed lasers, the over-reliance on three-dimensional information significantly increases the system integration complexity and hardware cost.

[0007] Therefore, how to introduce effective deep geometric supervision to improve the accuracy of key point detection in agricultural scenarios and simplify the system deployment and calibration process of end-effector laser inactivation without increasing inference overhead and hardware costs is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0008] To alleviate or partially alleviate the above-mentioned technical problems, the solution of the present invention is as follows:

[0009] This invention discloses a method for identifying and two-dimensionally inactivating tomato axillary buds, comprising:

[0010] Step S1: Obtain the training dataset, which includes RGB images, ground truth depth labels corresponding to the RGB images, and annotation information of the RGB images;

[0011] Step S2: Construct a joint network model, which includes a backbone network, a feature fusion network, a pose estimation detection head, and a depth auxiliary head connected in parallel to the feature fusion network;

[0012] Step S3: Input the RGB image into the joint network model. After the backbone network and the feature fusion network extract the fused feature map, the key point prediction results are output through the pose estimation detection head and the depth prediction results are output through the depth auxiliary head.

[0013] Step S4: Calculate the pose estimation loss based on the key point prediction results and the annotation information of the RGB image; calculate the depth auxiliary loss based on the depth prediction results and the depth ground truth labels corresponding to the RGB image; weight the pose estimation loss and the depth auxiliary loss to obtain the total loss; and update the parameters of the joint network model by backpropagation based on the total loss.

[0014] Step S5: After training is completed, the joint network model is switched to inference mode. In inference mode, the forward computation of the depth auxiliary head is skipped, and the depth auxiliary head is removed when exporting the joint network model to obtain the deployment model.

[0015] Step S6: The tomato plant image is inferred using the deployment model and the pixel coordinates of the key points of the axillary buds are output. Based on the pre-calibrated mapping relationship, the coordinates are converted into two-dimensional execution coordinates of the laser device, and the laser device is driven to complete the axillary bud inactivation.

[0016] In one embodiment, a data layer processing step is included before step S3:

[0017] The RGB image and the depth ground truth label are concatenated along the channel dimension to form a four-channel tensor;

[0018] Data augmentation operations are performed synchronously on the four-channel tensor. The data augmentation operations include at least one of mosaic enhancement, MixUp blending enhancement, random horizontal flipping, random rotation, random scaling, and random cropping. The data augmentation operations apply the same geometric transformation parameters to the channels of the RGB image and the channels of the depth ground truth label.

[0019] After data augmentation is completed, the four-channel tensor is separated into a three-channel RGB image and a single-channel depth ground truth label, which are then fed into the joint network model and the loss calculation module, respectively.

[0020] In one embodiment, in step S1, the depth truth label is generated in the following manner:

[0021] Load the pre-trained monocular depth estimation model, input all the RGB images into the pre-trained monocular depth estimation model for offline inference, and generate the corresponding original depth map;

[0022] The original depth map is preprocessed, invalid pixel values ​​are removed, and then normalization is performed, that is, the depth values ​​are mapped to the [0, 1] interval;

[0023] The normalized depth map is used as the ground truth label for the depth, and a one-to-one mapping relationship is established with the corresponding RGB image and stored.

[0024] In one embodiment, the structure of the depth-assisting head sequentially includes:

[0025] The first 1×1 convolutional layer is used to perform channel dimensionality reduction on the input fused feature map;

[0026] Bilinear upsampling layer is used to increase the spatial resolution of the dimensionality-reduced feature map by a factor of 2;

[0027] A 3×3 convolutional layer is used to aggregate spatial context information on the upsampled feature map to further compress the number of channels;

[0028] The second 1×1 convolutional layer is used to compress the number of channels to 1 and output a single-channel depth prediction map;

[0029] A parameterized modified linear unit activation layer is used to output prediction results adapted to the normalized relative depth; where,

[0030] The total number of parameters of the depth assist head is less than 50,000.

[0031] In one type of embodiment, the total loss is specifically expressed as:

[0032] L total = L pose + λL depth ;

[0033] Among them, L total For the total loss, L pose The loss is for attitude estimation, λ is a preset weighting coefficient for depth loss, and L is the depth loss value. depth For deep auxiliary loss;

[0034] The depth-assisted loss can be any one of L1 loss, inverse Huber loss, scale-invariant logarithmic loss, or mean squared error loss.

[0035] The pose estimation loss includes bounding box regression loss, classification loss, distribution focus loss, keypoint localization loss, and / or keypoint visibility loss.

[0036] In one embodiment, when calculating the depth-assisted loss, if the spatial resolution of the depth prediction map is inconsistent with that of the depth ground truth label, bilinear interpolation is used to adaptively align the depth prediction map to the resolution of the depth ground truth label, and the alignment operation is only applied to the depth prediction map. Then, the pixel-by-pixel error is calculated, the mean is taken, and then multiplied by the depth loss weighting coefficient to obtain the depth-assisted loss.

[0037] In one embodiment, skipping the forward computation of the depth-assist head in the inference mode specifically includes:

[0038] Conditional branch control is achieved by utilizing the training state attributes of the joint network model;

[0039] When the joint network model is in training mode, the forward computation of the depth auxiliary head is performed and the depth prediction result is output.

[0040] When the joint network model is in the inference mode, the forward computation of the depth auxiliary head is skipped directly, and the key point prediction results are output only by the pose estimation detection head.

[0041] In one embodiment, the depth auxiliary head is removed using an open neural network exchange format dead code elimination mechanism, specifically including:

[0042] Switch the trained joint network model to evaluation mode, and deactivate the forward propagation branch of the deep auxiliary head.

[0043] The Open Neural Network Exchange Format export operation is performed, and the symbolic tracking mechanism is used to track only the forward propagation path activated in the evaluation mode. The computation nodes of the deep auxiliary head are judged as dead code because they are never executed and are not included in the exported Open Neural Network Exchange Format computation graph.

[0044] In one embodiment, the depth assist head is removed by physical pruning, specifically including:

[0045] Remove the depth auxiliary head from the object in the joint network model;

[0046] Traverse the model weight parameter set and delete all weight parameter entries associated with the depth assist head;

[0047] If the optimizer state is retained, the momentum and variance states corresponding to the depth auxiliary head parameters in the optimizer state are deleted simultaneously.

[0048] If an exponential moving average strategy is used, the relevant weights of the depth auxiliary head are removed from the exponential moving average model object simultaneously;

[0049] The pruned weight parameters are loaded into the native keypoint detection model in strict loading mode for compatibility verification to ensure that there are no missing keys or unexpected keys, thus obtaining the deployment model.

[0050] In one embodiment, the pre-calibrated mapping relationship is converted into two-dimensional execution coordinates of the laser device, specifically including:

[0051] With the relative positions of the camera and the laser device fixed, images of the calibration board are acquired and the pixel coordinates of the feature points of the calibration board are recorded. At the same time, the physical coordinates corresponding to each feature point are obtained.

[0052] A mapping matrix from the pixel coordinates to the physical coordinates is established by affine transformation or polynomial fitting, and the mapping matrix is ​​stored for subsequent coordinate transformation.

[0053] The technical solution of this invention has one or more of the following beneficial technical effects:

[0054] (1) Based on the model training state attributes, conditional branch control is implemented. The deep auxiliary branch is completely skipped during the inference stage, with no additional computational overhead and memory occupation. The auxiliary module is completely removed by ONNX dead code elimination or physical pruning. The number of parameters and inference performance of the deployed model are completely consistent with those of the original model, achieving high-precision gain of lossless deployment.

[0055] (2) Offline use of pre-trained deep models to generate depth pseudo-labels in batches, eliminating the need for manual pixel-by-pixel annotation and significantly reducing the construction cost of depth supervision data in agricultural scenarios; through four-channel synchronous enhancement to ensure RGB and depth label space alignment, combined with a lightweight depth auxiliary head with fewer than 5000 parameters, geometric supervision is introduced at extremely low computing power cost, significantly improving the localization accuracy of axillary bud key points in complex occlusion scenarios.

[0056] (3) Abandoning the traditional inactivation and positioning scheme that relies on depth cameras and complex 3D calibration, the laser execution coordinates are directly converted based on the pre-calibrated pixel-physical coordinate mapping relationship; the system hardware cost and calibration and maintenance difficulty are significantly reduced, the physical positioning accuracy is high, and the engineering application needs of laser-based inactivation of tomato axillary buds are fully met.

[0057] Furthermore, other beneficial effects of the present invention will be mentioned in the specific embodiments. Attached Figure Description

[0058] Figure 1 This is a schematic diagram of the Tab-Depth joint network model structure according to an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of the Tab-Depth joint network model training and inference architecture according to an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the tomato axillary bud two-dimensional inactivation point positioning and laser execution system according to an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0062] To facilitate a clear description of the technical solutions in the embodiments of the present invention, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order.

[0063] This invention provides a method for identifying and locating two-dimensional inactivation of tomato axillary buds. The method uses the YOLOv11-Pose keypoint detection model (You Only Look Once v11-Pose, YOLOv11-Pose) as the baseline architecture. A lightweight depth auxiliary head (DepthHead) is connected in parallel to the end of its feature fusion network to construct a Tab-Depth joint network model (Tab refers to the tomato axillary bud detection task, and Depth refers to the depth-assisted supervision task). This method uses a pre-trained monocular depth estimation model (Depth Anything V2) offline to generate pixel-level normalized depth pseudo-labels, achieving low-cost batch preparation of depth supervision signals. During training, a four-channel synchronous data augmentation mechanism ensures complete consistency between the spatial correspondence between RGB images and depth labels. A weighted pixel-level depth loss function is combined to apply geometric constraints to the shared feature extraction network, guiding the network to learn the spatial structural features of the plant, effectively improving the detection accuracy of axillary bud keypoints in complex greenhouse scenarios. This method utilizes the training state attributes of model objects to achieve conditional branch control, completely skipping the forward computation of the DepthHead in evaluation and inference modes. Furthermore, during model export, dead code elimination via Open Neural Network Exchange (ONNX) or physical pruning can be used to completely remove the DepthHead, ensuring that the final deployed model is completely consistent with the original baseline model in terms of parameter count and inference speed. The difference in inference speed is minimal and meets the real-time requirements of agricultural edge devices, achieving near-zero additional inference overhead and accuracy gain. At the inactivation execution end, this method directly converts the pixel coordinates of the axillary bud key points from the detection output into two-dimensional execution coordinates of the laser device through a pre-calibrated pixel-physical coordinate mapping relationship. This achieves accurate axillary bud laser inactivation localization without the need for depth information or three-dimensional coordinate calculations, significantly reducing system hardware costs and deployment / maintenance complexity.

[0064] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0065] This embodiment provides a lightweight, deep-assisted supervised method for identifying tomato axillary buds and locating two-dimensional inactivation points. Using the YOLOv11-Pose keypoint detection model within the Ultralytics framework as a baseline, a lightweight, deep-assisted supervised keypoint detection model is constructed by paralleling a DepthHead through a bypass at the end of the feature fusion network. In one embodiment of this invention, this is referred to as the Tab-Depth joint model. This embodiment uses offline-generated deep pseudo-labels as supervisory signals. During the training phase, geometric constraints are applied to the shared feature extraction network to enhance its learning ability of spatial structural features. During the inference and deployment phases, the computational links of the DepthHead are completely removed through conditional branch control and model pruning mechanisms, achieving high-precision keypoint detection with zero additional inference overhead. Finally, based on the two-dimensional keypoint pixel coordinates and the pre-calibrated pixel-physical mapping relationship, a laser device is driven to accurately locate and execute the axillary bud inactivation points. The overall process can be divided into eight stages: offline depth label generation, data layer processing, configuration layer parsing, network layer construction, model training, inference detection, model export and two-dimensional inactivation point localization. Depth information only participates in the supervised calculation during the training stage and does not intervene in the inference and inactivation localization process, effectively improving the detection accuracy in complex greenhouse scenarios while ensuring lightweight deployment.

[0066] Figure 1 This is a schematic diagram of the Tab-Depth joint network model structure according to an embodiment of the present invention. Figure 1 As shown, the Tab-Depth joint network model is divided into three main task modules: Backbone Network, Feature Fusion Network, and Pose Estimation Head, as well as a deep auxiliary head connected in parallel to the Feature Fusion Network. The hierarchical connection relationship and function of each module are as follows.

[0067] The backbone network is based on the YOLOv11 standard feature extraction architecture. The input is a three-channel RGB image, and it consists of multiple stacked convolutional-batch-normalization-SiLU activation modules (CBS), a C3K2 feature extraction module, a spatial pyramid pooling fast layer (SPPF), and a C2PSA attention module. The CBS module is the basic convolutional operation unit, integrating convolution, batch normalization, and SiLU activation functions. The C3K2 module is a feature extraction module with a bottleneck structure; the C3K parameter configuration (C3K=False or C3K=True) controls whether the corresponding convolutional structure is enabled, used to complete feature extraction and channel transformation at different levels. The SPPF module is located at the end of the backbone network, fusing global features and expanding the network's receptive field through multi-scale pooling operations. The C2PSA module introduces an attention mechanism, weighting the channel and spatial dimensions of features to enhance the expression of effective features. The backbone network outputs multi-scale feature maps through multiple downsampling operations, which are then fed into subsequent feature fusion networks for further processing.

[0068] The feature fusion network adopts a Feature Pyramid Network-Path Aggregation Network (FPN-PAN) structure, including an upsampling module, a channel concatenation module, a CBS module, and a C3K2 module. This network improves the spatial size of low-resolution deep features through upsampling, and concatenates feature maps of different levels along the channel dimension using the concatenation module. Combined with a bidirectional feature fusion path from top to bottom and bottom to top, it ultimately constructs three fused feature maps of different resolutions: P3, P4, and P5. P3 is the shallowest feature layer with the highest resolution, emphasizing detailed texture information, while P5 is the deepest feature layer with the lowest resolution, emphasizing high-level semantic information. The three fused feature maps are fed into the corresponding scale detection heads, and the P3 feature map is also used as input to the DepthHead.

[0069] The pose estimation detection head sets up three detection branches for the three feature maps of P3, P4 and P5 respectively. Based on the input multi-scale fusion features, it completes target detection and key point regression calculation, and finally outputs the bounding box regression result, classification confidence, key point pixel coordinates and visibility parameters of the axillary bud target, realizing the pose estimation detection function of tomato axillary bud.

[0070] The DepthHead is connected in parallel to the P3 feature output of the feature fusion network. It adopts a lightweight structure design of first reducing dimensionality, then upsampling, and then refining. The overall structure includes a first 1×1 convolutional layer (Conv 1×1), an upsampling layer, a 3×3 convolutional layer (Conv 3×3), a second 1×1 convolutional layer, and a parametric rectified linear unit (PReLU) activation layer. The first 1×1 convolutional layer reduces the number of channels in the input P3 feature map from 256 to 64, corresponding to approximately 16.4k parameters. This layer performs channel compression before the upsampling operation. Compared with the design of upsampling followed by dimensionality reduction, it can significantly reduce the overall computational cost of subsequent upsampling and convolution operations. In one embodiment, it can reduce the floating-point operation cost in the upsampling stage by 75%.

[0071] The upsampling layer uses bilinear interpolation to enlarge the feature map resolution from 80×80 to 160×160. This module has no learnable parameters. The 3×3 convolutional layer aggregates local spatial context information on the upsampled high-resolution feature map, mapping the number of channels from 64 to 32, corresponding to approximately 18.5k parameters, to improve the edge detail accuracy of the depth prediction result. The second 1×1 convolutional layer compresses the number of feature channels to 1, corresponding to approximately 33 parameters, and outputs a single-channel depth prediction map. Finally, the depth prediction result is output through the PReLU activation layer. PReLU stands for Parametric Corrected Linear Unit, which allows negative output values ​​and can be adapted to the normalized numerical representation of relative depth.

[0072] Preferably, the bilinear upsampling layer uses bilinear interpolation to perform a 2x upsampling, maintaining smooth feature map edges. Preferably, in the n-scale model configuration of YOLOv11n-Pose, the total number of parameters of the depth auxiliary head is approximately 50,000, accounting for only 1.9% of the total parameters of the joint network model, having a minimal impact on the computational overhead during training. It is only enabled during training to generate depth prediction maps to construct the depth-assisted loss, applying geometric supervision to the shared feature network. During inference, the forward computation of this module is completely skipped via conditional branches.

[0073] In the offline preparation phase of this method, the first step is to generate ground truth depth labels in batches to address the high cost of depth annotation in agricultural scenarios. This embodiment uses a pre-trained Depth Anything V2 monocular depth estimation model as the teacher network. Offline inference is performed on all RGB images in the training set to generate corresponding relative depth maps as ground truth labels for depth supervision. Specifically, the pre-trained Depth Anything V2 model is loaded and switched to evaluation mode. All network parameters are frozen, and forward inference is performed on each RGB image of a tomato plant in the training set to generate an original depth map with the same spatial resolution as the input image. Preferably, the Depth Anything V2 model uses the ViT-Small backbone version, with approximately 24M parameters, offering fast inference speed and suitability for batch label generation on large-scale datasets.

[0074] The original depth map is then preprocessed. First, infinity values ​​and invalid pixel values ​​are removed. Then, min-max normalization is performed based on the minimum and maximum effective depth values ​​within a single image, mapping the depth values ​​to the [0,1] interval. Finally, the normalized depth map is stored in a single-channel image format, with a one-to-one mapping between the filename and the corresponding RGB image. Preferably, the normalized depth map is stored in a single-channel 16-bit PNG or 32-bit floating-point array format. A one-to-one mapping between the depth map filename and the corresponding RGB image can be established through a unified naming rule or a separate mapping configuration file, allowing the data loader to read the data in pairs by index.

[0075] The depth label generation process only needs to be executed offline once and does not participate in the real-time calculation of the training process. The generated depth map is only used as a supervision label for the depth-assisted loss during the training phase and does not participate in any calculation during the inference phase and the two-dimensional inactivation point localization phase.

[0076] During the data layer processing stage, this embodiment employs a four-channel synchronous enhancement strategy to ensure that the spatial correspondence between the RGB image and the depth map is completely consistent during the data enhancement process. The data loader synchronously reads the corresponding RGB image and depth map according to the image index, where the RGB image is a three-channel tensor and the depth map is a single-channel tensor. The two are concatenated along the channel dimension to form a four-channel tensor.

[0077] In the data augmentation stage, all data augmentation operations are performed simultaneously on the stitched four-channel tensor: Mosaic and MixUp enhancements are performed concurrently. Mosaic enhancement randomly selects four images and corresponding depth maps from the dataset, generating a composite image using a four-grid stitching method, ensuring complete alignment of the stitching boundaries between the RGB and depth channels. MixUp enhancement performs pixel-weighted fusion of the two images and their corresponding depth maps using random weights, with the fusion parameters applied synchronously to both the RGB and depth channels. Geometric transformation enhancements are also performed concurrently, including random horizontal flipping, random rotation, random scaling, and random cropping operations. All geometric transformations use uniform random parameters applied to the four-channel tensor, maintaining complete consistency in the spatial coordinate transformations of the RGB and depth channels.

[0078] Preferably, the execution probability of random horizontal flipping is set to 0.5, the random rotation angle range is set to [-10°, +10°], and the random scaling ratio range is set to [0.8, 1.2]. All geometric transformation parameters are uniformly generated and synchronously applied to the four-channel tensor. Preferably, the target clipping size after data augmentation is set to 640×640, and the RGB channel and depth channel use completely identical clipping windows.

[0079] After all data augmentation operations are completed, the four-channel tensor is re-separated along the channel dimension into a three-channel RGB image and a single-channel depth map. The RGB image is fed into the backbone network for feature extraction, while the depth map, as a supervision label, is passed through to the computing device along with the batch data via a custom batch assembly function (collate_fn). During the batch assembly stage, the batch assembly function stacks the single-channel depth ground truth maps of each sample into a batch depth tensor of dimension B×1×H×W (B is the batch size, and H and W are the height and width of the depth map, respectively). This tensor is stored in the batch data set with "depth" as the identifier and is then transferred to the graphics processor memory along with the RGB image tensor and annotation information through the training loop for direct use by the subsequent loss calculation module.

[0080] To achieve seamless compatibility between the deep-assisted supervision function and the native training framework, this method extends the design at the configuration layer. The standard YOLOv11-Pose YAML model configuration file is extended, adding a new group of deep-assisted header configuration fields, specifically including four parameters: feature enable switch, depth loss weight coefficient, depth loss function type, and depth map target resolution. Typical parameter configurations are: feature enable switch set to enabled, depth loss weight coefficient set to 0.1, depth loss function type set to L1 loss, and depth map target resolution set to 160×160.

[0081] When the function enable switch is set to off or the loss weight coefficient is set to 0, the depth auxiliary head and depth data loading functions are completely disabled, and the model degenerates into the standard YOLOv11-Pose, fully compatible with the training and inference chain of the native model, ensuring backward compatibility. Preferably, the recommended value range for the depth loss weight coefficient is 0.05~0.2, which can be flexibly adjusted according to the dataset annotation quality and task requirements. Simultaneously, the trainer module is extended to parse the depth map path field in the dataset annotation file during the data loader construction phase, establishing a one-to-one mapping relationship between RGB image paths and depth map paths.

[0082] During the training configuration parsing phase, the value range of the depth loss weight coefficients is read and verified, and the corresponding depth loss calculation module is initialized according to the configured loss function type. In the model instantiation phase, after the feature fusion network is built, the depth assistance function switch in the configuration file is read. If the function is enabled, the number of output channels of the P3 feature map is automatically inferred through virtual forward propagation. Based on this number of channels, DepthHead is instantiated and deployed to the same computing device (CPU / GPU) as the backbone network. Specifically, in the model initialization process, after the feature fusion network is built, a virtual forward propagation is performed through the channel number inference logic to automatically infer the number of channels of the P3 output features (256 for the n-scale model).

[0083] Figure 2 This is a schematic diagram of the Tab-Depth joint network model training and inference architecture according to an embodiment of the present invention. Figure 2 The diagram is divided into two parts: (a) the training phase (multi-task learning) architecture and (b) the inference phase (main task output only) architecture. This clearly demonstrates the core design idea of ​​this invention: introducing deep auxiliary supervision to enhance spatial feature learning during the training phase and removing auxiliary branches during the inference phase to achieve zero additional computational overhead.

[0084] In this embodiment, during the training phase shown in (a), offline inference is first performed on all RGB images in the training set using the pre-trained Depth Anything V2 monocular depth estimation model to generate corresponding relative depth maps, which serve as ground truth labels for depth supervision. This generation process only needs to be performed offline once and does not participate in the real-time calculation during the training process. During training, the backbone network, which shares RGB image inputs, completes multi-scale feature extraction. The extracted features are then fed into the feature fusion network, which uses an "FPN+PAN" structure to complete bidirectional fusion of multi-scale features. The output fused features are transmitted in two paths: one path is sent to the pose estimation detection head, which is the main task, to complete axillary bud target detection and keypoint pose estimation calculation, and outputs the keypoint detection results; the other path is sent to the depth auxiliary head, which is the auxiliary task, to generate a depth prediction map based on the fused features, and outputs the depth detection results.

[0085] The total loss during the training phase consists of the pose estimation loss L. pose With depth-assisted loss L depth The weighted summation is used to construct the pose estimation loss L. pose Includes bounding box regression loss L box Classification loss L cls Distribution focal loss L dfl Key point localization loss L kpt,loc With keypoint visibility loss L kpt,vis Its structure is consistent with the loss function of the native YOLOv11-Pose.

[0086] Depth-assisted loss can employ pixel-level loss functions such as L1 loss, BerHu loss, scale-invariant logarithmic loss (SI-log), or mean square error (MSE) loss. Before calculation, it is first determined whether the spatial resolution of the depth prediction map and the ground truth depth map are consistent. If there is a difference, bilinear interpolation is used to adaptively align the size of the depth prediction map to the resolution of the ground truth depth map. The alignment operation only applies to the depth prediction map to maintain the original accuracy of the ground truth depth map. Subsequently, the pixel-by-pixel error is calculated, and the average of the pixel-by-pixel errors is multiplied by a preset depth loss weight coefficient and added to the total loss function.

[0087] The formula for calculating L1 loss is as follows:

[0088] ;

[0089] The formula for calculating MSE loss is:

[0090]

[0091] In the formula, N is the total number of effective pixels, pred i Let gt be the depth prediction value of the i-th pixel. i Let be the true depth value of the i-th pixel.

[0092] Preferably, considering the characteristics of widespread shading by branches and leaves and local noise in depth pseudo-labels in greenhouse agriculture scenarios, the L1 loss function is preferred for depth loss, as it is more robust to outliers and can reduce the interference of shading area labeling noise on training. When the depth labels have been manually verified and the labeling quality is high, the mean squared error loss function can be selected for depth loss, as it has a stronger penalty for outliers and can improve the overall accuracy of depth prediction.

[0093] Combining the above loss components, the total loss can be expressed as:

[0094] ;

[0095] This can be simplified to: L total = L pose + λL depth ;

[0096] Where λ is the preset depth loss weight coefficient, L total For the total loss, L box For bounding box regression loss, L cls For classification loss, L dfl For the distribution focus loss, L kpt,loc To locate the loss at the key point, L kpt,vis For keypoint visibility loss, L pose For attitude estimation loss, L depth This is a deep auxiliary loss.

[0097] During backpropagation, the gradient of the total loss propagates back along the deep auxiliary head, the feature fusion network, and the backbone network, forcing the shared feature extraction network to learn the spatial geometric features of the plant, thereby effectively improving the accuracy of key point detection.

[0098] In the inference phase shown in (b), the model completely shields the forward propagation path of the depth auxiliary head through a training-inference conditional branch control mechanism, retaining only the computational link of the main task. This conditional branch is implemented by detecting the model's built-in training state attribute (self.training, Boolean type), which is embedded in the model's main forward propagation logic: when the training state attribute is true (training mode), after the backbone network and feature fusion network complete feature extraction, one path sends the fused features to the pose estimation detection head, and the other path reuses the P3 feature map to send it to the depth auxiliary head to generate a depth prediction map. Finally, it returns an output group consisting of the pose estimation result and the depth prediction map for multi-task loss calculation; when the training state attribute is false (evaluation mode), the forward propagation logic directly skips all calculations of the depth auxiliary head.

[0099] The input RGB image is sequentially processed by the backbone network and the feature fusion network to extract fused features. Only the pose estimation detection head outputs the detection results for the axillary bud target and the keypoint localization results, i.e., the keypoint prediction results, without performing any depth prediction-related calculations. This architecture design ensures that the model computation path and parameter count during the inference phase are completely consistent with the native keypoint detection model, with no additional computational nodes related to depth prediction. The difference in inference speed is within an acceptable range, improving detection accuracy while maintaining lightweight deployment.

[0100] After training, this method provides two model export paths to ensure that the deployed model is completely consistent with the native YOLOv11-Pose. The first path is the ONNX dead code elimination path, which is the preferred export method of this invention. In specific implementation, the joint model is first switched to evaluation mode. At this time, the training state attribute is false, and the forward propagation branch of DepthHead is inactive. When performing the ONNX export operation, the symbol tracking mechanism only tracks the forward propagation path activated in evaluation mode. The computation nodes of DepthHead are judged as dead code because they are never executed and will not be included in the exported ONNX computation graph. The final exported ONNX model is completely consistent with the ONNX model exported from the native YOLOv11-Pose in terms of the number of nodes and parameters, and there is no significant difference in inference speed. Preferably, when performing the ONNX export operation, the operator set version number is set to 12 and constant folding optimization is enabled to further improve the inference efficiency of the exported model.

[0101] The second approach is physical pruning, which is suitable for scenarios where PyTorch models are deployed directly. In practice, it first performs a module-level deletion operation, removing the depth auxiliary head sub-module from the model object. Then, it iterates through the model's weight parameter set, deleting all weight parameters whose names contain the depth auxiliary head identifier, and retaining only the parameter entries corresponding to the backbone network, feature fusion network, and pose estimation detection head. If it is necessary to retain the optimizer state to support resuming training, it synchronously iterates through the optimizer state dictionary, deleting the momentum and variance states corresponding to the DepthHead parameter.

[0102] If an Exponential Moving Average (EMA) strategy was used during training, the relevant weights of DepthHead were simultaneously removed from the EMA model object and its weight parameter set. After all cleanup operations were completed, the pruned checkpoint file was saved, and the pruned weights were loaded into the native YOLOv11-Pose model in strict loading mode for compatibility verification, ensuring that there were no missing or unexpected keys. The size of the pruned model file was very small compared to the native YOLOv11-Pose pre-trained weights, which could be less than 5% in one type of implementation, and the inference parameter count was consistent with the original version.

[0103] Figure 3 This is a schematic diagram of the tomato axillary bud two-dimensional inactivation point positioning and laser execution system according to an embodiment of the present invention. Figure 3 The diagram illustrates the complete workflow from image acquisition, axillary bud recognition, coordinate transformation to laser inactivation.

[0104] The laser emission module integrates a camera and a laser, both rigidly fixed to maintain a constant relative pose, providing a stable hardware reference for mapping pixel coordinates to physical execution coordinates. This module can be fixedly mounted on a greenhouse support frame or integrated into a mobile robot platform. The fixed solution uses a 2D galvanometer to adjust the laser spot position for inactivation, while the mobile solution uses a navigation system to drive the robot to the target workstation before performing laser irradiation, adapting to greenhouse operation scenarios of different sizes. During operation, the camera faces the tomato plants inside the greenhouse to acquire images, obtaining RGB images of the plants and transmitting them to the AI ​​processing module.

[0105] After receiving the image, the AI ​​processing module runs the deployed Tab-Depth model to perform image analysis. It uses a keypoint detection network to identify the target of the tomato axillary bud and locate its central keypoint, outputting the pixel coordinates of the axillary bud's central keypoint. This completes the identification and output of the axillary bud and keypoints, and verifies the keypoint visibility parameter. Only when the visibility flag is in a valid state (corresponding to a flag value of 1) is it determined to be a valid keypoint, and subsequent localization operations are performed. Occluded targets with invalid visibility are skipped directly to prevent localization errors caused by occlusion and avoid mistakenly deactivating non-target tissues such as the main stem and fruit. Preferably, in the keypoint set output by the attitude estimation detection head, a predefined index 0 corresponds to the axillary bud's central keypoint, which is directly used for coordinate transformation of subsequent deactivation points.

[0106] After obtaining the valid key point pixel coordinates, the system calls the pre-calibrated pixel-physical coordinate mapping relationship to convert the two-dimensional pixel coordinates in the image coordinate system into the two-dimensional spatial physical coordinates corresponding to the laser emission module. This mapping process is only based on planar calibration and does not require depth information or three-dimensional coordinate calculation.

[0107] The pixel-physical coordinate mapping relationship is established through pre-calibration. Specifically, after fixing the relative positions of the camera and the laser inactivation device, a calibration board is placed on the plant working plane. Images of the calibration board are acquired and the pixel coordinates (u, v) of each feature point are recorded. Simultaneously, the physical coordinates (x, y) corresponding to each feature point are obtained through measurement. A mapping function from pixel coordinates to physical coordinates is established through affine transformation or polynomial fitting. The matrix expression for the affine transformation is as follows:

[0108] ;

[0109] In the formula, M is a 3×3 affine transformation matrix, which is obtained by fitting multiple sets of feature point coordinate pairs using the least squares method; (u, v) are the pixel coordinates of each feature point; (x, y) are the physical coordinates of each feature point; and T is the transpose marker.

[0110] The obtained mapping matrix is ​​stored as a configuration file for runtime use. It can be periodically recalibrated according to the equipment's operating cycle to compensate for positional shifts caused by temperature and humidity changes and mechanical vibrations. Preferably, the calibration cycle is set to once a week or once a month to compensate for minor equipment displacements caused by temperature and humidity changes and mechanical vibrations, ensuring long-term stable positioning accuracy.

[0111] The converted spatial coordinates are transmitted to the control motherboard, which generates corresponding laser drive control commands. Based on the received spatial coordinates, the control motherboard outputs control signals to the laser emitting module, driving the two-dimensional galvanometer of the laser emitting module to adjust its deflection angle, where Δθ... x Let Δθ be the deflection angle of the galvanometer in the x-axis direction. y The y-axis deflection angle of the galvanometer is used to precisely position the laser spot at the physical location corresponding to the axillary bud through dual-axis coordinated deflection. After positioning, the laser outputs a near-infrared laser beam with preset parameters.

[0112] Preferably, the laser wavelength is selected in the near-infrared band of 808nm or 980nm, the power density is set to 2~5W / cm² according to the characteristics of the axillary bud tissue, and the laser irradiation duration is set to 100~300ms according to the diameter of the axillary bud, so as to achieve targeted inactivation of the axillary bud tissue through thermal coagulation. In a typical embodiment of the present invention, an 808nm near-infrared laser is used, the power density is set to 3W / cm², and the laser irradiation duration is set to 200ms, which can complete the thermal coagulation inactivation of the axillary bud without damaging the surrounding tissue. The entire inactivation process only relies on two-dimensional physical execution coordinates and does not require depth information. The system structure is simple and easy to calibrate, effectively reducing hardware costs and maintenance difficulty.

[0113] To verify the effectiveness of the present invention, a comparative experiment was conducted on a self-built tomato axillary bud dataset. This dataset contains images of tomato plants under different light, angle, and shading conditions in a greenhouse environment. The annotations include three types of key points: axillary bud bounding boxes, center points, growth points, and basal connection points, as well as corresponding depth maps generated offline and normalized using Depth Anything V2.

[0114] The experiment used the standard YOLOv11n-Pose as the baseline model. The keypoint shape parameter kpt_shape (used to define the number of keypoints and the parameter dimension of each keypoint) was set to 3 keypoints, and each keypoint corresponded to 3-dimensional parameters. The depth loss weight λ of the Tab-Depth model of this invention was set to 0.1, and the depth loss type adopted was L1 loss. The depth labels were generated offline by Depth Anything V2-Small and normalized to the [0,1] interval. Data augmentation adopted a four-channel synchronous mosaic enhancement, random flipping and random scaling strategy. After the enhancement was completed, the RGB image and the depth map were separated. The model input resolution was uniformly 640×640. This experiment was conducted using the PyTorch 2.9 framework, with a single RTX 5080 GPU for training. The batch size was set to 32, and the training epochs were 250. Precision, recall, mAP@0.5, mAP@0.5:0.95, average F1-Score, number of parameters, and inference speed (Frames Per Second, FPS) were selected as evaluation metrics. The experimental results are shown in Table 1.

[0115] Table 1: Experimental Results

[0116]

[0117] As shown in Table 1, on the tomato axillary bud dataset, compared to the YOLOv11n-Pose baseline, the precision of this invention improved from 87.3% to 95.3% (+8.0 percentage points), mAP@0.5 improved from 91.6% to 95.5% (+3.9 percentage points), mAP@0.5:0.95 improved from 65.8% to 70.6% (+4.8 percentage points), and the average F1-Score improved from 88.58% to 94.54% (+5.96 percentage points). The recall rate remained at a high level of 93.8%, indicating that deep assisted supervision improved precision without causing a significant increase in missed detections. Tests on the inactivation point dataset showed that the mAP@0.5 of this invention reached 97.3%, mAP@0.5:0.95 reached 97.2%, and the average F1-Score reached 95.69%, verifying that deep assisted supervision can significantly improve localization accuracy.

[0118] The model in this invention and the baseline model both have 2.65M parameters. This indicates that the number of parameters in DepthHead during the training phase (approximately 50k) is extremely low (1.9%), and is skipped in model statistics due to the evaluation mode, thus not affecting the complexity metric of the deployed model. The inference speed decreased from 188.7 FPS to 178.6 FPS, a reduction of approximately 5.4%, which is perfectly acceptable within the real-time requirements of agricultural vision (typically >30 FPS).

[0119] Based on high-precision key point detection (mAP@0.5:0.95 reaching 70.6%) and pre-calibrated pixel-physical coordinate mapping, the two-dimensional inactivation positioning accuracy of this invention can reach the sub-pixel level (<2 pixels). In a typical greenhouse scenario (camera resolution 1920×1080, field of view coverage 1.2m×0.7m), the corresponding physical accuracy is about 1.2mm, which fully meets the accuracy requirements of laser focused inactivation.

[0120] To better illustrate the present invention, numerous specific details have been provided in the detailed embodiments described above. Those skilled in the art should understand that the present invention can be practiced even without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of the present invention.

[0121] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for identifying and two-dimensionally inactivating tomato axillary buds, characterized in that, include: Step S1: Obtain the training dataset, which includes RGB images, ground truth depth labels corresponding to the RGB images, and annotation information of the RGB images; Step S2: Construct a joint network model, which includes a backbone network, a feature fusion network, a pose estimation detection head, and a depth auxiliary head connected in parallel to the feature fusion network; Step S3: Input the RGB image into the joint network model. After the backbone network and the feature fusion network extract the fused feature map, the key point prediction results are output through the pose estimation detection head and the depth prediction results are output through the depth auxiliary head. Step S4: Calculate the pose estimation loss based on the key point prediction results and the annotation information of the RGB image; calculate the depth auxiliary loss based on the depth prediction results and the depth ground truth labels corresponding to the RGB image; weight the pose estimation loss and the depth auxiliary loss to obtain the total loss; and update the parameters of the joint network model by backpropagation based on the total loss. Step S5: After training is completed, the joint network model is switched to inference mode. In inference mode, the forward computation of the depth auxiliary head is skipped, and the depth auxiliary head is removed when exporting the joint network model to obtain the deployment model. Step S6: The tomato plant image is inferred using the deployment model and the pixel coordinates of the key points of the axillary buds are output. Based on the pre-calibrated mapping relationship, the coordinates are converted into two-dimensional execution coordinates of the laser device, and the laser device is driven to complete the axillary bud inactivation.

2. The tomato axillary bud identification and two-dimensional inactivation localization method according to claim 1, characterized in that, Prior to step S3, a data layer processing step is also included: The RGB image and the depth ground truth label are concatenated along the channel dimension to form a four-channel tensor; Data augmentation operations are performed synchronously on the four-channel tensor. The data augmentation operations include at least one of mosaic enhancement, MixUp blending enhancement, random horizontal flipping, random rotation, random scaling, and random cropping. The data augmentation operations apply the same geometric transformation parameters to the channels of the RGB image and the channels of the depth ground truth label. After data augmentation is completed, the four-channel tensor is separated into a three-channel RGB image and a single-channel depth ground truth label, which are then fed into the joint network model and the loss calculation module, respectively.

3. The tomato axillary bud identification and two-dimensional inactivation localization method according to claim 1, characterized in that, In step S1, the depth truth label is generated in the following manner: Load the pre-trained monocular depth estimation model, input all the RGB images into the pre-trained monocular depth estimation model for offline inference, and generate the corresponding original depth map; The original depth map is preprocessed, invalid pixel values ​​are removed, and then normalization is performed, that is, the depth values ​​are mapped to the [0, 1] interval; The normalized depth map is used as the ground truth label for the depth, and a one-to-one mapping relationship is established with the corresponding RGB image and stored.

4. The tomato axillary bud identification and two-dimensional inactivation localization method according to claim 1, characterized in that, The structure of the depth assist head includes, in sequence: The first 1×1 convolutional layer is used to perform channel dimensionality reduction on the input fused feature map; Bilinear upsampling layer is used to increase the spatial resolution of the dimensionality-reduced feature map by a factor of 2; A 3×3 convolutional layer is used to aggregate spatial context information on the upsampled feature map to further compress the number of channels; The second 1×1 convolutional layer is used to compress the number of channels to 1 and output a single-channel depth prediction map; A parameterized modified linear unit activation layer is used to output prediction results adapted to the normalized relative depth; where, The total number of parameters of the depth assist head is less than 50,000.

5. The tomato axillary bud identification and two-dimensional inactivation localization method according to claim 1, characterized in that, The total loss is specifically expressed as follows: THE total = L pose + λL depth ; Among them, L total For the total loss, L pose The loss is for attitude estimation, λ is a preset weighting coefficient for depth loss, and L is the depth loss value. depth For deep auxiliary loss; The depth-assisted loss can be any one of L1 loss, inverse Huber loss, scale-invariant logarithmic loss, or mean squared error loss. The pose estimation loss includes bounding box regression loss, classification loss, distribution focus loss, keypoint localization loss, and / or keypoint visibility loss.

6. The tomato axillary bud identification and two-dimensional inactivation localization method according to claim 5, characterized in that: When calculating the depth-assisted loss, if the spatial resolution of the depth prediction map is inconsistent with that of the depth ground truth label, bilinear interpolation is used to adaptively align the depth prediction map to the resolution of the depth ground truth label, and the alignment operation is only applied to the depth prediction map. Then, the pixel-by-pixel error is calculated, the mean is taken, and then multiplied by the depth loss weight coefficient to obtain the depth-assisted loss.

7. The tomato axillary bud identification and two-dimensional inactivation localization method according to claim 1, characterized in that, Skipping the forward computation of the depth assist head in the inference mode specifically includes: Conditional branch control is achieved by utilizing the training state attributes of the joint network model; When the joint network model is in training mode, the forward computation of the depth auxiliary head is performed and the depth prediction result is output. When the joint network model is in the inference mode, the forward computation of the depth auxiliary head is skipped directly, and the key point prediction results are output only by the pose estimation detection head.

8. The tomato axillary bud identification and two-dimensional inactivation localization method according to claim 1, characterized in that: The depth auxiliary head is removed using an open neural network exchange format dead code elimination mechanism, specifically including: Switch the trained joint network model to evaluation mode, and deactivate the forward propagation branch of the deep auxiliary head. The Open Neural Network Exchange Format export operation is performed, and the symbolic tracking mechanism is used to track only the forward propagation path activated in the evaluation mode. The computation nodes of the deep auxiliary head are judged as dead code because they are never executed and are not included in the exported Open Neural Network Exchange Format computation graph.

9. The tomato axillary bud identification and two-dimensional inactivation localization method according to claim 1, characterized in that, The depth assist head is removed by physical pruning, specifically including: Remove the depth auxiliary head from the object in the joint network model; Traverse the model weight parameter set and delete all weight parameter entries associated with the depth assist head; If the optimizer state is retained, the momentum and variance states corresponding to the depth auxiliary head parameters in the optimizer state are deleted simultaneously. If an exponential moving average strategy is used, the relevant weights of the depth auxiliary head are removed from the exponential moving average model object simultaneously; The pruned weight parameters are loaded into the native keypoint detection model in strict loading mode for compatibility verification to ensure that there are no missing keys or unexpected keys, thus obtaining the deployment model.

10. The tomato axillary bud identification and two-dimensional inactivation localization method according to claim 1, characterized in that, The conversion of the pre-calibrated mapping relationship into two-dimensional execution coordinates for the laser device specifically includes: With the relative positions of the camera and the laser device fixed, images of the calibration board are acquired and the pixel coordinates of the feature points of the calibration board are recorded. At the same time, the physical coordinates corresponding to each feature point are obtained. A mapping matrix from the pixel coordinates to the physical coordinates is established by affine transformation or polynomial fitting, and the mapping matrix is ​​stored for subsequent coordinate transformation.

Images