Tobacco leaf picking point positioning method based on ViTPose key point detection

Abstract

The invention discloses a tobacco leaf picking point positioning method based on ViTPose key point detection, and relates to the field of tobacco leaf picking, the method uses a deep learning model ViTPose model to carry out end-to-end key point positioning, combines geometric space information to carry out key point detection, estimates the spatial attitude of tobacco leaves, and obtains a tobacco leaf picking point by collecting tobacco plant image data of different scenes. An RGB image is converted into a feature tensor containing global context information by using an encoder module in the ViTPose model, a decoder module in the ViTPose model receives the feature tensor, through up-sampling and feature conversion operation, a multi-channel key point thermodynamic diagram is output, key point coordinates are obtained through Gaussian peak regression and non-maximum suppression, and a multi-channel key point thermodynamic diagram is obtained. And fitting a tobacco leaf main direction vector according to the key point coordinates. Based on the ViTPose key point detection network, the spatial position is calculated in combination with the depth map, the picking point coordinates of the tobacco leaves can be accurately recognized in a complex environment, the dual requirements for precision and real-time performance in industrial production are met, and automatic picking of the tobacco leaves is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of tobacco leaf picking, and in particular to a tobacco leaf picking point positioning method based on ViTPose key point detection. BACKGROUND

[0002] Three-dimensional positioning of picking points is a key link in tobacco leaf picking, directly determining the accuracy of picking tasks. Accurate and stable determination of the picking position of tobacco leaves can not only improve the work efficiency of robots and reduce economic losses caused by mispicking or damage, but also provides key three-dimensional coordinates for subsequent path planning and mechanical arm movement, which is of great significance to the automation of picking operations. However, current picking point positioning technology still has many challenges. On the one hand, traditional image processing-based positioning methods usually rely on hand-designed feature extraction and rule-based judgment, which are simple to implement and fast to calculate, but in complex field environments, the positioning accuracy and robustness are significantly reduced in the face of tobacco leaf occlusion, shape changes and light interference. On the other hand, the end-to-end key point detection method based on deep learning that has emerged in recent years has strong feature adaptive ability and can maintain high positioning accuracy in complex scenes, but this method has high requirements for large-scale labeled data and computing resources, and also has the problem of slow inference speed. In practical applications, how to strike a reasonable balance between precision, real-time performance and computing overhead is still a technical problem to be solved.

[0003] Therefore, the skilled in the art is committed to developing a tobacco leaf picking point positioning method based on ViTPose key point detection. SUMMARY

[0004] In view of the above defects of the prior art, the technical problem to be solved by the present application is that the positioning method based on image processing cannot effectively cope with tobacco leaf occlusion, shape changes and light interference in complex field environments, and has low positioning accuracy.

[0005] To achieve the above-mentioned purpose, the present application provides a tobacco leaf picking point positioning method based on ViTPose key point detection, which uses a deep learning model ViTPose model for end-to-end key point positioning and combines geometric spatial information for key point detection to estimate the spatial pose of the tobacco leaf. The method comprises the following steps:

[0006] S101: Collect tobacco plant image data of different scenes and mark the tobacco leaves and main stems in the tobacco plant image data, the tobacco plant image data including RGB image data and depth image data and covering different light conditions, different tobacco plant angles and different tobacco leaf and main stem occlusion conditions;

[0007] S103: Preprocess the tobacco plant image data, the preprocessing including completing the depth image, registering the RGB image and the depth image, and data augmentation;

[0008] S105: Using the encoder module in the ViTPose model, the RGB image is transformed into a feature tensor containing global context information, and the feature tensor is used to interact with the decoder module in the ViTPose model.

[0009] S107: In the ViTPose model, the decoder module receives the feature tensor and outputs a multi-channel key point heat map through upsampling and feature transformation operations. The heat map includes tobacco leaf picking points and auxiliary positioning points.

[0010] S109: Obtain the coordinates of key points through Gaussian peak regression and non-maximum suppression, and fit the main direction vector of tobacco leaves based on the coordinates of the key points.

[0011] Further, in step S103, a depth completion algorithm is used to fill in the missing regions in the depth image data. The depth completion algorithm includes depth image missing region detection, depth estimation model application, and optimization of the completion effect.

[0012] The missing region detection in the depth image uses a validity threshold to detect missing regions. The missing region detection method is as follows:

[0013]

[0014] The depth estimation model is applied to fill in the missing regions.

[0015] The optimized completion effect is achieved by smoothing the completed image using Gaussian filtering or bilateral filtering. The filtering operation employs the following methods:

[0016]

[0017] in, This represents a region where the depth image is missing. For depth images, The spatial coordinates of the image, The minimum depth value, The depth image after smoothing. For pixels The neighborhood, Gaussian weighting factor For the completed depth image, This is the normalization constant.

[0018] Further, in the step S103, when the RGB image and the depth image are registered, the following steps are included:

[0019] S1031: Obtain the internal and external parameter matrices of the RGB camera and the depth camera through camera calibration;

[0020] S1032: Calculate the coordinate position of each pixel in the RGB image in the 3D space through geometric transformation:

[0021]

[0022] S1033: Convert the three-dimensional point in the RGB camera coordinate system to the depth camera coordinate system:

[0023]

[0024] S1034: Align the RGB image and the depth image at the pixel level through an interpolation method;

[0025] wherein, is the pixel point coordinate in the RGB image, is the position under the 3D space coordinate, is the position under the depth camera coordinate, is the depth value, is the inverse matrix of the internal parameter matrix of the RGB camera, is the external parameter matrix of the RGB camera, is the external parameter matrix of the depth camera.

[0026] Further, in the step S103, the data augmentation includes rotating the tobacco plant image data, translating the tobacco plant image data, randomly scaling the tobacco plant image data, adding Gaussian noise to the tobacco plant image data, adjusting the brightness and contrast of the tobacco plant image data.

[0027] Further, the encoder module takes the Vision Transformer as the backbone, performs block division and feature mapping on the input image, and outputs a feature tensor containing global context information, the Vision Transformer includes a plurality of identical Transformer encoding layers stacked, each Transformer includes a multi-head self-attention mechanism, a feedforward network and layer normalization, and the processing procedure of each Transformer is as follows:

[0028]

[0029]

[0030] wherein, is the input sequence, is the output sequence, for capturing dependencies of elements within a sequence, for non-linear transformation of features, for layer normalization operation.

[0031] Further, the step S105 comprises the following sub-steps:

[0032] S1051: receiving an RGB image, adjusting the resolution of the RGB image, and performing a pixel value normalization operation to output a standardized image tensor:

[0033] S1052: converting the image tensor into a one-dimensional token sequence using a two-dimensional convolutional layer;

[0034] S1053: fusing position encoding in the one-dimensional token sequence through element-level addition;

[0035] S1054: performing global context modeling using the Vision Transformer, abstracting and fusing global information layer by layer, and capturing long-range dependencies between key points;

[0036] S1055: reorganizing the processed one-dimensional token sequence into a two-dimensional spatial feature map and outputting.

[0037] Further, the pixel value normalization operation adopts the following method:

[0038]

[0039] In the one-dimensional token sequence, position encoding is fused, which is processed using the following method:

[0040]

[0041] wherein, is the final output image tensor conforming to the standard normal distribution, is the image tensor after size unification and pixel value scaling, is the preset mean value for channel-wise normalization, is the standard deviation vector for channel-wise normalization, is the final output sequence, is the original token sequence, is the position encoding tensor.

[0042] Further, the decoder module adopts a classic lightweight decoder composed of a deconvolutional layer and a final prediction layer. The deconvolutional layer gradually up-samples the feature map, combines batch normalization and ReLU activation function, restores the feature map to the size of the original image, and generates a heat map of key points. The number of channels of the heat map corresponds to the number of key points.

[0043] Further, step S107 includes the following sub-steps:

[0044] S1071: Receive the feature tensor generated by the encoder, and perform initial channel dimensionality reduction and spatial upsampling on the feature tensor to generate an upsampled feature map;

[0045] S1072: Perform main upsampling on the upsampled feature map, improve the spatial resolution through bilinear interpolation, and then refine the features through convolutional layers, batch normalization and ReLU activation to output a high-resolution feature map.

[0046] S1073: Map the high-resolution feature map into a multi-channel prediction heatmap, with each channel corresponding to the spatial probability distribution of a key point to be detected.

[0047] Furthermore, a Gaussian heatmap is used as a monitoring signal to generate a ground truth heatmap for each labeled key point. The Gaussian heatmap is generated using a two-dimensional Gaussian function, which is:

[0048]

[0049] During model training, the mean squared error loss function is used to measure the difference between the predicted value and the true value. The mean squared error loss function is as follows:

[0050]

[0051] in, For the true heatmap on coordinates Pixel value at that location, It refers to the precise coordinates of key points. The standard deviation of the Gaussian kernel. This is the total loss value. This represents the number of samples in the batch. For a single sample index, For the visibility mask of key points, No. Predicted heatmaps for each sample For the first True value heatmap of each sample.

[0052] In a preferred embodiment of the present invention, compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. This invention introduces a key point detection method from deep learning. By using the ViTPose model for end-to-end key point localization, it can accurately locate the picking point, avoiding the manual feature design and rule judgment in traditional image processing methods. This significantly improves the localization accuracy in complex environments and exhibits good robustness under various conditions.

[0054] 2. This invention uses the ViTPose framework, which extracts global contextual information and multi-scale features from RGB images through a Transformer encoder, directly regresses key point heatmaps, effectively estimates the spatial pose of tobacco leaves, and has stronger feature adaptation capabilities.

[0055] The following will further explain the concept, specific structure, and technical effects of the present invention in conjunction with the accompanying drawings, so as to fully understand the purpose, features, and effects of the present invention. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the steps of a preferred embodiment of the tobacco leaf picking point positioning method of the present invention;

[0057] Figure 2 This is a schematic diagram of a preferred embodiment of the present invention, showing the process of a robotic arm grasping and executing tobacco leaf picking.

[0058] Figure 3 This is a schematic diagram of the ViTPose model composition according to a preferred embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram illustrating the prediction effect of the ViTPose model after training, according to a preferred embodiment of the present invention. Detailed Implementation

[0060] The following description, with reference to the accompanying drawings, illustrates several preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0061] In the accompanying drawings, components with the same structure are indicated by the same numerical designation, and components with similar structures or functions are indicated by similar numerical designations. The dimensions and thicknesses of each component shown in the drawings are arbitrary, and the present invention does not limit the dimensions and thicknesses of each component. To make the illustrations clearer, the thickness of some components has been appropriately exaggerated in the drawings.

[0062] like Figure 1 and Figure 2As shown, existing image processing-based positioning methods cannot effectively cope with tobacco leaf shading, morphological changes, and light interference in complex field environments, resulting in low positioning accuracy. This invention provides a picking point detection network based on the ViTPose model, which directly predicts the coordinates of the picking point in an end-to-end manner. After obtaining the coordinates of the picking point on the image, it calculates its spatial position by combining the depth map and transforms it to the camera coordinate system to achieve accurate positioning of the tobacco leaf picking point.

[0063] like Figure 1 As shown, the tobacco leaf picking point localization method based on ViTPose keypoint detection provided by this invention uses the ViTPose deep learning model for end-to-end keypoint localization and combines geometric spatial information for keypoint detection to estimate the spatial pose of the tobacco leaf. The method includes the following steps:

[0064] S101: Collect tobacco plant image data from different scenes and mark the tobacco leaves and main stems in the tobacco plant image data.

[0065] In this embodiment, the collected tobacco plant image data includes RGB image data and depth image data, and covers different lighting conditions, different tobacco plant angles, and different situations of tobacco leaves and main stem shading.

[0066] S103: Preprocess the tobacco plant image data.

[0067] In this embodiment, preprocessing includes completing the depth image, registering the RGB image and the depth image, and data augmentation.

[0068] 1) Complete the depth image

[0069] In this embodiment, a depth completion algorithm is used to fill in missing regions in the depth image data. The depth completion algorithm includes depth image missing region detection, depth estimation model application, and optimization of the completion effect.

[0070] Depth image missing region detection uses a validity threshold to detect missing regions. The missing region detection method is as follows:

[0071]

[0072] Application of depth estimation models: using depth estimation models to fill in missing regions;

[0073] To optimize the completion effect, Gaussian filtering or bilateral filtering is used to smooth the completed image. The filtering operation adopts the following method:

[0074]

[0075] in, This represents a region where the depth image is missing. For depth images, The spatial coordinates of the image, The minimum depth value, The depth image after smoothing. For pixels The neighborhood, Gaussian weighting factor For the completed depth image, This is the normalization constant.

[0076] 2) Registration of RGB and depth images

[0077] The registration process between RGB and depth images includes the following steps:

[0078] S1031: Obtain the intrinsic and extrinsic parameter matrices of the RGB camera and depth camera through camera calibration;

[0079] S1032: Calculate the coordinates of each pixel in the RGB image in 3D space through geometric transformation:

[0080]

[0081] S1033: Transform 3D points in the RGB camera coordinate system to the depth camera coordinate system.

[0082]

[0083] S1034: Perform pixel-level alignment between RGB and depth images using interpolation methods;

[0084] in, These are the pixel coordinates in the RGB image. The position in 3D spatial coordinates. The position in depth camera coordinates. For depth value, This is the inverse of the RGB camera intrinsic parameter matrix. This is the extrinsic parameter matrix of the RGB camera. This is the extrinsic parameter matrix of the depth camera.

[0085] 3) Data Augmentation

[0086] In this embodiment, data enhancement includes rotating, translating, randomly scaling, adding Gaussian noise to the image, and adjusting the image brightness and contrast.

[0087] S105: Using the encoder module in the ViTPose model, the RGB image is transformed into a feature tensor containing global context information, and the feature tensor interacts with the decoder module in the ViTPose model.

[0088] In this embodiment, the encoder module uses the Vision Transformer as its backbone to segment and feature map the input image, outputting a feature tensor containing global context information. The Vision Transformer consists of multiple identical Transformer encoding layers stacked together. Each Transformer layer includes a multi-head self-attention mechanism, a feedforward network, and layer normalization. The processing flow of each Transformer layer is as follows:

[0089]

[0090]

[0091] in, Given the input sequence, For the output sequence, Used to capture dependencies between elements within a sequence. Nonlinear transformations used for features This is a layer normalization operation.

[0092] In this embodiment, the encoder module completes the corresponding processing using the following sub-steps:

[0093] S1051: Receives an RGB image, adjusts its resolution, performs pixel value normalization, and outputs a normalized image tensor.

[0094] S1052: Use a two-dimensional convolutional layer to convert an image tensor into a one-dimensional token sequence;

[0095] S1053: Fusing positional encoding in a one-dimensional token sequence through element-level addition;

[0096] S1054: Use Vision Transformer to model the global context, abstract and fuse global information layer by layer, and capture long-range dependencies between key points;

[0097] S1055: Reorganize the processed one-dimensional token sequence into a two-dimensional spatial feature map and output it.

[0098] The encoder module uses the following method for pixel value normalization:

[0099]

[0100] Position encoding is incorporated into a one-dimensional token sequence using the following method:

[0101]

[0102] in, The final output image tensor conforms to a standard normal distribution. This is the image tensor after being sized and scaled by pixel values. This is a preset mean value used for channel-by-channel standardization. This is the standard deviation vector used for channel-by-channel standardization. For the final output sequence, This is the original token sequence. This is a position-encoded tensor.

[0103] S107: In the ViTPose model, the decoder module receives the feature tensor and outputs a multi-channel key point heatmap through upsampling and feature transformation operations. The heatmap includes tobacco picking points and auxiliary positioning points.

[0104] In this embodiment, the decoder module adopts a classic lightweight decoder, which consists of a deconvolution layer and a final prediction layer. The deconvolution layer progressively upsamples the feature map and combines batch normalization and ReLU activation function to restore the feature map to the size of the original image and generate a heatmap of key points. The number of channels in the heatmap corresponds to the number of key points.

[0105] The decoder module completes the relevant processing using the following sub-steps:

[0106] S1071: Receive the feature tensor generated by the encoder, and perform initial channel dimensionality reduction and spatial upsampling on the feature tensor to generate an upsampled feature map;

[0107] S1072: Perform main upsampling on the upsampled feature map, improve the spatial resolution through bilinear interpolation, and then refine the features through convolutional layers, batch normalization and ReLU activation to output a high-resolution feature map.

[0108] S1073: Maps high-resolution feature maps to multi-channel prediction heatmaps, with each channel corresponding to the spatial probability distribution of a key point to be detected.

[0109] In this embodiment, a Gaussian heatmap is used as the monitoring signal to generate a ground truth heatmap for each labeled key point. The Gaussian heatmap is generated using a two-dimensional Gaussian function, which is:

[0110]

[0111] During model training, the mean squared error loss function is used to measure the difference between the predicted value and the true value. The mean squared error loss function is:

[0112]

[0113] in, For the true heatmap on coordinates Pixel value at that location, It refers to the precise coordinates of key points. The standard deviation of the Gaussian kernel. This is the total loss value. This represents the number of samples in the batch. For a single sample index, For the visibility mask of key points, No. Predicted heatmaps for each sample For the first True value heatmap of each sample.

[0114] S109: Key point coordinates are obtained through Gaussian peak regression and non-maximum suppression, and the main direction vector of tobacco leaf is fitted based on the key point coordinates.

[0115] Compared with existing technologies, the present invention provides a method for locating tobacco picking points based on ViTPose key point detection, which has the following beneficial effects:

[0116] 1. To address the problem that traditional image processing-based localization methods cannot effectively cope with tobacco leaf shading, morphological changes, and light interference in complex field environments, resulting in low localization accuracy, this invention is based on key point detection methods in deep learning. It uses the ViTPose model for end-to-end key point localization, avoiding the manual feature design and rule judgment required in traditional image processing methods.

[0117] 2. This invention uses the ViTPose framework to extract global contextual information and multi-scale features from RGB images through a Transformer encoder, and directly regresses the key point heatmap. This gives it a stronger feature adaptation capability, thereby effectively estimating the spatial pose of tobacco leaves and significantly improving the positioning accuracy in complex environments (such as changes in lighting and tobacco leaf occlusion). The positioning error is only 7.23 pixels, far exceeding that of image processing algorithms (31.35 pixels). The system exhibits good robustness under various conditions.

[0118] 3. While existing deep learning methods offer high accuracy, they are computationally expensive and have slow inference speeds, making it difficult to balance accuracy, real-time performance, and computational overhead. This invention is based on ViTPose with a lightweight decoder, using Vision Transformer as the encoder (ViTPose backbone) and a lightweight decoder module. This optimizes computational overhead and improves inference speed while maintaining accuracy. The stacked L layers of the ViTransformer encoder effectively capture long-range dependencies and global information, while the decoder performs channel compression (768→256) and spatial upsampling through deconvolution layers in the early stages of decoding, significantly reducing the computational complexity and memory usage of subsequent layers. This balanced approach optimizes computational overhead and improves inference speed while maintaining high accuracy, meeting the dual requirements of accuracy and real-time performance in industrial production.

[0119] The present invention will now be described in detail with reference to preferred embodiments.

[0120] Using tobacco plant data collected in a tobacco field in China as a typical application scenario, the "Tobacco Leaf Picking Point Location Based on Vitpose Key Point Detection" proposed in this invention is deployed to improve the target positioning accuracy of the picking task.

[0121] (I) Dataset Scenario Description

[0122] In tobacco harvesting tasks, dataset construction is crucial. To ensure the training model's robustness in complex environments, we collected RGB-D data in multiple different scenarios. The diversity of the dataset includes varying lighting conditions, tobacco plant angles, and shading between tobacco leaves and the main stem. The following is a description of a specific scenario: In a greenhouse environment, lighting is relatively uniform, but due to variations in humidity and environmental conditions, there may be some reflection and texture errors. This environment helps us simulate harvesting scenarios under low-light or artificial lighting conditions.

[0123] Test Field: Under bright sunlight, the lighting conditions change drastically, and shadows and reflections may appear in the acquired images, increasing the challenge of the dataset. Under cloudy or low-light conditions, the acquired images are usually darker and may be affected by clouds and fog, testing the model's adaptability to low-light and high-noise environments.

[0124] All collected data were manually labeled, especially key areas such as tobacco leaves and main stems. The tobacco leaves and main stems in each image were precisely labeled to ensure the trained model could accurately identify and distinguish the targets.

[0125] (II) Overall System Technical Framework

[0126] This system consists of two core modules:

[0127] 1. Encoder module

[0128] ViTPose's encoder backbone is the Vision Transformer. The input image is first segmented and feature-mapped using the PatchEmbedding module. The input image size is 256 × 192, and the patch size is set to 16, resulting in 16 × 16 patches per image. Each image generates 192 patches, representing an input sequence length of 192. Each patch has a dimension of 16 × 16 × 3 = 768. The initial features output from Patch Embedding are then fed into a 12-layer Transformer encoder. Each Transformer layer consists of a multi-head self-attention mechanism, a feedforward network, and layer normalization.

[0129] 2. Decoder Module

[0130] The decoder module maps the features output by the encoder to a keypoint heatmap. This invention employs a classic lightweight decoder consisting of deconvolutional layers and a final prediction layer. Two deconvolutional layers progressively upsample the feature map, incorporating batch normalization and ReLU activation functions. After upsampling, the feature map resolution is restored from 16 × 12 to the original image size of 256 × 192. Subsequently, the feature map is input to a 1 × 1 convolutional layer to generate a keypoint heatmap. The number of channels in the output heatmap corresponds to the number of keypoints; in this paper, it is 3. Each channel of the heatmap represents the spatial distribution of a keypoint; a larger value indicates a higher probability of the keypoint's location.

[0131] The overall technical framework of the system is as follows Figure 2 and Figure 3 As shown. In Figure 2 In this process, the input is an RGB-D image, which is semantically segmented to obtain a main stem / leaf mask. Contour fitting is performed on the main stem region to extract two main stem edges, which are then extended outwards according to the main stem width to generate extension lines. The intersections of these extension lines with each leaf mask are calculated, and the longest effective line segment is obtained by vertical sorting. Based on operational safety and obstacle avoidance principles, points are preferentially selected on this line segment (prioritizing the upper third, then selecting the main point), and geometric constraints such as orientation angle, area, and length thresholds are set to eliminate abnormal candidates. Finally, the pixel coordinates are combined with depth values ​​and camera intrinsic parameters to project a 3D picking point for the robotic arm to grasp and execute. Figure 3In this architecture, a fixed-size RGB image is used as input. Patch Embedding is used to segment and embed the image into sequence features, which are then fed into a multi-layer Transformer encoder to model long-range dependencies and global deformations. The decoder uses lightweight deconvolution to progressively upsample to the original image scale, outputting a multi-channel keypoint heatmap (including picking points and auxiliary positioning points). Pixel-level coordinates are obtained through Gaussian peak regression and non-maximum suppression, and the principal direction vector of the leaf can be fitted. This structure is robust to occlusion, leaf shape variations, and complex backgrounds, and can be integrated with... Figure 1 The geometric rules work together to improve positioning accuracy and real-time performance.

[0132] (III) Detailed Explanation of System Implementation Steps

[0133] Step 1: Data Preprocessing

[0134] Data preprocessing is a crucial step in ensuring input data quality and system stability. In this system, data preprocessing mainly includes depth image completion, RGB image and depth image registration, and data augmentation. Through these steps, the system obtains accurate and consistent input data, thereby improving the accuracy and robustness of subsequent keypoint detection.

[0135] (1) Depth completion

[0136] During data acquisition, depth sensors may generate depth images with holes or missing regions due to changes in ambient lighting, surface reflections, and other factors. To ensure data consistency in subsequent processing, a depth completion algorithm is used to fill in these missing regions. The specific steps are as follows:

[0137] Depth Image Missing Region Detection

[0138] First, missing regions are detected using a validity threshold for the depth image. Assume the depth image is... ,in These are the spatial coordinates of the image. The pixels in the missing region. It may be less than a certain set minimum depth value (e.g., ).

[0139] The formula for missing region detection is as follows:

[0140]

[0141] This yields a binary image that marks the location of the missing region.

[0142] Application of depth estimation models

[0143] For missing regions, we use a depth estimation model (such as Depth Anything V2) to fill in these gaps. This model infers the depth value of the missing region from the depth information of the surrounding valid region. During the completion process, we leverage the inference capability of the depth estimation model to calculate the depth value of the missing region. .

[0144] Optimize completion effect

[0145] After depth completion, we smooth the completed image, for example, by applying Gaussian filtering or bilateral filtering, to preserve edge information and eliminate unnatural transitions in the completed region. Assume the completed depth image is... The filtering operation can be expressed by the following formula:

[0146]

[0147] in, It is a pixel. The neighborhood, It is a Gaussian weighting factor. It is a normalization constant.

[0148] (2) Registration of RGB images and depth images

[0149] Since RGB images and depth images are typically acquired by different sensors, their coordinate systems and viewpoints differ. Therefore, registration is necessary to ensure that each pixel in the RGB image accurately corresponds to its corresponding depth value. The registration process involves the following steps:

[0150] The intrinsic and extrinsic parameter matrices of the RGB and depth cameras are obtained through camera calibration. Assume the intrinsic parameter matrix of the RGB camera is... The intrinsic parameter matrix of the depth camera is Their extrinsic matrices are respectively and (This represents the transformation from the world coordinate system to the RGB camera coordinate system). The goal of calibration is to determine the transformation relationship between the depth map and the RGB image.

[0151] To align each pixel in the RGB image with its corresponding depth value in the depth image, we need to perform a geometric transformation. Assume the coordinates of a pixel in the RGB image are... The position of the pixel in 3D space Depth values ​​in the depth map The calculation is done using the following formula:

[0152]

[0153] in, It is the depth value of that pixel. It is the inverse of the intrinsic parameter matrix of the RGB camera.

[0154] Next, the 3D points in the RGB camera coordinate system will be transformed to the depth camera coordinate system. This will be done using the extrinsic parameter matrix. and Complete the transformation:

[0155]

[0156] Obtain the corresponding pixel coordinates in the depth image Then, the pixels of the RGB image can be mapped to the depth image.

[0157] Pixel-level alignment of RGB and depth images is performed using interpolation methods (such as bilinear interpolation) to ensure that each RGB pixel and its corresponding depth value correspond precisely in the same coordinate system.

[0158] (3) Data augmentation

[0159] To enhance the model's robustness and generalization ability, we augment the data to simulate variations in different acquisition scenarios. Data augmentation not only expands the dataset but also helps the model adapt to different lighting conditions, viewing angles, noise levels, and other factors.

[0160] Common enhancement methods include rotation, translation, scaling, adjusting image brightness and contrast, etc.

[0161] The image is rotated randomly, with the angle range set between -45° and +45°.

[0162]

[0163] The image is randomly translated horizontally or vertically, with the translation range limited to ±100 pixels.

[0164]

[0165] Randomly scale the image, setting the ratio between 0.5 and 1.5.

[0166]

[0167] Add Gaussian noise to the image to simulate sensor noise or environmental interference.

[0168]

[0169] in, This indicates that the mean is 0 and the variance is . Gaussian noise.

[0170] Adjust the brightness and contrast of the image to simulate different lighting conditions.

[0171]

[0172] in, It is the contrast ratio. This is the brightness adjustment value.

[0173] Step 2: Encoder Module

[0174] The overall function of this encoder module is to transform the input RGB image into a feature tensor containing global context information. This tensor serves as a unified interface for interaction with the decoder module. Its core task is to understand the content of a two-dimensional image and convert it into a serialized representation rich in global semantics, suitable for processing by the Transformer model.

[0175] (1) Image input and standardization

[0176] This step aims to receive and process the input image to establish a standardized foundation for subsequent processes. Specifically, the system receives an RGB image as input; in one specific embodiment, the resolution of this image is uniformly adjusted to [resolution not specified]. Subsequently, pixel value normalization is performed on this uniformly sized image, a process that can be represented by the following formula:

[0177]

[0178] in, This represents the final output image tensor that conforms to a standard normal distribution. This indicates that the dimensions have been uniformized and the pixel values ​​have been scaled (e.g., the range has been adjusted to). The resulting image tensor; and These represent the preset mean and standard deviation vectors used for channel-by-channel normalization, respectively. The process ultimately outputs a normalized four-dimensional image tensor. The key technical advantage of this standardization process lies in ensuring the numerical stability and regularity of subsequent processing.

[0179] (2) Patch Embedding

[0180] This step is responsible for transforming the standardized two-dimensional image tensor... The input tensor is transformed into a one-dimensional token sequence. This transformation is achieved through a two-dimensional convolutional layer that operates on the input tensor with a kernel size of 16 pixels and a stride of 16 pixels. In one embodiment, the number of output channels is set to 768. The result is further flattened to generate a token sequence. The sequence length 192 represents the total number of image patches. The technical advantage of this operation is that it not only efficiently compresses local pixel information into high-dimensional semantic vectors, but its dimensional design is also compatible with mainstream pre-trained visual models, laying the foundation for leveraging transfer learning results, accelerating model convergence, and improving performance.

[0181] (3) Fusion of Positional Encoding

[0182] To enable the model to perceive the spatial arrangement of each image patch in the original image, this step involves using the token sequence generated in the previous step. Inject location information. This fusion process is achieved through element-wise addition:

[0183]

[0184] in, This represents the final output sequence that incorporates location information; This represents the original token sequence from the image patch embedding module; This represents a position-encoded tensor, whose dimension is... This can be entirely identical, consisting of learnable parameters or fixed codes generated based on a specific function (such as the sine-cosine function). This step aims to compensate for the inherent sequence order invariance of the Transformer structure, providing crucial prior information for the model to understand the spatial structure of an image.

[0185] (4) Transformer encoder layer stacking

[0186] As the core processing unit of the model, this step utilizes a deep network consisting of 12 identical stacked Transformer encoding layers to model the global context. Each layer processes the input sequence... The processing involves core computations including Multi-Head Self-Attention (MHSA) and Feedforward Network (FFN), and is stabilized through residual connections and LayerNorm. The single-layer processing flow can be summarized as follows:

[0187]

[0188]

[0189] in, and These represent the input and output sequences of this layer, respectively. The module captures the dependencies between elements in a sequence by calculating the scaling dot product attention among the query, key, and value. It is a network consisting of two linear layers and a nonlinear activation function (such as GELU) for nonlinear transformation of features; This is a layer normalization operation. The technical advantage of this deep stacked structure lies in its ability to abstract and fuse global information layer by layer, effectively capturing long-range dependencies between key points.

[0190] (5) Encoder output interface formatting

[0191] This step aims to transform the one-dimensional token sequence processed by the encoder. The sequence is reorganized into a two-dimensional spatial feature map to serve as the standard interface for output to the decoder module. This is achieved by performing a dimensionality reshaping operation (e.g., transpose and view) on the sequence. from The shape is transformed into a feature tensor with spatial dimensions. This two-dimensional spatial format is naturally compatible with the convolution and upsampling operations commonly used in decoders, while fully preserving the high-level semantic information learned by the Transformer. The technical advantage of defining this standard interface lies in its promotion of modular decoupling between the encoder and decoder, facilitating distributed deployment schemes such as edge-cloud collaborative computing.

[0192] Step 3: Decoder Module

[0193] The overall function of this decoder module is to receive the low-resolution, high-dimensional feature tensor output by the encoder. Through a series of upsampling and feature transformation operations, the resolution is restored to be close to that of the original input image, and finally a pixel-level probability heatmap is generated for each key point to be detected.

[0194] (1) Feature reception and preliminary upsampling

[0195] The decoder first receives the feature tensor from the encoder. The algorithm then performs initial channel dimensionality reduction and spatial upsampling. In one embodiment, this is achieved through a deconvolutional layer (ConvTranspose2d), which doubles the spatial resolution while efficiently compressing the number of feature channels from 768 to 256. After this operation, batch normalization and the ReLU activation function are concatenated, ultimately outputting a vector array of size [size missing]. The upsampled feature map. The technical advantage of this design is that by performing channel compression at the early stage of decoding, the computational complexity and memory consumption of subsequent layers are significantly reduced, thereby improving the overall efficiency of the model.

[0196] (2) Main upsampling and feature refinement

[0197] To restore the spatial resolution of the feature map to the target output size, this step modifies the input... Feature maps undergo primary upsampling. In a robust implementation, bilinear interpolation is used to directly increase the spatial resolution by 8 times. Then through a The convolutional layers and subsequent batch normalization and ReLU activation refine the features. This process outputs a high-resolution feature map with a size of [size missing]. The technical advantage of this interpolation and convolution combination method is that, compared with single large-stride deconvolution, it has fewer parameters, a more stable training process, and is conducive to generating high-quality and detailed feature representations.

[0198] (3) Heatmap generation

[0199] This step maps the decoded high-resolution feature map into a K-channel prediction heatmap, where each channel corresponds to the spatial probability distribution of a keypoint to be detected. This mapping is achieved through a kernel size of... The convolutional layer is implemented, which linearly transforms 256 input channels into K output channels (K=3 in this embodiment) without adding any non-linear activation function. The final output is a predicted heatmap. Directly regress to the target value. Convolution serves as an efficient linear transformer, with the technical advantage of being able to convert deep features that fuse global and local information into response intensities at specific key points pixel by pixel.

[0200] (4) Generation and application of training objectives (true value heatmap)

[0201] To effectively supervise the network, a ground truth heatmap needs to be generated for each labeled keypoint. For any coordinate... The visible key points, and their corresponding truth heatmaps Generated using a two-dimensional Gaussian function:

[0202]

[0203] in, Is the truth heatmap in coordinates Pixel value at; These are the precise coordinates of the key points; It is the standard deviation of the Gaussian kernel, which controls the size of the peak region (e.g., 2-4 pixels).

[0204] During training, the mean squared error (MSE) loss function is used to measure the difference between the prediction and the true value:

[0205]

[0206] in, This represents the total loss value; This refers to the number of samples in the batch processing. Indexing a single sample; It is the visibility mask of the key point (1 for visible, 0 for invisible); and They are the first The predicted heatmap and ground truth heatmap for each sample. The technical advantage of using Gaussian heatmaps as a supervision signal is that it provides richer local spatial distribution information than single-point coordinates, making the model learning process more robust.

[0207] (5) Analysis from heatmap to coordinates: In the reasoning stage, this involves analyzing the predicted probability heatmap. To extract precise keypoint coordinates, the system first finds the pixel position with the largest response value in each keypoint channel, i.e., performs an argmax operation to obtain integer coordinates. To further improve positioning accuracy, methods such as soft-argmax can be used for sub-pixel level refinement, the calculation process of which can be expressed as:

[0208]

[0209] in, These are the calculated sub-pixel coordinates; It is a local neighborhood window centered on integer coordinates; These are the pixel coordinates within the window; It predicts the response value of the heat map at the corresponding coordinates; It is a temperature coefficient used to adjust the sharpness of the probability distribution. The technical advantage of this sub-pixel refinement step is that it can significantly reduce the quantization error caused by the discretization representation of the heatmap, and is a key technology for recovering high-precision physical coordinates from the probability distribution.

[0210] (iv) Key Indicators and Results

[0211] This experiment uses a tobacco leaf harvesting scenario as a background to evaluate the performance differences between image processing-based and keypoint detection-based harvesting point localization algorithms on tobacco plant data. The tobacco leaf image data required for the experiment was acquired using an IntelRealsense D435i depth camera. After processing the RGB-D data using the segmentation algorithm described in Chapter 2, masks for tobacco leaves and main stems were generated for image processing-based harvesting point localization. Simultaneously, for the keypoint detection model experiment, a tobacco leaf harvesting dataset was pre-constructed, and the 3D coordinates of three key points were labeled in the tobacco leaf images as the basis for model training and validation. After training, the keypoint detection model can predict the pixel coordinates of tobacco leaf harvesting points end-to-end, mapping them to the 3D camera coordinate system using depth information. Table 1 shows the localization errors of the two algorithms for harvesting points.

[0212] Table 1 Comparison of Picking Point Location Errors

[0213] Image processing algorithm error (pixels) Keypoint detection algorithm error (pixels) 31.35 7.23

[0214] Table 2 shows the average running time of the two algorithms. According to the data in the table, the image processing-based picking point localization algorithm has high computational efficiency and is suitable for scenarios with high real-time requirements. The keypoint detection algorithm, due to its involvement in deep learning model inference, has a longer running time, but offers higher localization accuracy, with a localization error of only 7.23 pixels.

[0215] Table 2 Comparison of Average Running Time

[0216] Algorithm Average run time (ms) Image processing localization algorithm 11.3 Keypoint detection algorithm 97.2

[0217] The effects of image processing-based angle estimation and keypoint detection model-based straight line fitting angle estimation were compared. For each test sample, the angle was estimated by both methods and compared with the manually labeled angle. The mean absolute error (MAE) was calculated to evaluate the accuracy, and the results are shown in Table 3.

[0218] Table 3. Mean Absolute Error (MAE) for Accuracy Assessment

[0219] Fusion law error (°) Keypoint fitting law error (°) 2.89 1.43

[0220] like Figure 4The diagram shows the visualization results of the loss and training accuracy during the training process of the method of this invention. In the training loss curve, the loss value decreased from the initial value of 2.0 to 0.2 after 20 iterations, and finally converged to 0.1 after 100 iterations. The training accuracy curve rapidly increased from the initial value to 0.9 after 15 iterations, and converged to 0.98 after 100 iterations. The curves show that the training set loss continuously decreased, while the validation set accuracy steadily increased and stabilized in the later stages, indicating that the network structure and optimization strategy were effective and there was no obvious overfitting. Data augmentation (scale / rotation / occlusion), L2 regularization, and cosine annealing of the learning rate further promoted convergence. Under the target input resolution and inference settings, this model can achieve a good balance between accuracy and real-time performance, meeting the requirements of online picking and sensing.

[0221] The tobacco leaf picking point localization method based on ViTPose keypoint detection proposed in this invention has significant practical value and industrialization prospects in the field of tobacco leaf picking, specifically reflected in the following aspects:

[0222] 1. Technological advantages

[0223] 1) Precise tobacco picking point location: By combining stereo vision technology with the ViTPose deep learning model, this solution can accurately identify the key points of tobacco leaves and main stems in complex field environments, ensuring high-precision completion of the picking task. This technical solution effectively compensates for the shortcomings of traditional image processing methods in environments with complex lighting and tobacco leaf shading, providing more stable and accurate positioning results.

[0224] 2) Multi-scene adaptability: This method can handle different lighting conditions (such as greenhouses, strong light, weak light, cloudy days, etc.) and changes in the angle of tobacco plants, exhibiting excellent environmental adaptability. Through field data collection in different environments, the robustness of the model is improved, providing strong technical support for practical applications.

[0225] 3) Balancing Real-Time Performance and Accuracy: By using a ViTPose-based keypoint detection network combined with depth map-based spatial location calculation, this invention can optimize computational overhead and improve inference speed while ensuring high positioning accuracy. This balanced approach meets the dual requirements of accuracy and real-time performance in industrial production, promoting the automation of tobacco harvesting.

[0226] 2. Performance advantages

[0227] 1) High-precision picking point positioning: Experimental data shows that the present invention can accurately identify the coordinates of the picking point of tobacco leaves in complex tobacco field environments. Especially under conditions of shading and changes in light, the model shows excellent positioning accuracy, far exceeding traditional image processing methods.

[0228] 2) Fast Response and Stability: The system's design optimizes inference speed while maintaining high stability. Under varying climatic conditions and tobacco growing environments, the model remains stable and outputs accurate harvesting point locations.

[0229] 3. Industrial application adaptability

[0230] 1) No additional hardware investment required: This method only relies on existing RGB-D camera equipment for data acquisition and uses algorithms for data processing and analysis. Therefore, it does not require large-scale hardware upgrades to existing production equipment, resulting in lower deployment costs and implementation difficulty.

[0231] 2) Adaptable to various tobacco picking scenarios: By applying this technology to tobacco fields in Qujing, Yunnan, its applicability in various environments has been demonstrated. Whether in greenhouses or open fields, the system can achieve efficient positioning of tobacco picking points.

[0232] 3) Scalability and cross-industry applications: Although this invention is mainly applied to the tobacco harvesting field, its core technology—the precise positioning method based on key point detection—can be extended to other agricultural production fields, such as fruit tree harvesting and crop monitoring, and can even be extended to multiple fields such as industrial automation and robotics.

[0233] 4. Industrialization Prospects

[0234] 1) Improve production efficiency: This invention can significantly improve the automation level of tobacco leaf picking, reduce manual operation, reduce labor intensity, improve picking efficiency and accuracy, and at the same time reduce economic losses caused by mispicking and damage.

[0235] 2) Advancement of intelligent agriculture: With the acceleration of agricultural automation and intelligentization, tobacco leaf harvesting solutions based on key point detection technology have broad application prospects. They can not only help farmers reduce production costs, but also be promoted and applied globally to improve the modernization level of the agricultural industry.

[0236] In summary, this invention provides an efficient, accurate, and low-cost tobacco picking point positioning solution by optimizing key point detection and localization algorithms and combining stereo vision and deep learning technologies. It has strong practicality and industrial application value.

[0237] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for locating tobacco leaf picking points based on ViTPose keypoint detection, characterized in that, The method uses the ViTPose deep learning model for end-to-end keypoint localization and combines geometric spatial information for keypoint detection to estimate the spatial pose of tobacco leaves. The method includes the following steps: S101: Collect tobacco plant image data in different scenarios and mark the tobacco leaves and main stems in the tobacco plant image data. The tobacco plant image data includes RGB image data and depth image data, and covers different lighting conditions, different tobacco plant angles, and different occlusion situations of tobacco leaves and main stems. S103: Preprocess the tobacco plant image data, the preprocessing including completing the depth image, registering the RGB image and the depth image, and data augmentation; S105: Using the encoder module in the ViTPose model, the RGB image is transformed into a feature tensor containing global context information, and the feature tensor is used to interact with the decoder module in the ViTPose model. S107: In the ViTPose model, the decoder module receives the feature tensor and outputs a multi-channel key point heat map through upsampling and feature transformation operations. The heat map includes tobacco leaf picking points and auxiliary positioning points. S109: Obtain the coordinates of key points through Gaussian peak regression and non-maximum suppression, and fit the main direction vector of tobacco leaves based on the coordinates of the key points.

2. The method as described in claim 1, characterized in that, In step S103, a depth completion algorithm is used to fill in the missing regions in the depth image data. The depth completion algorithm includes depth image missing region detection, depth estimation model application, and optimization of the completion effect. The missing region detection in the depth image uses a validity threshold to detect missing regions. The missing region detection method is as follows: ； The depth estimation model is applied to fill in the missing regions. The optimized completion effect is achieved by smoothing the completed image using Gaussian filtering or bilateral filtering. The filtering operation employs the following methods: in, This represents a region where the depth image is missing. For depth images, The spatial coordinates of the image, The minimum depth value, The depth image after smoothing. For pixels The neighborhood, Gaussian weighting factor For the completed depth image, This is the normalization constant.

3. The method as described in claim 2, characterized in that, In step S103, the registration of the RGB image and the depth image includes the following steps: S1031: Obtain the intrinsic and extrinsic parameter matrices of the RGB camera and depth camera through camera calibration; S1032: Calculate the coordinates of each pixel in the RGB image in 3D space through geometric transformation: ； S1033: Transform 3D points in the RGB camera coordinate system to the depth camera coordinate system. ； S1034: Perform pixel-level alignment between RGB and depth images using interpolation methods; in, These are the pixel coordinates in the RGB image. The position in 3D spatial coordinates. The position in depth camera coordinates. For depth value, This is the inverse of the RGB camera intrinsic parameter matrix. This is the extrinsic parameter matrix of the RGB camera. This is the extrinsic parameter matrix of the depth camera.

4. The method as described in claim 3, characterized in that, In step S103, the data enhancement includes rotating, translating, randomly scaling, adding Gaussian noise to the image, and adjusting the image brightness and contrast.

5. The method as described in claim 4, characterized in that, The encoder module uses the Vision Transformer as its backbone to segment and feature-map the input image, outputting a feature tensor containing global context information. The Vision Transformer consists of multiple identical Transformer encoding layers stacked together. Each Transformer layer includes a multi-head self-attention mechanism, a feedforward network, and layer normalization. The processing flow of each Transformer layer is as follows: ； in, Given the input sequence, For the output sequence, MHSA is used to capture the dependencies between elements within the sequence, FFN is used for non-linear transformation of features, and LayerNorm is the layer normalization operation.

6. The method as described in claim 5, characterized in that, Step S105 includes the following sub-steps: S1051: Receive an RGB image, adjust the resolution of the RGB image, perform pixel value normalization, and output a normalized image tensor. S1052: Convert the image tensor into a one-dimensional token sequence using a two-dimensional convolutional layer; S1053: Fuse positional encoding in the one-dimensional token sequence by element-level addition; S1054: Use the Vision Transformer to perform global context modeling, abstract and fuse global information layer by layer, and capture long-range dependencies between key points; S1055: Reorganize the processed one-dimensional token sequence into a two-dimensional spatial feature map and output it.

7. The method as described in claim 6, characterized in that, The pixel value normalization operation uses the following method: ； Position encoding is incorporated into the one-dimensional token sequence using the following method: ； in, The final output image tensor conforms to a standard normal distribution. For image tensors that have been standardized in size and scaled in pixel values, This is a preset mean value used for channel-by-channel standardization. This is the standard deviation vector used for channel-by-channel standardization. For the final output sequence, This is the original token sequence. For position-encoded tensors.

8. The method as described in claim 7, characterized in that, The decoder module adopts a classic lightweight decoder, which consists of a deconvolution layer and a final prediction layer. The deconvolution layer progressively upsamples the feature map and combines batch normalization and ReLU activation function to restore the feature map to the size of the original image and generate a heatmap of key points. The number of channels in the heatmap corresponds to the number of key points.

9. The method as described in claim 8, characterized in that, Step S107 includes the following sub-steps: S1071: Receive the feature tensor generated by the encoder, and perform initial channel dimensionality reduction and spatial upsampling on the feature tensor to generate an upsampled feature map; S1072: Perform main upsampling on the upsampled feature map, improve the spatial resolution through bilinear interpolation, and then refine the features through convolutional layers, batch normalization and ReLU activation to output a high-resolution feature map. S1073: Map the high-resolution feature map into a multi-channel prediction heatmap, with each channel corresponding to the spatial probability distribution of a key point to be detected.

10. The method as described in claim 9, characterized in that, A Gaussian heatmap is used as the monitoring signal to generate a ground truth heatmap for each labeled key point. The Gaussian heatmap is generated using a two-dimensional Gaussian function, which is: ； During model training, the mean squared error loss function is used to measure the difference between the predicted value and the true value. The mean squared error loss function is as follows: ； in, For the true heatmap in coordinates Pixel value at that location, It refers to the precise coordinates of key points. The standard deviation of the Gaussian kernel. This is the total loss value. This represents the number of samples in the batch. For a single sample index, For the visibility mask of key points, No. Predicted heatmaps for each sample For the first True value heatmap of a sample.

Images