A vision-based agricultural environment perception and robot-assisted picking method

Abstract

The present invention discloses a vision-based agricultural environment perception and robot-assisted picking method. The method adopts an end-to-end detection, tracking and classification framework, including five independent modules: object detection and tracking, feature extraction, activity recognition, target position estimation, and robot scheduler. It uses RGB cameras and two-dimensional lidar to obtain information, and processes data through a customized long-short-term memory network. Compared with other methods, the end-to-end architecture adopted in this method enables the system to comprehensively process the entire process from picker activity detection to robot-assisted operation. The multi-camera information fusion adopted can accurately estimate the position of the picker within different distance ranges, and the customized long-short-term memory network adopted can accurately predict the picker's behavioral activities.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, deep learning, and sensor fusion, and in particular to a vision-based agricultural environment perception and robot-assisted picking method. Background Art

[0002] Today, the global agricultural sector faces a series of severe challenges and changes. With continued population growth and rising living standards, demand for agricultural products, including fruits, continues to rise. However, the shortage of agricultural labor is becoming increasingly prominent, becoming a key factor restricting agricultural development. Traditional fruit picking relies primarily on manual labor, but manual harvesting has many limitations. For example, long, intensive work periods are not only inefficient but also pose a serious threat to workers' health, increasing medical costs and the risk of labor turnover.

[0003] To address labor shortages, agricultural robots and autonomous farming systems have developed rapidly in the past few years. However, to date, the complex and dynamic nature of most agricultural operations, including irregular terrain, variable crop growing environments, and the diverse locations and shapes of fruit, has made fully automated harvesting by robots a significant challenge. Despite continuous technological advancements, there is still a long way to go before robots can accurately identify fruit, avoid obstacles, and efficiently complete harvesting tasks in complex agricultural environments.

[0004] While human-robot collaboration has achieved some success in industrial settings, its application in agriculture has been relatively slow. Agricultural environments, such as orchards, are highly open and complex, presenting significant challenges for robot perception and decision-making. Robots must accurately understand and interpret human behavior patterns, extracting valuable insights to achieve effective collaboration. However, due to environmental interference and the diversity and uncertainty of human behavior, robots still face numerous challenges in processing environmental data and identifying human actions and intentions. Furthermore, ensuring stable and efficient communication between robots and humans, as well as effective interaction between sensors, remain key challenges.

[0005] Embedded sensor technology, as a core means of enabling robotic perception, plays a vital role in the development of agricultural robots. However, achieving the optimal cost-effective design of autonomous systems is challenging. Farmers must consider multiple factors in practical applications, including investment cost, system performance, and maintenance difficulty. While multi-sensory perception systems can provide comprehensive environmental information, they come with high equipment procurement and maintenance costs.

[0006] At the same time, the limited collection of existing datasets also poses certain constraints on the development of agricultural robots. Most existing datasets were recorded under relatively ideal environmental conditions and fail to reflect the complexities of real-world agricultural scenarios. In the real world of fruit picking, weather conditions are highly variable, light intensity varies across fields, and human behavior is highly flexible and random. These complex real-world factors are difficult to capture in existing datasets, making it difficult for robots to adapt to real-world agricultural operations when learning and training using these data.

[0007] Faced with these challenges, we need an innovative solution that fully integrates the strengths of humans and robots, improves the efficiency and safety of fruit picking operations, and promotes the sustainable development of agricultural production. This paper addresses this need by proposing a vision-based human-robot collaborative fruit picking method. By optimizing perception, decision-making, and collaboration mechanisms, this approach provides a new approach to addressing labor shortages and low picking efficiency in the agricultural sector. Summary of the Invention

[0008] The present invention designs an end-to-end vision-based detection, tracking and classification architecture to achieve human-machine collaboration in fruit picking operations. First, the end-to-end vision-based detection, tracking and classification architecture proposed in the present invention integrates five independent modules: object detection and tracking, feature extraction, activity recognition, target position estimation and robot scheduler. This integrated design method enables the system to comprehensively handle the entire process from picker activity detection to robot-assisted operation. Secondly, RGB cameras and two-dimensional lidar are used to obtain information, and the fusion of the two sensors can accurately estimate the picker's position within different distance ranges. Finally, a customized long short-term memory network is used to process continuous data and capture the temporal dependency of landmark sequences extracted from images, thereby accurately relying on the picker's activities and analyzing the picker's continuous movements and behavioral changes during the picking process. To achieve the above purpose, the following steps are taken:

[0009] Step 1: Obtain a video dataset of fruit picking by different pickers in different environments, including videos of pickers with their backs, sides, and faces facing the robot, as well as videos under different occlusion conditions. Use cropping technology to process the collected video dataset into two types of video datasets: picking and non-picking.

[0010] Step 2: Build an object detection and tracking module, using the RGB input from the RealSense camera, using the YOLOv5 model architecture and the DeepSORT algorithm to detect and track human pickers, generating class labels, bounding boxes, and tracking IDs.

[0011] Step 3: Build a feature extraction module, using the OpenCV framework and the YOLOv5 object detection model to obtain a series of image frames, their bounding boxes, and the tracking ID corresponding to each picker. Then use the sliding window technique to extract features and classify the picker activities.

[0012] Step 3.1: Extract features by convolving the input image with the convolution kernel using the convolution layer of the convolutional neural network. The three-dimensional convolution expression is as follows:

[0013]

[0014] where Q i,j,k is the kth channel value of the output feature map at position (i, j), K m,n,c is the c-th channel value of the convolution kernel at position (m,n), I i+m,j+n,c is the c-th channel value of the output image at position (i+m,j+n), M and N are the convolution kernel sizes, and C is the number of input image channels;

[0015] Step 3.2: Use the frame accumulator to build a data structure to store the image data, boundary coordinate information, and tracking ID for each frame;

[0016] Step 3.3: Use the sliding window technique to slide a fixed-size window across the time series to obtain local feature information. The window size is determined by statistical analysis of the data features. The statistical analysis of the data features is as follows:

[0017]

[0018] in is the variance of the feature within the window, x i is the eigenvalue, is the mean of the eigenvalues within the window, w is the window size, n w is the current frame index;

[0019] Step 4: We build an activity recognition module that uses the landmarks detected by the feature extraction module as input and uses a custom long short-term memory network to predict the activity associated with each picker in the image. We then use a sliding window technique to determine the optimal size of each sequential input frame to accurately classify the activity.

[0020] Step 4.1: Use a custom LSTM network to predict the activity of each picker in the image. The custom LSTM network consists of three LSTM layers with a unit size of 32 and a softmax at the end. The internal expression of the LSTM unit is as follows:

[0021] f t =σ(Wf ·[h t-1 ,x t ]+b f ),

[0022] where f t is the output of the forget gate at time t, σ is the sigmoid activation function, W f is the weight matrix of the forget gate, h t-1 is the hidden state of the previous moment, x t is the input at the current moment, that is, the landmark features detected by the feature extraction module, b f is the bias vector of the forget gate;

[0023] c t =tanh(W c ·[h t-1 ,x t ]+b c ),

[0024] where c t is the cell state at time t;

[0025] o t =σ(W o ·[h t-1 ,x t ]+b o ),

[0026] Among them t is the output of the output gate at time t, W o is the weight matrix of the output gate, b o is the bias vector of the output gate;

[0027] h t =o t ⊙tanh(c t ),

[0028] where h t is the hidden state at time t, ⊙ is the multiplication of each element;

[0029]

[0030] Among them, P i is the probability of predicting the i-th activity category, c is the total number of activity categories, y i is the i-th element of the fully connected layer output y, and k is the predicted activity category;

[0031] Step 5: By building a target estimation module, using the RGB-D camera and 2D lidar to provide distance information, projecting the RGB image bounding box in the activity recognition module onto the depth image, extracting the center point depth, comparing it with the lidar distance, and estimating the distance obtained by the picker, using the cvReprojectImageTo3D function to convert the 2D image coordinates into 3D coordinates, publishing the target position in the camera frame, and converting it to the map frame map through ROSTF;

[0032] Step 5.1: Project the center point coordinates (u, v) of the bounding box in the RGB image onto the depth image to obtain the depth value d of the point;

[0033] Step 5.2: Convert the 2D image coordinates to 3D coordinates. The corresponding 3D coordinates (X, Y, Z) are calculated using the following expression:

[0034]

[0035] Z=d,

[0036] Where K is the intrinsic parameter matrix of the camera, f x and f y is the focal length of the camera in the x and y directions, c x and c y are the coordinates of the image center;

[0037] Step 5.3: Perform ROSTF conversion to convert the camera frame into a map frame. The corresponding conversion expression is as follows:

[0038]

[0039] in is the transformation matrix from camera frame to map frame, (X, Y, Z) is the 3D point in the camera frame, (X, Y, Z, 1) is the corresponding homogeneous formula, (X map ,Y map ,Z map ) is a 3D point in the map frame, (X map ,Y map ,Z map ,1) is the corresponding homogeneous form, r ij is the rotational component, t x , t y , t z is the translation component.

[0040] Step 6: By building a robot scheduling module, we set up the functions of task scheduling, path planning, resource management, and collision avoidance, and then integrate it with other modules. Deep image processing is used to update task priorities and adjust paths. Sensor data fusion is used to enhance task allocation and navigation decisions. Detailed logs and reports are generated to maintain transparency and facilitate diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is an overall flow chart of an embodiment of the present invention; DETAILED DESCRIPTION

[0042] In order to more clearly illustrate the purpose, technical solutions and advantages of the present invention, the present invention will be described in detail below with the help of drawings and specific embodiments.

[0043] Figure 1 This is a flow chart of an embodiment. This embodiment provides a vision-based agricultural environment perception and robot-assisted harvesting method. The specific process includes: collecting data, detecting and tracking the picker through the construction of an object detection and tracking module, extracting features and classifying the picker's activities through the construction of a feature extraction module, predicting the picker's related activities through the construction of an activity recognition module, and estimating the picker's distance through the construction of a target estimation module. The construction of a robot scheduling module implements path planning, task allocation, resource management, and task scheduling.

[0044] A vision-based agricultural environment perception and robot-assisted harvesting method includes the following steps:

[0045] Step 1: Obtain a video dataset of fruit picking by different pickers in different environments, including videos of pickers with their backs, sides, and faces facing the robot, as well as videos under different occlusion conditions. Use cropping technology to process the collected video dataset into two types of video datasets: picking and non-picking.

[0046] Step 2: Build an object detection and tracking module, using the RGB input from the RealSense camera, using the YOLOv5 model architecture and the DeepSORT algorithm to detect and track human pickers, generating class labels, bounding boxes, and tracking IDs.

[0047] Step 3: Build a feature extraction module, using the OpenCV framework and the YOLOv5 object detection model to obtain a series of image frames, their bounding boxes, and the tracking ID corresponding to each picker. Then use the sliding window technique to extract features and classify the picker activities.

[0048] Step 3.1: Extract features by convolving the input image with the convolution kernel using the convolution layer of the convolutional neural network. The three-dimensional convolution expression is as follows:

[0049]

[0050] where Q i,j,k is the kth channel value of the output feature map at position (i, j), K m,n,c is the c-th channel value of the convolution kernel at position (m,n), I i+m,j+n,c is the c-th channel value of the output image at position (i+m,j+n), M and N are the convolution kernel sizes, and C is the number of input image channels;

[0051] Step 3.2: Use the frame accumulator to build a data structure to store the image data, boundary coordinate information, and tracking ID for each frame;

[0052] Step 3.3: Use the sliding window technique to slide a fixed-size window across the time series to obtain local feature information. The window size is determined by statistical analysis of the data features. The statistical analysis of the data features is as follows:

[0053]

[0054] in is the variance of the feature within the window, x i is the eigenvalue, is the mean of the eigenvalues within the window, w is the window size, n w is the current frame index;

[0055] Step 4: We build an activity recognition module that uses the landmarks detected by the feature extraction module as input and uses a custom long short-term memory network to predict the activity associated with each picker in the image. We then use a sliding window technique to determine the optimal size of each sequential input frame to accurately classify the activity.

[0056] Step 4.1: Use a custom LSTM network to predict the activity of each picker in the image. The custom LSTM network consists of three LSTM layers with a unit size of 32 and a softmax at the end. The internal expression of the LSTM unit is as follows:

[0057] f t =σ(W f ·[h t-1 ,x t ]+b f ),

[0058] where f t is the output of the forget gate at time t, σ is the sigmoid activation function, W f is the weight matrix of the forget gate, h t-1 is the hidden state of the previous moment, x tis the input at the current moment, that is, the landmark features detected by the feature extraction module, b f is the bias vector of the forget gate;

[0059] c t =tanh(W c ·[h t-1 ,x t ]+b c ),

[0060] where c t is the cell state at time t;

[0061] o t =σ(W o ·[h t-1 ,x t ]+b o ),

[0062] Among them t is the output of the output gate at time t, W o is the weight matrix of the output gate, b o is the bias vector of the output gate;

[0063] h t =o t ⊙tanh(c t ),

[0064] where h t is the hidden state at time t, ⊙ is the multiplication of each element;

[0065]

[0066] Among them, P i is the probability of predicting the i-th activity category, c is the total number of activity categories, y i is the i-th element of the fully connected layer output y, and k is the predicted activity category;

[0067] Step 5: By building a target estimation module, using the RGB-D camera and 2D lidar to provide distance information, projecting the RGB image bounding box in the activity recognition module onto the depth image, extracting the center point depth, comparing it with the lidar distance, and estimating the distance obtained by the picker, using the cvReprojectImageTo3D function to convert the 2D image coordinates into 3D coordinates, publishing the target position in the camera frame, and converting it to the map frame map through ROSTF;

[0068] Step 5.1: Project the center point coordinates (u, v) of the bounding box in the RGB image onto the depth image to obtain the depth value d of the point;

[0069] Step 5.2: Convert the 2D image coordinates to 3D coordinates. The corresponding 3D coordinates (X, Y, Z) are calculated using the following expression:

[0070]

[0071] Z=d,

[0072] Where K is the intrinsic parameter matrix of the camera, f x and f y is the focal length of the camera in the x and y directions, c x and c y are the coordinates of the image center;

[0073] Step 5.3: Perform ROSTF conversion to convert the camera frame into a map frame. The corresponding conversion expression is as follows:

[0074]

[0075] in is the transformation matrix from camera frame to map frame, (X, Y, Z) is the 3D point in the camera frame, (X, Y, Z, 1) is the corresponding homogeneous formula, (X map ,Y map ,Z map ) is a 3D point in the map frame, (X map ,Y map ,Z map ,1) is the corresponding homogeneous form, r ij is the rotational component, t x , t y , t z is the translation component.

[0076] Step 6: By building a robot scheduling module, we set up the functions of task scheduling, path planning, resource management, and collision avoidance, and then integrate it with other modules. Deep image processing is used to update task priorities and adjust paths. Sensor data fusion is used to enhance task allocation and navigation decisions. Detailed logs and reports are generated to maintain transparency and facilitate diagnosis.

Claims

1. A vision-based agricultural environment perception and robot-assisted picking method, characterized in that: The steps include: Step 1: Obtain a dataset of fruit picking videos from different environments and by different pickers. This includes videos of the pickers facing away from, sideways to, and toward the robot, as well as videos under different occlusion conditions. The collected video dataset is then processed using cropping technology to split it into two types: picking and non-picking video datasets. Step 2: Build an object detection and tracking module. Using the RGB input from the RealSense camera, we use the YOLOv5 model architecture and the DeepSORT algorithm to detect and track the pickers, generating class labels, bounding boxes, and tracking IDs. Step 3: Build a feature extraction module, using the OpenCV framework and the YOLOv5 object detection model to obtain a series of image frames, their bounding boxes, and the tracking ID corresponding to each picker. Then use the sliding window technique to extract features and classify the picker activities. Features are extracted by using the convolution layer of the convolutional neural network to perform convolution operations with the input image through the convolution kernel, where the three-dimensional convolution expression is as follows: where Q i,j,k is the kth channel value of the output feature map at position (i, j), K m,n,c is the c-th channel value of the convolution kernel at position (m,n), I i+m,j+n,c is the c-th channel value of the output image at position (i+m,j+n), M and N are the convolution kernel sizes, and C is the number of input image channels; By using the frame accumulator to build a data structure to store the image data, boundary coordinate information and tracking ID of each frame; By using the sliding window technology to slide a fixed-size window on the time series to obtain local feature information, the size of the window is determined by statistical analysis of the data features. The statistical analysis of the data features is as follows: in is the variance of the feature within the window, x i is the eigenvalue, is the mean of the eigenvalues within the window, w is the window size, n w is the current frame index; Step 4: We build an activity recognition module that uses the landmarks detected by the feature extraction module as input and uses a custom long short-term memory network to predict the activity associated with each picker in the image. We then use a sliding window technique to determine the optimal size of each sequential input frame to accurately classify the activity. We use a custom LSTM network to predict the relevant activities of each picker in the image. The custom LSTM network consists of three LSTM layers with a unit size of 32 and a softmax at the end. The internal expression of the LSTM unit is as follows: f t =σ(W f ·[h t-1 ,x t ]+b f ), where f t is the output of the forget gate at time t, σ is the sigmoid activation function, W f is the weight matrix of the forget gate, h t-1 is the hidden state of the previous moment, x t is the input at the current moment, that is, the landmark features detected by the feature extraction module, b f is the bias vector of the forget gate; c t =tanh(W c ·[h t-1 ,x t ]+b c ), where c t is the cell state at time t; the t =σ(W o ·[h t-1 ,x t ]+b o ), Among them t is the output of the output gate at time t, W o is the weight matrix of the output gate, b o is the bias vector of the output gate; h t =o t ⊙tanh(c t ), where h t is the hidden state at time t, ⊙ is the multiplication of each element; Among them, P i is the probability of predicting the i-th activity category, c is the total number of activity categories, y i is the i-th element of the fully connected layer output y, and k is the predicted activity category; Step 5: By building a target estimation module, using the RGB-D camera and 2D lidar to provide distance information, projecting the RGB image bounding box in the activity recognition module onto the depth image, extracting the center point depth, comparing it with the lidar distance, and estimating the distance of the picker, using the cvReprojectImageTo3D function to convert the 2D image coordinates into 3D coordinates, publishing the target position in the camera frame, and converting it to the map frame map through ROSTF; Project the center point coordinates (u, v) of the bounding box in the RGB image onto the depth image to obtain the depth value d of the point; Convert the two-dimensional image coordinates into three-dimensional coordinates. The corresponding three-dimensional coordinates (X, Y, Z) are calculated using the following expression: Z=d, Where K is the intrinsic parameter matrix of the camera, f x and f y is the focal length of the camera in the x and y directions, c x and c y are the coordinates of the image center; Perform ROSTF conversion to convert the camera frame into a map frame. The corresponding conversion expression is as follows: in is the transformation matrix from camera frame to map frame, (X, Y, Z) is the 3D point in the camera frame, (X, Y, Z, 1) is the corresponding homogeneous formula, (X map ,Y map ,Z map ) is a 3D point in the map frame, (X map ,Y map ,Z map ,1) is the corresponding homogeneous form, r ij is the rotational component, t x , t y , t z is the translation component; Step 6: By building a robot scheduling module, we set up the functions of task scheduling, path planning, resource management, and collision avoidance, and then integrate it with other modules. Deep image processing is used to update task priorities and adjust paths. Sensor data fusion is used to enhance task allocation and navigation decisions. Detailed logs and reports are generated to maintain transparency and facilitate diagnosis.

Images