Vision-based agricultural environment perception and robot-assisted picking method

Abstract

The invention discloses an agricultural environment perception and robot-assisted picking method based on vision, and the method employs an end-to-end detection, tracking and classification frame, comprises five independent modules: object detection and tracking, feature extraction, activity recognition, target position estimation, and a robot scheduler, obtains information through employing an RGB camera and a two-dimensional laser radar, and carries out the detection and classification of the information. By customizing long and short-term memory network processing data, compared with other methods, an end-to-end architecture adopted by the method can enable the system to comprehensively process the whole process from activity detection of a picker to auxiliary operation of a robot, and adopted multi-camera information fusion can accurately estimate the positions of the picker in different distance ranges, so that the accuracy of picking is improved. And the behavior activity of the picker can be accurately predicted by adopting the customized long-short-term memory network.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, deep learning and sensor fusion, and in particular to a vision-based agricultural environment perception and robot-assisted picking method. Background Art

[0002] Today, the global agricultural sector is facing a series of severe challenges and changes. With the continuous growth of population and the improvement of living standards, people's demand for agricultural products such as fruits is growing. However, the problem of agricultural labor shortage has become increasingly prominent and has become one of the key factors restricting the development of agriculture. Traditional fruit picking mainly relies on manual work, but manual picking has many limitations. For example, long-term and high-intensity work is not only inefficient, but also poses a serious threat to the health of workers, while also increasing medical costs and the risk of labor loss.

[0003] In response to the labor shortage, agricultural robots and autonomous farming systems have developed rapidly in the past few years. However, to date, most agricultural operations face huge challenges in fully automated harvesting due to the complex dynamics of the operations, such as irregular terrain, variable crop growth environments, and the diversity of fruit growth positions and forms. Despite the continuous advancement of technology, there is still a long way to go before robots can accurately identify fruits, avoid obstacles, and efficiently complete harvesting tasks in complex agricultural environments.

[0004] In the field of human-machine collaboration, although certain successes have been achieved in industrial scenarios, the application progress in agricultural scenarios has been relatively slow. Agricultural environments such as orchards are highly open and complex, which brings great difficulties to the perception and decision-making of robots. Robots need to accurately understand and interpret workers' behavior patterns and extract valuable information from them to achieve effective collaboration. However, due to the interference of environmental factors and the diversity and uncertainty of workers' behavior, robots still face many difficulties in processing environmental data and identifying workers' actions and intentions. In addition, ensuring stable and efficient communication between robots and humans and efficiently realizing the interaction between sensors have also become key issues to be solved.

[0005] As the core means to realize robot perception, embedded sensor technology plays an important role in the development of agricultural robots. However, it is not easy to achieve the best cost-effective design of autonomous systems. Farmers need to consider various factors such as investment cost, system performance and maintenance difficulty in practical applications. Although multi-sensory perception systems can provide comprehensive environmental information, they have high equipment procurement and maintenance costs.

[0006] At the same time, the status of existing data set collection also imposes certain constraints on the development of agricultural robots. Most existing data sets are recorded under relatively ideal environmental conditions and cannot reflect the complex situations in real agricultural scenarios. In the actual fruit picking environment, weather conditions are changeable, light fields are variable, and human behavior is highly flexible and random. These complex real-world factors are difficult to reflect in existing data sets, making it difficult for robots to adapt to real agricultural operation scenarios when using these data for learning and training.

[0007] In the face of these challenges, we need an innovative solution that can fully integrate the advantages of humans and robots, improve the efficiency and safety of fruit picking operations, and promote the sustainable development of agricultural production. Based on this demand, this paper proposes a vision-based human-machine collaboration method for fruit picking, which provides new ideas and ways to solve the labor shortage and low picking efficiency in the agricultural field by optimizing perception, decision-making and collaboration mechanisms. Summary of the invention

[0008] The present invention designs an end-to-end vision-based detection, tracking and classification architecture to achieve human-machine collaboration in fruit picking operations. First, the end-to-end vision-based detection, tracking and classification architecture proposed in the present invention integrates five independent modules: object detection and tracking, feature extraction, activity recognition, target position estimation and robot scheduler. This integrated design method enables the system to comprehensively handle the entire process from picker activity detection to robot-assisted operation. Secondly, RGB cameras and two-dimensional lidars are used to obtain information, and the fusion of the two sensors can accurately estimate the position of the pickers within different distance ranges. Finally, a customized long short-term memory network is used to process continuous data and capture the time dependency of landmark sequences extracted from images, thereby accurately relying on the pickers' activities and analyzing the pickers' continuous movements and behavioral changes during the picking process. To achieve the above purpose, the following steps are implemented:

[0009] Step 1: Obtain a fruit picking video dataset of different pickers in different environments, including the pickers' activities with their backs, sides, and faces toward the robot, as well as videos under different occlusion conditions, and use cropping technology to process the collected video dataset into two types of video datasets: picking and non-picking.

[0010] Step 2: Build an object detection and tracking module, using the RGB input from the RealSense camera, using the YOLOv5 model architecture and the DeepSORT algorithm to detect and track human pickers, and generate class labels, bounding boxes, and tracking IDs.

[0011] Step 3: Build a feature extraction module, use the OpenCV framework and the YOLOv5 object detection model to obtain a series of image frames and their bounding boxes and the tracking ID corresponding to each picker, and then use the sliding window technique to extract features and classify the picker activities;

[0012] Step 3.1: Extract features by using the convolution layer of the convolutional neural network to perform convolution operations with the input image through the convolution kernel, where the three-dimensional convolution expression is as follows:

[0013] ,

[0014] in is the output feature map at position The Channel values, is the convolution kernel in Position Channel values, The output image is Position Channel values, , is the convolution kernel size, is the number of input image channels;

[0015] Step 3.2: Use the frame accumulator to build a data structure to store the image data, boundary coordinate information and tracking ID of each frame;

[0016] Step 3.3: Use the sliding window technique to slide a fixed-size window on the time series to obtain local feature information, and use the statistical analysis of data features to determine the size of the window. The statistical analysis of data features is as follows:

[0017] ,

[0018] in is the variance of the feature within the window, is the eigenvalue, is the window size, is the current frame index.

[0019] Step 4: We construct an activity recognition module that uses the landmarks detected by the feature extraction module as input and uses a customized long short-term memory network to predict the relevant activities of each picker in the image. We then use a sliding window technique to determine the optimal size of each sequence input frame to accurately classify the activities.

[0020] Step 4.1: Use a customized LSTM network to predict the relevant activities of each picker in the image. The customized LSTM network consists of three LSTM layers with a unit size of 32 and a Softmax at the end. The internal expression of the LSTM unit is as follows:

[0021] f t = σ ( W f · [ h t − 1 , x t ] + b f ) ,

[0022] in It is the gate of oblivion at the moment The output, is the sigmoid activation function, is the weight matrix of the forget gate, is the hidden state at the previous moment, is the input at the current moment, that is, the landmark features detected by the feature extraction module, is the bias vector of the forget gate;

[0023] c t = tanh( W c · [ h t − 1 , x t ] + b c ) ,

[0024] in is the cell state at time t;

[0025] o t = σ ( W o · [ h t − 1 , x t ] + b o ) ,

[0026] in is the output gate at time The output, is the weight matrix of the output gate, is the bias vector of the output gate;

[0027] ,

[0028] in It's time The hidden state of It is the multiplication of each element;

[0029] ,

[0030] ,

[0031] in It is predicted to be The probability of an activity class, is the total number of activity categories, is the output of the fully connected layer No. elements, is the predicted activity category;

[0032] Step 4.2: Determine the optimal size of each sequence input frame by using the sliding window technique, and evaluate its effectiveness by the variance of the features under different window sizes, where the variance expression is as follows:

[0033] ,

[0034] in is the number of frames in the window, It is The characteristic value of the frame, is the mean of the eigenvalues ​​within the window.

[0035] Step 5: By building a target estimation module, using the RGB-D camera and 2D lidar to provide distance information, projecting the RGB image bounding box in the activity recognition module onto the depth image, extracting the center point depth, comparing it with the lidar distance, to estimate the distance obtained by the picker, using the cvReprojectImageTo3D function to convert the 2D image coordinates into 3D coordinates, publishing the target position in the camera frame, and converting it to the map frame map through ROSTF;

[0036] Step 5.1: Set the center point coordinates of the bounding box in the RGB image Project it onto the depth image to get the depth value of the point ;

[0037] Step 5.2: Convert the 2D image coordinates into 3D coordinates. Calculated by the following expression:

[0038] K = [ f x 0 c x 0 f y c y 0 0 1 ] ,

[0039] ,

[0040] ,

[0041] ,

[0042] in is the camera’s intrinsic parameter matrix, and The camera is and The focal length of the direction, and are the coordinates of the center of the image;

[0043] Step 5.3: Perform ROSTF conversion to convert the camera frame into a map frame. The corresponding conversion expression is as follows:

[0044] T map camera [ r 11 r 12 r 13 t x r 21 r 22 r 23 t y r 31 r 32 r 33 t z 0 0 0 1 ] ,

[0045] [ X map Y map Z map 1 ] = T map camera [ X Y Z 1 ] ,

[0046] in is the transformation matrix from camera frame to map frame, is a 3D point in the camera frame, is the corresponding homogeneous form, is a 3D point in the map frame, is the corresponding homogeneous form, is the rotation component, , , is the translation component.

[0047] Step 6: By building a robot scheduling module, set up task scheduling, path planning, resource management, and collision avoidance functions, and then integrate with other modules. Deep image processing is used to update task priorities and adjust paths. Sensor data fusion is used to enhance task allocation and navigation decisions. Detailed logs and reports are generated to maintain transparency and facilitate diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is an overall flow chart of an embodiment of the present invention; DETAILED DESCRIPTION

[0049] In order to more clearly illustrate the purpose, technical solutions and advantages of the present invention, the present invention will be described in detail below with the help of drawings and specific embodiments.

[0050] Figure 1 This is a flow chart of an embodiment. This embodiment provides a vision-based agricultural environment perception and robot-assisted picking method. The specific process includes: collecting data, detecting and tracking pickers by building an object detection and tracking module, extracting features and classifying picker activities by building a feature extraction module, predicting picker related activities by building an activity recognition module, and estimating the distance of the picker by building a target estimation module. By building a robot scheduling module, the functions of path planning task allocation, resource management and task scheduling are realized.

[0051] A vision-based agricultural environment perception and robot-assisted harvesting method includes the following steps:

[0052] Step 1: Obtain a fruit picking video dataset of different pickers in different environments, including the pickers' activities with their backs, sides, and faces toward the robot, as well as videos under different occlusion conditions, and use cropping technology to process the collected video dataset into two types of video datasets: picking and non-picking.

[0053] Step 2: Build an object detection and tracking module, using the RGB input from the RealSense camera, using the YOLOv5 model architecture and the DeepSORT algorithm to detect and track human pickers, and generate class labels, bounding boxes, and tracking IDs.

[0054] Step 3: Build a feature extraction module, use the OpenCV framework and the YOLOv5 object detection model to obtain a series of image frames and their bounding boxes and the tracking ID corresponding to each picker, and then use the sliding window technique to extract features and classify the picker activities;

[0055] Step 3.1: Extract features by using the convolution layer of the convolutional neural network to perform convolution operations with the input image through the convolution kernel, where the three-dimensional convolution expression is as follows:

[0056] ,

[0057] in is the output feature map at position The Channel values, is the convolution kernel in Position Channel values, The output image is Position Channel values, , is the convolution kernel size, is the number of input image channels;

[0058] Step 3.2: Use the frame accumulator to build a data structure to store the image data, boundary coordinate information and tracking ID of each frame;

[0059] Step 3.3: Use the sliding window technique to slide a fixed-size window on the time series to obtain local feature information, and use the statistical analysis of data features to determine the size of the window. The statistical analysis of data features is as follows:

[0060] ,

[0061] in is the variance of the feature within the window, is the eigenvalue, is the window size, is the current frame index.

[0062] Step 4: We construct an activity recognition module that uses the landmarks detected by the feature extraction module as input and uses a customized long short-term memory network to predict the activities associated with each picker in the image. We then use a sliding window technique to determine the optimal size of each sequence input frame to accurately classify the activities.

[0063] Step 4.1: Use a customized LSTM network to predict the relevant activities of each picker in the image. The customized LSTM network consists of three LSTM layers with a unit size of 32 and a Softmax at the end. The internal expression of the LSTM unit is as follows:

[0064] f t = σ ( W f · [ h t − 1 , x t ] + b f ) ,

[0065] in It is the gate of oblivion at the moment The output, is the sigmoid activation function, is the weight matrix of the forget gate, is the hidden state at the previous moment, is the input at the current moment, that is, the landmark features detected by the feature extraction module, is the bias vector of the forget gate;

[0066] c t = tanh( W c · [ h t − 1 , x t ] + b c ) ,

[0067] in is the cell state at time t;

[0068] o t = σ ( W o · [ h t − 1 , x t ] + b o ) ,

[0069] in is the output gate at time The output, is the weight matrix of the output gate, is the bias vector of the output gate;

[0070] ,

[0071] in It's time The hidden state of It is the multiplication of each element;

[0072] ,

[0073] ,

[0074] in It is predicted to be The probability of an activity class, is the total number of activity categories, is the output of the fully connected layer No. elements, is the predicted activity category;

[0075] Step 4.2: Determine the optimal size of each sequence input frame by using the sliding window technique, and evaluate its effectiveness by the variance of the features under different window sizes, where the variance expression is as follows:

[0076] ,

[0077] in is the number of frames in the window, It is The characteristic value of the frame, is the mean of the eigenvalues ​​within the window.

[0078] Step 5: By building a target estimation module, using the RGB-D camera and 2D lidar to provide distance information, projecting the RGB image bounding box in the activity recognition module onto the depth image, extracting the center point depth, comparing it with the lidar distance, to estimate the distance obtained by the picker, using the cvReprojectImageTo3D function to convert the 2D image coordinates into 3D coordinates, publishing the target position in the camera frame, and converting it to the map frame map through ROSTF;

[0079] Step 5.1: Set the center point coordinates of the bounding box in the RGB image Project it onto the depth image to get the depth value of the point ;

[0080] Step 5.2: Convert the 2D image coordinates into 3D coordinates. Calculated by the following expression:

[0081] K = [ f x 0 c x 0 f y c y 0 0 1 ] ,

[0082] ,

[0083] ,

[0084] ,

[0085] in is the camera’s intrinsic parameter matrix, and The camera is and The focal length of the direction, and are the coordinates of the center of the image;

[0086] Step 5.3: Perform ROSTF conversion to convert the camera frame into a map frame. The corresponding conversion expression is as follows:

[0087] T map camera = [ r 11 r 12 r 13 t x r 21 r 22 r 23 t y r 31 r 32 r 33 t z 0 0 0 1 ] ,

[0088] [ X map Y map Z map 1 ] = T map camera [ X Y Z 1 ] ,

[0089] in is the transformation matrix from camera frame to map frame, is a 3D point in the camera frame, is the corresponding homogeneous form, is a 3D point in the map frame, is the corresponding homogeneous form, is the rotation component, , , is the translation component.

[0090] Step 6: By building a robot scheduling module, set up task scheduling, path planning, resource management, and collision avoidance functions, and then integrate with other modules. Deep image processing is used to update task priorities and adjust paths. Sensor data fusion is used to enhance task allocation and navigation decisions. Detailed logs and reports are generated to maintain transparency and facilitate diagnosis.

Claims

1. A vision-based agricultural environment perception and robot-assisted picking method, characterized in that: The steps include: Step 1: Obtain a fruit picking video dataset of different pickers in different environments, including the pickers' activities with their backs, sides, and faces toward the robot, as well as videos under different occlusion conditions, and use cropping technology to process the collected video dataset into two types of video datasets: picking and non-picking. Step 2: Build an object detection and tracking module, use the RGB input of the RealSense camera, use the YOLOv5 model architecture and DeepSORT algorithm to detect and track the pickers, and generate class labels, bounding boxes, and tracking IDs; Step 3: Build a feature extraction module, use the OpenCV framework and the YOLOv5 object detection model to obtain a series of image frames and their bounding boxes and the tracking ID corresponding to each picker, and then use the sliding window technique to extract features and classify the picker activities; Step 4: We construct an activity recognition module that uses the landmarks detected by the feature extraction module as input and uses a customized long short-term memory network to predict the relevant activities of each picker in the image. We then use a sliding window technique to determine the optimal size of each sequence input frame to accurately classify the activities. Step 5: By building a target estimation module, using the RGB-D camera and 2D lidar to provide distance information, projecting the RGB image bounding box in the activity recognition module onto the depth image, extracting the center point depth, comparing it with the lidar distance, to estimate the distance obtained by the picker, using the cvReprojectImageTo3D function to convert the 2D image coordinates into 3D coordinates, publishing the target position in the camera frame, and converting it to the map frame map through ROSTF; Step 6: By building a robot scheduling module, set up task scheduling, path planning, resource management, and collision avoidance functions, and then integrate with other modules. Deep image processing is used to update task priorities and adjust paths. Sensor data fusion is used to enhance task allocation and navigation decisions. Detailed logs and reports are generated to maintain transparency and facilitate diagnosis.

2. The method of agricultural environment perception and robot-assisted picking based on vision according to claim 1 is characterized in that: The feature extraction module described in step 3 uses the OpenCV framework and the YOLOv5 object detection model to obtain a series of image frames and their bounding boxes and the tracking ID corresponding to each picker, and then uses the sliding window technique to extract features and classify the picker activities; Step 3.1: Extract features by using the convolution layer of the convolutional neural network to perform convolution operations with the input image through the convolution kernel, where the three-dimensional convolution expression is as follows: ， in is the output feature map at position The Channel values, is the convolution kernel in Position Channel values, The output image is Position Channel values, , is the convolution kernel size, C is the number of input image channels; Step 3.2: Use the frame accumulator to build a data structure to store the image data, boundary coordinate information and tracking ID of each frame; Step 3.3: Use the sliding window technique to slide a fixed-size window on the time series to obtain local feature information, and use the statistical analysis of data features to determine the size of the window. The statistical analysis of data features is as follows: ， in is the variance of the feature within the window, is the eigenvalue, is the window size, is the current frame index.

3. The method of agricultural environment perception and robot-assisted harvesting based on vision according to claim 1, characterized in that: As described in step 4, we construct an activity recognition module, take the landmarks detected by the feature extraction module as input, use a customized long short-term memory network to predict the relevant activities of each picker in the image, and then use a sliding window technique to determine the optimal size of each sequence input frame for accurate classification of activities. The specific implementation steps are as follows: Step 4.1: Use a customized LSTM network to predict the relevant activities of each picker in the image. The customized LSTM network consists of three LSTM layers with a unit size of 32 and a Softmax at the end. The internal expression of the LSTM unit is as follows: ， in It is the gate of oblivion at the moment The output, is the sigmoid activation function, is the weight matrix of the forget gate, is the hidden state at the previous moment, is the input at the current moment, that is, the landmark features detected by the feature extraction module, is the bias vector of the forget gate; ， in It's time The cell state; ， in is the output gate at time The output, is the weight matrix of the output gate, is the bias vector of the output gate; ， in It's time The hidden state of It is the multiplication of each element; ， ， in It is predicted to be The probability of an activity class, is the total number of activity categories, is the output of the fully connected layer No. elements, is the predicted activity category; Step 4.2: Determine the optimal size of each sequence input frame by using the sliding window technique, and evaluate its effectiveness by the variance of the features under different window sizes, where the variance expression is as follows: ， in is the number of frames in the window, It is The characteristic value of the frame, is the mean of the eigenvalues ​​within the window.

4. The method of agricultural environment perception and robot-assisted harvesting based on vision according to claim 1 is characterized in that: As described in step 5, by building a target estimation module, using the RGB-D camera and the 2D lidar to provide distance information, projecting the RGB image bounding box in the activity recognition module onto the depth image, extracting the center point depth, and comparing it with the lidar distance, to estimate the distance obtained by the picker, using the cvReprojectImageTo3D function to convert the 2D image coordinates into 3D coordinates, publishing the target position in the camera frame, and converting it to the map frame map through ROSTF, the specific implementation is as follows: Step 5.1: Set the center point coordinates of the bounding box in the RGB image Project it onto the depth image to get the depth value of the point ; Step 5.2: Convert the 2D image coordinates into 3D coordinates. Calculated by the following expression: ， ， ， ， in is the camera’s intrinsic parameter matrix, and The camera is and The focal length of the direction, and are the coordinates of the center of the image; Step 5.3: Perform ROSTF conversion to convert the camera frame into a map frame. The corresponding conversion expression is as follows: ， ， in is the transformation matrix from camera frame to map frame, is a 3D point in the camera frame, is the corresponding homogeneous form, is a 3D point in the map frame, is the corresponding homogeneous form, is the rotation component, , , is the translation component.

Images