Operation method and system of a humanoid agricultural robot based on path planning

Abstract

The present invention discloses an operation method of a humanoid agricultural robot based on path planning, which relates to the technical field of agricultural robots and includes multi-sensor data acquisition, data processing and map generation, map update, target crop image recognition and positioning, global path planning, local path obstacle avoidance, and picking action execution. The present invention also discloses an operation system of a humanoid agricultural robot based on path planning, which includes a humanoid robot body, a vision sensor, a lidar, a depth camera, an inertial measurement unit, an auxiliary sensor, an actuator module, a control system, a communication module, and a power supply system. The humanoid robot of the present invention can use its sensitive visual perception system to identify the position and shape of tools in advance. It can deftly step over or bypass these tools by virtue of its own flexibility and can accurately determine the position and distribution of crops in three-dimensional space, generating a comprehensive picking environment map.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural robots, and particularly relates to an operation method and system for a humanoid agricultural robot based on path planning. Background Art

[0002] With the continuous development of agricultural modernization, the automation and intelligence of agricultural harvesting operations have become the key directions for improving agricultural production efficiency and reducing labor costs. In traditional agricultural harvesting, manual operation is mainly relied on, but this method has many limitations. To overcome the deficiencies of manual harvesting, various agricultural harvesting machines have emerged. Among them, vehicle-type and tracked robots have been applied to a certain extent in the field of agricultural harvesting. Vehicle-type robots usually have a relatively fast moving speed and a large load capacity, and are suitable for operations in large areas of farmland. They can carry various harvesting devices to achieve the harvesting and collection of crops.

[0003] However, vehicle-type robots have obvious deficiencies in terms of adaptability to complex terrains. In farmland, tools such as hoes and shovels may be randomly placed, and when there are raised mounds on the ground, this will pose obstacles to the progress of the robot, making it difficult for the robot to cross some obstacles for flexible operation.

[0004] Moreover, the existing path planning methods for agricultural robots have certain limitations to varying degrees. Although some global path planning algorithms can find the optimal path from the starting point to the end point, in practical applications, due to the inaccuracy and lack of real-time performance of the environmental model, the planned path may be difficult to execute in a complex environment.

[0005] Therefore, it is necessary to propose an operation method and system for a humanoid agricultural robot based on path planning to solve the above problems. Summary of the Invention

[0006] In view of the deficiencies of the prior art, the present invention provides an operation method and system for a humanoid agricultural robot based on path planning, which solves the problems that the existing vehicle-type robots have deficiencies in terms of adaptability to complex terrains, are difficult to cross some obstacles for flexible operation, and due to the inaccuracy and lack of real-time performance of the environmental model, the planned path is difficult to execute in a complex environment.

[0007] To achieve the above object, the present invention is realized through the following technical solutions:

[0008] An operation method for a humanoid agricultural robot based on path planning, comprising the following steps:

[0009] Step 1: Multi-sensor data acquisition: The humanoid agricultural robot is equipped with a variety of sensors, including a vision sensor, a lidar, a depth camera, and an inertial measurement unit;

[0010] Step 2: Data processing and map generation: Preprocess and fuse the data collected by different sensors to generate a picking environment map;

[0011] Step 3: Map update: Continuously collect new environmental data to update the map in real time;

[0012] Step 4: Target crop image recognition and localization: Use the convolutional neural network algorithm to recognize and classify the crop images collected by the vision sensor, and combine the map information to determine the position of the target crop;

[0013] Step 5: Global path planning: Based on the constructed picking environment map and the target crop position information, use the A algorithm for global path planning;

[0014] Step 6: Local path obstacle avoidance: Monitor the environmental information. When encountering sudden obstacles, adjust the movement direction and speed of the robot to avoid the obstacles;

[0015] Step 7: Picking action execution: According to the position and posture of the target crop, plan the motion trajectory of the robotic arm, use the inverse kinematics algorithm to solve the angle values of each joint of the robotic arm, and adjust the action parameters of the picking tool to achieve fruit picking.

[0016] Optionally, the steps of target crop image recognition and localization in Step 4 are as follows:

[0017] Training process: Using the method of supervised learning, input several images and their labels into the model, calculate the predicted output of the model through forward propagation, and calculate the loss function according to the difference between the predicted output and the true label; Through the backpropagation algorithm, adjust the parameters of the model according to the loss function to reduce the value of the loss function;

[0018] Recognition process: During the picking operation, input the real-time collected crop images into the trained CNN model. The model calculates the probability distribution of the input images belonging to various crops through forward propagation, and determines the crop category to which the input image belongs according to the maximum value in the probability distribution, so as to obtain the recognition result of the target crop;

[0019] The way to obtain the position information of the target in the image is: After obtaining the recognition result, introduce a region proposal network in the CNN model to generate region proposals containing the target crop in the image, assign a score to each proposal, and then screen and adjust these proposals according to the scores to obtain the target position box;

[0020] Determine the position of the target in the robot coordinate system: Obtain the relative pose relationship between the vision sensor and the robot through sensor calibration. The purpose of sensor calibration is to determine the internal and external parameters of the vision sensor. The sensor calibration method is the checkerboard calibration method to solve the internal and external parameters of the camera.

[0021] Determine the target position: The pixel coordinates in the image are used to solve the coordinates of the target in the camera coordinate system. Then, according to the relative pose relationship between the camera and the robot, the coordinates of the target in the camera coordinate system are transformed into the robot coordinate system to determine the position of the target crop in the robot coordinate system.

[0022] An operation system of a humanoid agricultural robot based on path planning, including:

[0023] Humanoid robot body: It includes a head, a torso, two arms, and two legs. Each part is connected by joints, and motors and reducers are equipped at the joints.

[0024] Vision sensor: Installed on the robot's head or arms to collect image information of crops.

[0025] Lidar and depth camera: Installed at different positions of the robot, including the head and shoulders, to obtain the three-dimensional structure information of the environment.

[0026] Inertial measurement unit: Installed at the center of gravity of the robot to monitor the posture and motion state of the robot in real time.

[0027] Auxiliary sensors: Include contact sensors and pressure sensors to sense the contact situation between the robot and the surrounding environment and provide feedback information for the execution of picking actions.

[0028] Actuator module: Includes a robotic arm and a leg walking mechanism. The humanoid robot is equipped with two robotic arms, and the end effector of the robotic arm replaces different types of tools according to different picking task requirements, including grippers and suction cups.

[0029] Control system: Includes a main controller and a motion controller. The main controller is responsible for the coordinated control and management of the entire robot system, and the motion controller is used to control the motion of the motors of each joint of the robot to achieve motion trajectory tracking.

[0030] Data processing unit: Responsible for collecting and processing data from various sensors and converting it into a format that can be recognized and used by the robot control system.

[0031] Central processing unit: Used to process data of computationally intensive tasks such as image recognition and path planning and various mathematical operations.

[0032] The present invention provides an operation method and system of a humanoid agricultural robot based on path planning, which has the following beneficial effects:

[0033] 1. The humanoid robot of the present invention can use its keen visual perception system to identify the position and shape of tools in advance. It can rely on its own flexibility to cleverly step over or around these tools to avoid being tripped or damaging the tools. Moreover, the humanoid robot can easily cross small mounds in the field with its flexible leg structure, so that the humanoid robot can maintain efficient mobility in complex farmland terrain. During the movement, the humanoid robot can also dynamically adjust the path according to the ever-changing environment. If a new small mound suddenly appears or someone moves the position of the hoe or shovel, it can respond quickly. By quickly processing the new environmental information, the humanoid robot can replan the path to ensure that it can always reach the target location smoothly and reduce the interference caused by terrain changes.

[0034] 2. The present invention combines crop information acquired by visual sensors with spatial information acquired by lidar and depth cameras, which can more accurately determine the position and distribution of crops in three-dimensional space. The generated comprehensive harvesting environment map contains information on topography, crop distribution, obstacle locations, and other aspects, providing a comprehensive and accurate basis for the robot's path planning, task allocation, and operation strategy formulation. Based on such a map, the robot can more intelligently plan the optimal path, avoid collisions with obstacles, efficiently complete the harvesting operation, and improve work efficiency and quality.

[0035] 3. The present invention designs special picking tools for different types of fruits, and adopts suction cups for soft fruits, which improves the adaptability of the picking tools to fruits of different shapes and textures, reduces damage to the fruits during the picking process, and ensures the integrity and quality of the fruits. The action parameters of the picking tools are adjusted according to factors such as the size, shape and maturity of the fruits. By setting a reasonable initial opening and closing degree, adsorption force and proportional coefficient, the picking force can be accurately controlled according to actual conditions to avoid picking too loosely or too tightly, further improving the accuracy and efficiency of picking, and reducing damage to fruit trees. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the process of the operation method of the humanoid agricultural robot of the present invention;

[0037] Figure 2 Schematic diagram of the modules of the operating system of the humanoid agricultural robot of the present invention. DETAILED DESCRIPTION

[0038] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention. Embodiment 1

[0039] Please refer to FIG. 1. A working method of a humanoid agricultural robot based on path planning includes the following steps:

[0040] Step 1: Multi-sensor data collection: The humanoid agricultural robot is equipped with a variety of sensors, including a vision sensor, a lidar, a depth camera, and an inertial measurement unit (IMU). Before entering the picking area, start each sensor to comprehensively scan and collect data on the surrounding environment. The vision sensor is used to obtain the position, color, and shape information of the crops; the lidar and the depth camera are used to obtain the three-dimensional structure information of the environment, laying the foundation for constructing a high-precision point cloud map; the IMU is used to real-time monitor the posture and motion state of the robot, providing data support for positioning and navigation.

[0041] Step 2: Data processing and map generation: Preprocess and fuse the data collected by different sensors to generate a picking environment map containing comprehensive information on terrain, crop distribution, and obstacle positions.

[0042] Step 3: Map update: During the operation of the robot, continuously collect new environmental data and update the map in real time to adapt to environmental changes and ensure the accuracy of path planning.

[0043] Step 4: Target crop image recognition and positioning: Use the convolutional neural network (CNN) algorithm in deep learning to recognize and classify the crop images collected by the vision sensor. Train the CNN model with a number of crop sample images so that it can accurately recognize different types and different growth stages of crops. During the picking operation, input the real-time collected crop images into the trained CNN model to identify the target crops and obtain their position information; according to the relative position relationship between the vision sensor and the robot body, convert the pixel coordinates of the target crops in the image into the actual coordinates in the robot's working space, and combine with the map information to determine the position of the target crops in the picking area, providing target points for the path planning and picking actions of the robot.

[0044] Step Five: Global Path Planning: Based on the constructed picking environment map and the position information of the target crop, the A* algorithm is used for global path planning. The A* algorithm comprehensively considers the length and cost of the path and can find the optimal path while taking into account the motion energy consumption of the robot. During the planning process, the obstacle information in the map is used as a constraint condition to prevent the robot from colliding with obstacles. Through global path planning, the initial path from the starting position of the robot to the position of the target crop is determined;

[0045] Step Six: Local Path Obstacle Avoidance: During the process of the robot walking along the global path, the surrounding environment information is monitored in real time. When encountering sudden obstacles, local path planning is started, and the dynamic window approach (DWA) is used for local path planning. According to the current speed, acceleration, angular velocity of the robot and the position and speed information of the surrounding obstacles, the safe motion trajectory of the robot is calculated in real time. By continuously adjusting the motion direction and speed of the robot, it can avoid obstacles and return to the global path at the same time, and continue to move forward towards the target crop;

[0046] Step Seven: Picking Action Execution: When the robot reaches near the target crop, according to the position and posture of the target crop, the motion trajectory of the robotic arm is planned, and the inverse kinematics algorithm is used to solve the angle values of each joint of the robotic arm, so that the end effector of the robotic arm approaches the target crop. During the motion process, the motion trajectory of the robotic arm is adjusted through real-time feedback control to ensure that it can complete the picking action smoothly and accurately. The end effector of the robotic arm is equipped with a special picking tool, such as a flexible gripper or a suction cup. When the end effector of the robotic arm approaches the target crop fruit, according to the size, shape and maturity information of the fruit, the action parameters of the picking tool are adjusted to achieve fruit picking. The picked fruit is conveyed to the storage container carried by the robot through the internal conveying device to complete a picking operation;

[0047] Step Eight: Cyclic Operation: After completing a picking operation, the robot automatically searches for the next target crop according to the preset task list and the current picking situation that has been completed.

[0048] The vision sensor (such as a camera) obtains visual information by taking pictures of the picking area. The camera usually captures color or grayscale images at a certain resolution (such as 1920×1080 pixels) and frame rate (such as 30 frames per second). During the acquisition process, the lens captures the image formed by the reflection of light, and each pixel contains color (for a color camera) or brightness (for a grayscale camera) information, and these information reflect the surface characteristics of the crops, obstacles and terrain;

[0049] LiDAR measures distance by emitting laser beams and receiving the reflected light. It emits a series of rapid pulses of laser beams into the surrounding environment. When the laser beams encounter obstacles, they will reflect back. The receiver calculates the distance based on the time difference between the emission and reception of the laser. Through rotational and pitching movements, LiDAR scans in two-dimensional or three-dimensional space to obtain the point cloud data of the surrounding environment. Each point in the point cloud data contains the three-dimensional coordinate information of that point. , where , represents the position in the horizontal direction, represents the height information; LiDAR can provide high-precision distance measurement, accurately describe the object contours and positions in the environment, and has a wide measurement range;

[0050] A depth camera uses infrared light or structured light technology to obtain the depth information of an object. A depth camera based on infrared light calculates the distance between the object and the camera by emitting infrared light and receiving the reflected light; A structured light depth camera projects a known light pattern (such as stripes, speckles) and determines the depth of the object surface by analyzing the deformation of the light pattern. The data output by the depth camera is presented in the form of a depth image, and the gray value of each pixel point represents the distance from that point to the camera; A depth camera can provide relatively accurate depth information in an indoor environment and can also measure the distance of objects with less obvious textures well.

[0051] An Inertial Measurement Unit (IMU) is a device that integrates accelerometer, gyroscope, and magnetometer sensors. The accelerometer is used to measure the linear acceleration of the robot in all directions, the gyroscope is used to measure the angular velocity of the robot, and the magnetometer measures the direction of the earth's magnetic field to determine the orientation of the robot. Through the combination of these sensors, the IMU monitors the posture (including roll, pitch, and yaw angles) and motion state (such as acceleration, angular velocity) of the robot in real time; IMU data has the characteristics of high frequency and low noise, can provide accurate posture and motion information, and helps the robot maintain balance and determine its own position in a dynamic environment.

[0052] The data preprocessing methods in data processing and map generation in Step 2 are as follows:

[0053] Visual sensor data preprocessing: Filtering algorithms (such as Gaussian filtering, median filtering) are used to remove noise points in the image. Gaussian filtering performs weighted average operations on image pixels through a convolution kernel to smooth the image and reduce the impact of noise on subsequent analysis. Since uneven illumination may cause color distortion in the image, color correction is required. The gray histogram of the image is adjusted through histogram equalization to make the gray distribution of pixel points uniform and enhance the contrast of the image. Image segmentation algorithms (such as threshold segmentation, region growing) are used to separate crops, obstacles, and the background in the image. The image segmentation method based on threshold segmentation compares the gray value of pixel points with a set threshold and divides the pixel points into different categories;

[0054] Lidar data preprocessing: Statistical filtering (such as mean filtering, statistical outlier removal) is used to remove noise points in the point cloud data. To reduce the data volume and improve the processing efficiency, downsampling is performed on the point cloud data. The downsampling method is voxel grid downsampling. Voxel grid downsampling divides the point cloud space into regular three-dimensional grids (voxels) and replaces the points within the voxel with the average value of the voxel center point;

[0055] Depth camera data preprocessing: The bilateral filtering method is used to filter the depth image to remove noise and outliers in the depth data. Bilateral filtering not only considers the spatial position relationship of pixel points but also considers the depth value difference of pixel points, and can smooth the depth image while retaining edge information. Due to object occlusion or sensor measurement limitations, there may be holes in the depth image. The holes are filled through interpolation algorithms (such as nearest neighbor interpolation, linear interpolation). The nearest neighbor interpolation algorithm finds the depth value of the non-hole point closest to the hole point as the depth value of the hole point;

[0056] IMU data preprocessing: The low-pass filtering method is used to remove high-frequency noise in the IMU data. Low-pass filtering allows signals below the cut-off frequency to pass through and blocks signals above the cut-off frequency by setting the cut-off frequency, thereby smoothing the IMU data. Since the data acquisition frequencies of different sensors may be different, the IMU data is time-synchronized with other sensor data to ensure data consistency, and data synchronization is achieved through hardware clock synchronization or software algorithms (such as timestamp matching, linear interpolation).

[0057] The data fusion processing method in data processing and map generation in Step 2 is as follows:

[0058] Coordinate system transformation: The data of different sensors are fused into a unified coordinate system, and coordinate system transformation is performed on the data of each sensor. Let the coordinate systems of the visual sensor, lidar, depth camera, and IMU be 、 、 and , and the unified target coordinate system is , through the coordinate transformation matrix R and the translation vector , the point is transformed from the coordinate system to the coordinate system : , where and are respectively the rotation matrix and the translation vector from the coordinate system to the coordinate system . For the coordinate system transformation between multiple sensors, it is achieved through cascaded transformation;

[0059] Visual and LiDAR data fusion: Project the point cloud data obtained by the LiDAR onto the image plane, and use the feature points (such as corner points and edge points) of the image to register with the point cloud data. The registration method is the Iterative Closest Point algorithm (ICP). The Iterative Closest Point algorithm solves the optimal coordinate transformation parameters (rotation matrix and translation vector) by minimizing the sum of the squares of the distances between the corresponding points of the point cloud and the image feature points; On the basis of registration, combine the color information in the visual image with the three-dimensional coordinate information in the point cloud data to construct a fused map. Each pixel point in the fused map contains color information and the corresponding three-dimensional coordinate information, which can more comprehensively reflect the characteristics of the picking environment;

[0060] Visual, LiDAR and depth camera data fusion: Fuse the depth image obtained by the depth camera with the point cloud data of the LiDAR to improve the accuracy and reliability of the depth information. Through weighted fusion, different weight coefficients are assigned according to the accuracy of the depth camera and the LiDAR in different regions, and then the weighted average depth value is calculated; Combine the semantic segmentation results in the visual image (classify the pixel points in the image into different categories such as crops and obstacles) with the depth information to further fuse the content of the map, and mark the boundaries and positions of different category objects in the fused map to provide more accurate information for subsequent path planning and decision-making;

[0061] Incorporate inertial measurement unit data fusion: Use the attitude information measured by the inertial measurement unit to correct the attitude of the feature points (such as the center position of the crop and the edge of the obstacle) in the fused map. The coordinates of the feature points are transformed into the coordinates in the global coordinate system through the coordinate transformation matrix and adjusted according to the attitude information of the inertial measurement unit; Combine the motion state information (acceleration, angular velocity) of the inertial measurement unit to optimize the motion trajectory of the robot in the picking environment. Adjust the speed planning of the robot according to the acceleration information measured by the IMU to improve the picking efficiency while ensuring safety.

[0062] The way to update the map in step three is:

[0063] New data acquisition: The sensor collects new environmental data at a set frequency and preprocesses the data.

[0064] Change detection: Compare the newly collected data with the existing map data to detect whether the environment has changed. Change detection is achieved by calculating the differences in data (such as the change in grayscale values of pixel points, the change in density of point cloud data).

[0065] Local update: If a change in the environment is detected, update the local area where the change occurred. Local update can reduce the computational amount and improve the update efficiency. For example, when it is detected that the crops in a certain area have been picked, only update the map information of that area (such as deleting the corresponding crop model).

[0066] Global update: Every 1 hour, perform a global update on the entire map. Global update can re-integrate all sensor data to construct a new fused map.

[0067] The update algorithm is the Kalman filter: The Kalman filter is a recursive state estimation algorithm used for map update. The state variables of the map (such as the position and color of feature points) are regarded as the system state, and the sensor measurement data is regarded as the observation value. The current state of the map is estimated through two steps: prediction and update. In the prediction step, the current state of the map is predicted based on the motion model of the robot and the map state at the previous moment; in the update step, the predicted map state is corrected using the sensor measurement data to obtain the updated map state.

[0068] The global path planning method in Step Five is as follows:

[0069] The A* algorithm is a centralized search strategy that combines the advantages of the best-first search and the Dijkstra algorithm. It selects the next expansion node through an evaluation function where represents the actual cost from the start node to node n (such as path length, energy consumption), represents the estimated cost from node n to the target node (heuristic function);

[0070] S1: Construct the search space: In the picking environment map, define the starting position of the robot, the position of the target crop, and some key intermediate points (such as turning points, intersections) as nodes. Each node is represented by its coordinates in the map For example, the start node , the target node ; The line segment connecting two adjacent nodes is called an arc. The cost of the arc is determined based on the distance between the nodes, the terrain complexity (such as flat terrain, rugged terrain), and the energy consumption factor. The determination method is based on the Euclidean distance calculation: where Represents the cost of the arc from the source node to the destination node ;

[0071] S2: Create an open list and a closed list: The open list is used to store the nodes to be expanded, sorted in ascending order according to the value of the evaluation function f(n). Initially, the starting node S is placed in the open list; the closed list is used to store the nodes that have been expanded. During the execution of the algorithm, once a node is expanded, it is transferred from the open list to the closed list; place the starting node S in the open list, and set the , of the starting node according to the estimated distance from the starting node to the target node (using the Euclidean distance);

[0072] S3: Select the node n with the smallest evaluation function value from the open list as the current expansion node. If the current expansion node n is the target node G, the optimal path is found and the algorithm ends. At this time, trace back from the target node along the parent node pointer step by step to the starting node to obtain the complete path; otherwise, transfer the current expansion node n from the open list to the closed list; for each adjacent node of the current expansion node n (i.e., the nodes directly connected to n), if is already in the closed list, skip this node; if is not in the open list, add it to the open list, calculate its evaluation function and set its parent node pointer to point to the current node n. If is already in the open list, compare the new path cost with the original path cost. If the new path cost is smaller, update 's evaluation function value and parent node pointer, and loop until the open list is empty or the target node is found;

[0073] S4: Path backtracking: If the target node is found, start from the target node and return to the previous node in turn according to the parent node pointer until returning to the starting node to obtain the optimal path.

[0074] In this embodiment, the collaborative work of multiple sensors can obtain the environmental information of the picking area from multiple dimensions. The vision sensor obtains the position, color, and shape information of the crops, which helps to accurately identify different types and maturity levels of crops. The 3D structure information obtained by the lidar and depth camera can accurately construct a point cloud map of the environment, providing high-precision spatial data support for subsequent path planning and operation. The IMU monitors the posture and motion state of the robot in real time to ensure that the robot can operate stably in complex terrains and improve the accuracy of positioning and navigation. Different types of sensors can adapt to various complex agricultural environments and operation scenarios. Whether it is an outdoor farmland with changing lighting conditions or a relatively closed greenhouse environment with occlusion, through the complementary advantages of each sensor, the environmental information can be comprehensively and accurately collected, enabling the robot to better meet the operation requirements in different environments. By preprocessing and fusing the data collected by different sensors, the scattered and heterogeneous data can be integrated, the redundancy and contradiction between the data can be eliminated, and more valuable information can be extracted. For example, by combining the crop information obtained by the vision sensor with the spatial information obtained by the lidar and depth camera, the position and distribution of the crops in the 3D space can be determined more accurately.

[0075] The generated comprehensive picking environment map contains information on terrain, crop distribution, and obstacle positions, providing a comprehensive and accurate basis for the robot's path planning, operation task allocation, and operation strategy formulation. Based on such a map, the robot can more intelligently plan the optimal path, avoid colliding with obstacles, efficiently complete the picking operation, and improve the operation efficiency and quality. During the operation process, the sensors continuously collect environmental information, and the data processing system can update the picking environment map in real time to enable it to dynamically adapt to environmental changes. For example, when the position of the crops changes during growth or new obstacles appear, the robot can timely adjust the path planning and operation strategy to ensure the smooth progress of the operation.

[0076] Embodiment 2

[0077] This embodiment makes the following optimizations based on Embodiment 1. Specifically, the architecture of the convolutional neural network (CNN) algorithm in the target crop image recognition and positioning in Step 4 includes:

[0078] Input layer: Receives the crop image data preprocessed (such as normalized processing). The image is a color image with a certain resolution (such as 640×480 pixels). Each pixel contains the information of three RGB channels, representing the intensity values of the red, green, and blue color components, with a value range of 0-255. The pixel values are mapped to the 0-1 interval to accelerate model training and improve numerical stability.

[0079] Convolutional layer: The convolutional layer is the core component of a CNN. It performs a convolution operation by sliding a convolution kernel (also known as a filter) over the image to extract local features of the image. For example, for a 3×3 convolution kernel, it performs a convolution operation on a 3×3 region of the image to calculate a feature value. The parameters (weights) of the convolution kernel are continuously updated during the training process to learn the patterns that best represent the image features;

[0080] Pooling layer (downsampling layer): The pooling layer usually follows the convolutional layer immediately. Its main function is to reduce the dimensionality of the output feature map of the convolutional layer, reduce the amount of data, and retain important information at the same time. The pooling operations are max pooling and average pooling;

[0081] Fully connected layer: After being processed by multiple convolutional layers and pooling layers, the obtained feature map is flattened into a one-dimensional vector and input into the fully connected layer. Each neuron in the fully connected layer is connected to all neurons in the previous layer, performs a weighted sum of the previously extracted features, and performs a non-linear transformation (through an activation function);

[0082] Output layer: Each neuron in the output layer represents a type of crop. The output vector is converted into a probability distribution through the Softmax function, indicating the probability that the input image belongs to various crops. For example, for a crop recognition task with three types of crops (such as apples, oranges, and bananas), after the output vector is processed by the Softmax function, a probability vector of length 3 is obtained , where , , respectively represent the probabilities that the input image belongs to apples, oranges, and bananas, and , and the expression of the Softmax function is: , where: represents the probability that the input image belongs to the i-th type of crop; is the output value of the neuron in the output layer; is the number of crop types;

[0083] The steps for target crop image recognition and localization in Step Four are as follows:

[0084] Training process: To enable the CNN model to accurately identify crops of different types and growth stages, a large number of crop sample images are required to train the model. Using the method of supervised learning, that is, for each training image, there is a corresponding label (indicating the category of the crop). During training, several images and their labels are input into the CNN model. Through forward propagation, the predicted output of the model is calculated. Then, according to the difference between the predicted output and the true label, a loss function (such as the cross-entropy loss function) is calculated. Through the backpropagation algorithm, according to the loss function, the parameters of the model (including the weights of the convolutional kernels, the weights and biases of the fully connected layers) are adjusted to reduce the value of the loss function. The core of the backpropagation algorithm is to calculate the gradient of the loss function with respect to the model parameters, and then update the parameters according to the principle of gradient descent.

[0085] Recognition process: During the picking operation, the real-time collected crop image is input into the trained CNN model. The model calculates the probability distribution of the input image belonging to various crops through forward propagation. According to the maximum value in the probability distribution, the crop category to which the input image belongs is determined, so as to obtain the recognition result of the target crop. For example, if the probability distribution output by the model is [0.1, 0.8, 0.1], it means that the probability of this image belonging to the second type of crop (such as an orange) is the highest, so the recognition result is an orange.

[0086] The method for obtaining the position information of the target in the image is as follows: After obtaining the recognition result of the target crop, it is necessary to further determine the precise position of the target in the image. By introducing a Region Proposal Network (RPN) into the CNN model, the RPN generates a series of region proposals in the image that may contain the target crop, and assigns a score to each proposal, indicating the possibility that the proposal contains the target. Then, according to the scores, these proposals are screened and adjusted to obtain the target position box.

[0087] Determining the position of the target in the robot coordinate system: To convert the position of the target crop from the image coordinate system to the robot coordinate system, the relative pose relationship between the visual sensor and the robot is obtained through sensor calibration. The purpose of sensor calibration is to determine the internal parameters (such as focal length, principal point, etc.) and external parameters (such as rotation matrix, translation vector) of the visual sensor. The sensor calibration method is the checkerboard calibration method. By taking pictures of checkerboard images with known sizes and grid spacings, and using the geometric relationship of the checkerboard and the pixel coordinates in the image, the internal and external parameters of the camera are solved.

[0088] Determining the target position: The pixel coordinates in the image are used to solve the coordinates of the target in the camera coordinate system. Then, according to the relative pose relationship between the camera and the robot, the coordinates of the target in the camera coordinate system are converted to the robot coordinate system to determine the position of the target crop in the robot coordinate system.

[0089] In this embodiment, a convolutional neural network (CNN) algorithm is used to identify and classify crop images collected by a visual sensor. A large number of sample crop images are used for training, allowing the model to learn to distinguish between crops of different types and growth stages, thereby accurately identifying target crops in actual operations. Over time, new sample images can be continuously collected to update and optimize the CNN model, thereby improving its ability to recognize newly appearing or changing targets, ensuring a consistently high recognition accuracy rate.

[0090] The pixel coordinates of the target crop in the image are converted into actual coordinates in the robot's workspace. This step can match the information in the image with the location in the real world, allowing the robot to more accurately determine the location of the target crop. Using the constructed harvesting environment map, the converted coordinates can be further verified and corrected to ensure that the final target location is consistent with the image recognition results and coordinated with the surrounding environment, providing an accurate target point for subsequent path planning.

[0091] Deep learning models, especially CNNs, excel in image recognition. They can handle complex background interference and effectively identify target crops even in the presence of diverse vegetation or other obstructions. Through extensive training with samples, CNN models are robust to changes in lighting, maintaining good recognition performance under varying lighting conditions and reducing misidentifications caused by light intensity.

[0092] Accurate identification and positioning can directly provide clear target points for the robot's path planning and picking actions, avoiding unnecessary searches and explorations, saving time and resources, and improving overall operational efficiency.

[0093] Example 3

[0094] This embodiment is based on Example 1 or Example 2 and is optimized as follows. Specifically, the steps of local path planning in the local path obstacle avoidance in step 6 are as follows:

[0095] DWA Principle: The dynamic window method is a local path planning method based on velocity space sampling. It represents the robot's motion state as a state space (including position, velocity, and acceleration). By sampling and predicting the state space, it evaluates whether each sampling point (i.e., possible action) will cause a collision and selects the optimal action to control the robot's motion.

[0096] State space representation: The state of the robot is represented by a vector Indicates that is the position coordinate of the robot in the two-dimensional plane, is the orientation angle of the robot (relative to a certain reference direction), is the robot's and velocity components in the direction;

[0097] Sampling: At each moment, based on the current state of the robot and the control input set (including discrete combinations of acceleration and angular acceleration), multiple next states (i.e., predicted states) are generated. The sampling density is adjusted according to factors such as the robot's speed and environmental complexity. For example, sparse sampling is performed in a simple environment, and dense sampling is performed in a complex environment;

[0098] Prediction: According to the kinematic and dynamic models of the robot, predict the new state of each sampled state after a period of time (such as );

[0099] Collision detection: For each predicted state, check whether the robot collides with an obstacle. Replace the obstacle with a geometric shape and determine whether there is a collision by calculating the distance between the robot and the obstacle. If the distance between the predicted state and any obstacle is less than the safety threshold (considering the size of the robot and the buffer), it is considered that a collision will occur.

[0100] The local path obstacle avoidance in Step 6 also includes obstacle recognition and classification and dynamic crossing strategies. The steps of obstacle recognition and classification are as follows:

[0101] Data collection and annotation: Before actual application, collect a number of images, point cloud data, and corresponding sensor data containing various obstacles in different terrain backgrounds. Various obstacles include mounds of different sizes, shapes, and postures, hoes and shovels of various types and placement methods, and different terrain backgrounds include soils of different textures and whether there is crop coverage. These data are annotated through manual annotation or automatic annotation tools, marking the types and key features of the obstacles, including the height, slope, and volume of the mounds, and the type, size, placement posture, and whether they are occluded of the hoes and shovels;

[0102] Feature extraction and model training: Use deep learning algorithms to extract features from the collected data. For image data, the CNN automatically learns the shape features, color texture features of the obstacles; for point cloud data, extract three-dimensional shape features and spatial distribution features through the PointNet network structure. Use these extracted features as inputs and use a support vector machine (SVM) or random forest for model training. During the training process, adjust the parameters of the model (such as the number of layers of the neural network) to enable the model to classify different obstacles;

[0103] Classification result output: In actual applications, the trained model inputs the sensor data collected in real time into the model and outputs the classification results of obstacles. The mounds are classified by height into mounds with heights , and . For hoes and shovels, identify their types, sizes, placement postures, and whether they are blocked by crops or soil. The types include ordinary hoes, long-handled hoes, and small shovels. The size is divided according to length, width, and weight parameters. The placement postures include upright, obliquely inserted, lying flat, and fallen;

[0104] The steps of the dynamic crossing strategy for mounds are as follows:

[0105] Scenario 1, the control mode for mounds with a height of :

[0106] Motion planning: When the robot recognizes a mound ahead, based on the height, slope, and shape information of the small mound, plan the movement trajectory of the leg joints. Assume the height of the small mound is 4 cm. The robot first adjusts the center of gravity of the body to move forward, so that the center of gravity is in a position forward between the two feet to increase the forward tilt trend. Then, control the leg joints to move according to the predetermined trajectory. The hip joint and knee joint work together. The hip joint first flexes to lift the leg, and the knee bends. Then the hip joint extends, and at the same time the knee joint extends, so that the toes can smoothly cross the top of the mound. Finally, the sole of the foot lands smoothly. During the whole process, the movement speed of the leg joints is adjusted according to the actual situation of the mound, and the crossing speed is 1 m / s to ensure the smoothness and accuracy of the action;

[0107] Balance control: During the crossing process, use the inertial measurement unit to continuously monitor the body attitude angular velocity and acceleration information of the robot, combined with the foot pressure change feedback from the pressure sensor, and adjust the center of mass position of the robot through the PID controller. For example, when the robot starts to cross the mound, due to the forward movement of the center of gravity and the action of lifting the leg, it may cause the body to lean forward. At this time, the PID controller will automatically adjust the torque output of the hip joint and knee joint according to the pitch angle change detected by the IMU and the pressure change of the pressure sensor, so that the center of mass of the robot remains within a stable range and avoid falling due to imbalance;

[0108] Scenario 2, the control mode for mounds with a height of :

[0109] Body posture adjustment: The robot first lowers its center of gravity by adjusting the angles of the leg joints. The hip joint and knee joint flex simultaneously (the hip joint flexes by 2 degrees and the knee joint flexes by 3 degrees), moving the center of gravity of the body downward and closer to the support area of the feet. At the same time, the upper body leans forward to further move the center of mass of the body forward to better cope with terrain changes ahead;

[0110] Footstep motion planning: Shorten the stride length to increase the stability of the footsteps. When taking a step, the sole part touches the ground first, gradually increasing the contact area to disperse the pressure. As the foot moves forward, adjust the direction and force of the footsteps in real time according to terrain changes. For example, when the ground slope is large, the robot increases the friction between the sole and the ground by adjusting the anti-slip material on the sole or a special sole structure. At the same time, use the swing of the arms to assist in maintaining balance, and the swing amplitude and frequency of the arms are adaptively adjusted according to the movement state of the body. During the entire crossing process, the moving speed of the robot is 0.5 m / s to ensure that each step is steady and reliable;

[0111] Scenario 3, for the control mode when encountering a mound of a certain height :

[0112] Path planning: The robot first activates the path planning module, which comprehensively considers the height, slope, volume of the large mound and the surrounding terrain information (such as whether there is a flat area for detouring), and searches for a feasible path in the global terrain model. For example, if there is a relatively flat area on one side of the mound, the robot will plan a path to bypass this area;

[0113] Obstacle avoidance and replanning: During the process of walking along the planned path, the robot continuously monitors the surrounding terrain changes and the situation of other obstacles. If a new obstacle is found or the original path is infeasible, the robot will immediately stop the current action, re-run the path planning algorithm, and search for a new feasible path. When replanning the path, the robot will update the terrain model and obstacle information according to the newly obtained information to ensure that the re-planned path is more reasonable and safe;

[0114] The steps of the dynamic crossing strategy for hoe and shovel tools are as follows:

[0115] Scenario 1, for the mode of upright or obliquely inserted tools:

[0116] Approach strategy: For upright or obliquely inserted tools, the robot first approaches the tool, maintaining a distance of 1 meter. During the approach process, use the end effector of the robotic arm to make preparatory movements to prepare for grasping the tool;

[0117] Grasping and Moving: When the robot is 0.5 meters away from the tool, adjust the movement trajectory of the robotic arm and the posture of the gripper or suction cup according to the type and size of the tool. For example, for a longer hoe, the robotic arm extends, the gripper opens, and grasps the rod or neck of the hoe. For tools inserted obliquely, the angle and grasping position of the gripper need to be adjusted to adapt to the inclination angle of the tool. After grasping the tool, the robotic arm lifts it upward to lift the tool off the ground. Then, according to the terrain conditions, the robot places the tool in an open area to avoid affecting the robot's movement.

[0118] Scenario 2, the mode for lying or lodging tools:

[0119] Recognition and Judgment: For tools lying flat on the ground, the robot first recognizes their type, size, and placement posture. By scanning the external contour and size information of the tool through a stereo vision camera or lidar, and comparing it with the tool model database, the specific information of the tool is determined.

[0120] Crossing Decision: If the width of the tool is less than 0.5 meters, the robot chooses to cross. When crossing, adopt the strategy of crossing the mound. If the tool is greater than 0.5 meters, the robot stops moving forward, evaluates the surrounding terrain and environmental conditions, and chooses to bypass the tool or use the robotic arm to move it to a safe place and then continue to move forward according to the situation.

[0121] In this embodiment, the DWA principle evaluates actions through sampling and prediction in the state space, and can quickly adapt to the complex and changeable farmland environment. The motion state of the robot is represented as a state space including position, speed, and acceleration, which can comprehensively describe the dynamic characteristics of the robot. In actual farmland operations, operations such as starting, stopping, and turning of the robot all involve these factors. DWA can make accurate plans based on this to ensure smooth and reliable movement. Adjust the sampling density according to the robot speed and environmental complexity. Sparsely sample in a simple environment to improve computational efficiency, and densely sample in a complex environment to enhance accuracy. For example, reduce the sampling points in an open farmland and increase the sampling points in an area with dense obstacles to improve efficiency while ensuring the quality of the plan.

[0122] Collect images, point clouds, and sensor data of various obstacles in different terrain backgrounds, and carefully annotate the obstacle types and key features. This enables the model to learn rich features and accurately identify obstacles in different situations, such as mounds of different sizes and tools in different placement postures. Use deep learning algorithms to extract features, and then combine with support vector machine (SVM) or random forest for model training, giving full play to the advantages of each algorithm. CNN is good at processing image texture features, PointNet can extract 3D shape features of point clouds, and SVM and random forest each have advantages in classification, jointly improving the accuracy and robustness of obstacle classification. The trained model can output the obstacle classification results in real time, providing a basis for subsequent actions. For example, select different crossing strategies according to the classification result of the mound height, and decide to grasp or avoid after identifying the tool, enabling the robot to make intelligent decisions in a timely manner and adapt to the complex farmland scenario, which cannot be achieved by wheeled / crawler robots.

[0123] For mounds with a height less than 10 cm, by reasonably planning the movement trajectories of the leg joints and balance control, ensure that the crossing action is stable and accurate. The center of gravity is adjusted and the joints work together, and the PID controller is used to adjust in real time according to the sensor feedback to avoid losing balance and falling, ensuring the stable operation of the robot in the small mound environment. For mounds with a height greater than 10 cm and less than 50 cm, first adjust the body posture to lower the center of gravity, shorten the stride to increase stability, adjust the foot direction and force in real time according to the terrain changes, and use the arm swing to assist in balancing. These strategies meet the stability requirements when crossing large mounds, and at the same time ensure the reliability of each step by adjusting the moving speed, improving the crossing ability in complex terrains. For larger mounds with a height greater than 50 cm, start the path planning module to find a feasible path, comprehensively considering the mound and the surrounding terrain information, continuously monitor and re-plan during walking, can avoid colliding with the mound, and at the same time update the model according to the new information to ensure the path is reasonable and safe, reflecting the adaptability and flexibility in complex terrains.

[0124] When approaching the tool, make preparatory actions, adjust the movement trajectory of the robotic arm and the posture of the gripper or suction cup according to the tool type and size to ensure accurate grasping. After grasping, place the tool according to the terrain to avoid affecting walking, realizing the effective handling of such tools and improving the farmland operation efficiency. Identify and judge the tool information through a stereo vision camera or lidar, and select strategies such as crossing, bypassing, or moving away according to the width. This intelligent decision-making avoids the interference of the tool on the robot's walking and ensures the continuity and stability of the operation.

[0125] Example 4

[0126] This embodiment is a further optimization based on Embodiment 3. Specifically, the action execution method for picking in Step 7 is as follows:

[0127] An agricultural picking robot is an automated system that combines image processing, robot control, and artificial intelligence technologies, aiming to improve the harvesting efficiency and accuracy of crops. This system captures images of farmland through a camera, uses image recognition algorithms to locate the position and posture of crops, plans the motion trajectory of the robotic arm, and performs precise picking actions;

[0128] Camera imaging: It is achieved through a vision sensor. Let the coordinates of a point in space be (X, Y, Z), and the coordinates of its projection point on the image plane be , and the focal length be . Then the imaging model is expressed as:

[0129]

[0130] where Z is the distance from point P to the optical center of the camera;

[0131] Image preprocessing: To improve the accuracy of image recognition, the original image needs to be preprocessed, including operations such as denoising, white balance, and sharpening. Gaussian filters are used for denoising;

[0132] Feature extraction and object detection: Deep learning algorithms (such as convolutional neural networks) are used to extract features and detect objects from the preprocessed image. Let the input image be I. After passing through a series of convolutional layers and pooling layers, a feature map is obtained. The object detection network outputs the class probability and the bounding box coordinates of each object;

[0133] Robotic arm motion planning: The motion planning of the robotic arm is based on the inverse kinematics algorithm. This algorithm solves the angle values of each joint according to the target position and posture of the end effector of the robotic arm. During the motion of the robotic arm, its actual position and speed are monitored in real time through sensors, and compared with the planned trajectory. A PID controller is used to correct the error to ensure that the robotic arm can accurately follow the planned trajectory;

[0134] The end effector of the robotic arm is equipped with specialized picking tools, including flexible grippers or suction cups. The design of the picking tools should consider factors such as the size, shape, and maturity of the fruits. For example, for spherical fruits, arc-shaped grippers can be used; for soft fruits, suction cups can be used;

[0135] According to the information of the fruits, the action parameters of the picking tools are adjusted, including the opening and closing degree of the gripper and the adsorption force of the suction cup. Let the radius of the fruit be , and the maturity be (value range 0 - 1). Then the opening and closing degree of the gripper and the adsorption force of the suction cup are respectively expressed as: , , where and are the initial opening degree and the adsorption force respectively, and are proportionality coefficients;

[0136] Fruit transportation and storage: The picked fruits are conveyed to the storage container carried by the robot through the internal conveying device. The conveying device can adopt the form of a conveyor belt or a pipeline. The storage container should have appropriate ventilation and fresh-keeping measures to extend the storage time of the fruits.

[0137] In this embodiment, through visual sensor imaging and using a specific imaging model (such as the relationship between coordinates and focal length), the projection information of the crops in space on the image plane can be accurately obtained, providing accurate basic data for subsequent image analysis, helping to improve the accuracy of crop position and attitude recognition. Preprocessing operations such as denoising, white balance, and sharpening are performed on the original image. Among them, Gaussian filter is used for denoising, which can remove the noise interference in the image, improve the image quality, make the image features more obvious, and is beneficial to subsequent feature extraction and target detection; Deep learning algorithms such as convolutional neural networks are used for feature extraction and target detection, which can automatically learn the complex features in the image, accurately identify different types of crops and their positions, attitudes and other information, obtain feature maps through a series of convolutional layers and pooling layers, and output the class probability and bounding box coordinates of the target, enabling high-precision target detection and reducing misrecognition and missed recognition situations;

[0138] Based on the inverse kinematics algorithm, the angle values of each joint are solved according to the target position and attitude of the end effector of the robotic arm, which can ensure that the robotic arm moves precisely to the crop position along the planned trajectory and achieve precise picking. During the movement of the robotic arm, its actual position and speed are monitored in real time through sensors and compared with the planned trajectory. The PID controller is used to correct the error. This real-time feedback control mechanism can adjust the movement state of the robotic arm in time, overcome the influence of external factors (such as wind force, vibration of the robot itself, etc.) on the movement of the robotic arm, ensure that the robotic arm can accurately track the planned trajectory, and improve the accuracy and stability of the picking action;

[0139] Specialized picking tools are designed for different types of fruits. For example, arc-shaped jaws are used for spherical fruits, and suction cups are used for soft fruits. This improves the adaptability of the picking tools to fruits of different shapes and textures, reduces damage to fruits during the picking process, ensures the integrity and quality of the fruits. The motion parameters of the picking tools are adjusted according to factors such as the size, shape, and maturity of the fruits. For example, the opening and closing degree of the jaws and the adsorption force of the suction cups. By setting reasonable initial opening and closing degrees, adsorption forces, and proportionality coefficients, the picking force can be accurately controlled according to the actual situation, avoiding over-loose or over-tight picking, further improving the accuracy and efficiency of picking, and reducing damage to fruit trees at the same time;

[0140] The picked fruits are promptly conveyed to the storage container carried by the robot through an internal conveying device (such as in the form of a conveyor belt or pipeline), realizing the continuity of the picking process, avoiding damage and contamination caused by fruits falling to the ground, improving the efficiency of fruit collection. The storage container has appropriate ventilation and preservation measures, which can extend the storage time of fruits, maintain the freshness and quality of fruits, reduce the loss of fruit spoilage caused by improper storage, and increase the economic benefits after agricultural harvest.

[0141] Please refer to Figure 2 , the present invention also discloses an operation system of a humanoid agricultural robot based on path planning, including:

[0142] Humanoid robot body: It includes a head, a torso, two arms, and two legs. Each part is connected by joints, simulating the human movement structure. Motors and reducers are equipped at the joints to achieve the motion control of each joint;

[0143] Vision sensor: Installed on the robot's head or arms, used to collect image information of crops;

[0144] Lidar and depth camera: Installed at different positions of the robot, including the head and shoulders, used to obtain the three-dimensional structure information of the environment;

[0145] Inertial measurement unit: Installed at the center of gravity of the robot, used to real-time monitor the posture and motion state of the robot;

[0146] Auxiliary sensors: Include contact sensors and pressure sensors, used to sense the contact situation between the robot and the surrounding environment, and provide feedback information for the execution of picking actions;

[0147] Actuator module: It includes robotic arms and a leg walking mechanism. The humanoid robot is equipped with two robotic arms, each of which has multiple degrees of freedom to simulate the movement of a human arm. The end effector of the robotic arm can replace different types of tools according to different picking task requirements, including grippers and suction cups; the leg walking mechanism adopts a humanoid gait design, can walk stably on complex terrains, cross some obstacles. At the same time, the leg walking mechanism also has certain climbing and balancing abilities to adapt to different agricultural operation environments;

[0148] Control system: It includes a main controller and a motion controller. The main controller is responsible for the coordinated control and management of the entire robot system, and the motion controller is used to control the movement of the motors of each joint of the robot to achieve motion trajectory tracking;

[0149] Data processing unit: It is responsible for collecting and processing data from various sensors and converting it into a format that can be recognized and used by the robot control system;

[0150] Central processing unit: It is used to process data of computationally intensive tasks such as image recognition and path planning and various mathematical operations;

[0151] Communication module: It is used for data transmission and communication between the robot and external devices;

[0152] Power supply system: It uses a lithium battery as the power supply of the robot to provide power support for each component of the robot.

[0153] In summary: Through the collaborative work of multiple sensors, the present invention can obtain environmental information of the picking area from multiple dimensions. The vision sensor obtains the position, color, and shape information of the crops, which helps to accurately identify different types and maturity levels of crops; the three-dimensional structure information obtained by the lidar and depth camera can accurately construct a point cloud map of the environment, providing high-precision spatial data support for subsequent path planning and operation; the IMU real-time monitors the posture and motion state of the robot to ensure that the robot can operate stably in complex terrains and improve the accuracy of positioning and navigation; different types of sensors can adapt to various complex agricultural environments and operation scenarios. Whether it is in outdoor farmlands with changing lighting conditions or in relatively enclosed and occluded greenhouse environments, through the complementary advantages of each sensor, it can comprehensively and accurately collect environmental information, enabling the robot to better meet the operation requirements in different environments;

[0154] The comprehensive picking environment map generated by the present invention contains information on terrain, crop distribution, obstacle positions, etc., providing a comprehensive and accurate basis for the path planning, operation task allocation, and operation strategy formulation of the robot. Based on such a map, the robot can more intelligently plan the optimal path, avoid colliding with obstacles, efficiently complete the picking operation, improve the operation efficiency and quality. During the operation process, the sensor continuously collects environmental information, and the data processing system can update the picking environment map in real time to enable it to dynamically adapt to environmental changes;

[0155] The humanoid robot of the present invention can select different crossing strategies according to the classification results of the mound height, and decide to grab or avoid after tool recognition, enabling the robot to make intelligent decisions in a timely manner and adapt to the complex farmland scenarios.

[0156] Finally, it should be noted that the above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, for those skilled in the art, they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An operation method of a humanoid agricultural robot based on path planning, characterized in that, It includes the following steps: Step 1: Multi-sensor data collection: The humanoid agricultural robot is equipped with a variety of sensors for data collection, including visual sensors, lidar, depth cameras, and inertial measurement units; The humanoid robot body includes a head, a torso, two arms, and two legs. Step 2: Data processing and map generation: The data collected by different sensors is preprocessed and data fusion processed to generate a picking environment map. Step 3: Map update: Continuously collect new environmental data to update the map in real time. Step 4: Target crop image recognition and positioning: Use the convolutional neural network algorithm to recognize and classify the crop images collected by the visual sensor, and combine the map information to determine the position of the target crop. Step 5: Global path planning: Based on the constructed picking environment map and the target crop position information, use the A algorithm for global path planning. Step 6: Local path obstacle avoidance: Monitor the environmental information. When encountering sudden obstacles, adjust the movement direction and speed of the robot to avoid the obstacles; The local path obstacle avoidance also includes obstacle recognition and classification. The steps of obstacle recognition and classification are as follows: Data collection and annotation: Collect a number of images, point cloud data containing various obstacles in different terrain backgrounds, and corresponding sensor data, and mark the types and key features of the obstacles. Feature extraction and model training: Use deep learning algorithms to extract features from the collected data, use the extracted features as input, and use the support vector machine SVM for model training to classify different obstacles by the model. Classification result output: In actual applications, the trained model inputs the sensor data collected in real time into the model and outputs the classification results of the obstacles. The mounds are divided into mounds with a height h < 10 cm, 10 cm ≤ h ≤ 50 cm, and h ≥ 50 cm in units of height h; For hoe and shovel tools, identify their types, sizes, placement postures, and whether they are blocked by crops or soil. The local path obstacle avoidance also includes a dynamic crossing strategy. The steps of the dynamic crossing strategy for mounds are as follows: Balance control: During the crossing process, use the inertial measurement unit to monitor the body attitude angular velocity and acceleration information of the robot in real time, combine the foot pressure changes feedback by the pressure sensor, and adjust the centroid position of the robot through the PID controller. Body attitude adjustment: The robot first lowers its body center of gravity, which is achieved by adjusting the angles of the leg joints, so that the center of gravity of the body moves down and is close to the support area of the feet. Obstacle avoidance and replanning: During the process of walking along the planned path, the robot continuously monitors the surrounding terrain changes and the situation of other obstacles. If new obstacles are found or the original path is infeasible, the robot will immediately stop the current action and re-run the path planning algorithm to find a new feasible path. The steps of the dynamic crossing strategy for hoe and shovel tools are as follows: Grasping and moving: When the robot is 0.5 meters away from the tool, it adjusts the movement trajectory of the robotic arm and the position of the gripper or suction cup according to the type and size of the tool. After grasping the tool, the robotic arm lifts it up and lifts it off the ground. The robot then places the tool in an open area based on the terrain. Crossing decision: If the tool is less than 0.5 meters wide, the robot chooses to cross, adopting a strategy of crossing over the dirt pile. If the tool is larger than 0.5 meters, the robot stops and assesses the surrounding terrain and environmental conditions. Depending on the situation, it chooses to bypass the tool or use the robotic arm to move it to a safe place before continuing forward. Step 7: Execute the picking action: Based on the position and posture of the target crop, plan the motion trajectory of the robotic arm, use the inverse kinematics algorithm to solve the angle values of each joint of the robotic arm, adjust the motion parameters of the picking tool, and realize the picking of the fruit; the picking action is executed as follows: Camera imaging: Using visual sensors to capture images of crops; Image preprocessing: preprocess the original image; Feature extraction and target detection: Use deep learning algorithms to perform feature extraction and target detection on preprocessed images; Robotic arm motion planning: The robot arm's motion planning is based on an inverse kinematics algorithm. This algorithm calculates the angle values of each joint based on the target position and posture of the robot arm's end effector. The actual position and speed are monitored by sensors and compared with the planned trajectory. A PID controller is used to correct errors. The motion parameters of the picking tool are adjusted based on the fruit information. The robot arm's end effector is equipped with a picking tool, including a flexible gripper or suction cup. The design of the picking tool should take into account the size, shape, and maturity of the fruit. Adjust the action parameters of the picking tool according to the information of the fruit, including the opening and closing degree of the gripper and the adsorption force of the suction cup. Let the radius of the fruit be r and the maturity be m, where the value range of m is 0-1. Then the opening and closing degree d of the gripper and the adsorption force F of the suction cup are respectively expressed as: d = d0 + k d rm, F = F0 + k f rm, where d0 and F0 are the initial opening and closing degree and adsorption force respectively, and k d 、k f are proportionality coefficients.

2. The operation method of a humanoid agricultural robot based on path planning according to claim 1, characterized in that: The data processing in step 2 and the data fusion processing in map generation are as follows: Coordinate system conversion: Fuse the data of different sensors into a unified coordinate system and perform coordinate system conversion on the data of each sensor; Fusion of visual and lidar data: The point cloud data acquired by the lidar is projected onto the image plane and registered with the point cloud data using the iterative closest point algorithm using the feature points of the image. The color information in the visual image is combined with the 3D coordinate information in the point cloud data to construct a fused map. Each pixel in the fused map contains both color information and corresponding 3D coordinate information. Fusion of visual, lidar, and depth camera data: The depth image acquired by the depth camera is fused with the point cloud data from the lidar. Through weighted fusion, different weight coefficients are assigned based on the accuracy of the depth camera and lidar in different areas, and then the weighted average depth value is calculated. The semantic segmentation results and depth information in the visual image are combined with the map content to mark the boundaries and positions of objects of different categories in the fused map. Inertial measurement unit data fusion is added: the posture information measured by the inertial measurement unit is used to correct the posture of the feature points in the fusion map. The coordinates of the feature points are converted into coordinates in the global coordinate system through the coordinate transformation matrix, and then adjusted according to the posture information of the inertial measurement unit; combined with the motion state information of the inertial measurement unit, the motion trajectory of the robot in the picking environment is optimized.

3. The operation method of a humanoid agricultural robot based on path planning according to claim 1, characterized in that: The map update method in Step 3 is as follows: New data acquisition: The sensor acquires new environmental data at a set frequency and preprocesses the data; Change detection: Compare the newly acquired data with the existing map data to detect whether the environment has changed. Change detection is achieved by calculating the difference in the data; Local update: If it is detected that the environment has changed, update the changed local area; Global update: Every 1 hour, globally update the entire map to construct a new fused map.

4. The operation method of a humanoid agricultural robot based on path planning according to claim 1, characterized in that: The steps for target crop image recognition and localization in Step 4 are as follows: Training process: Using the method of supervised learning, input several images and their labels into the model. Calculate the predicted output of the model through forward propagation, and calculate the loss function according to the difference between the predicted output and the true label; Through the backpropagation algorithm, adjust the parameters of the model according to the loss function to reduce the value of the loss function; Recognition process: During the picking operation, input the real-time acquired crop image into the trained CNN model. The model calculates the probability distribution of the input image belonging to various crops through forward propagation, and determines the crop category to which the input image belongs according to the maximum value in the probability distribution, so as to obtain the recognition result of the target crop; The method for obtaining the position information of the target in the image is: After obtaining the recognition result, by introducing a region proposal network in the CNN model, generate region proposals containing the target crop in the image, and assign a score to each proposal, and then screen and adjust the proposals according to the scores to obtain the target position box; Determine the position of the target in the robot coordinate system: Obtain the relative pose relationship between the vision sensor and the robot through sensor calibration. The purpose of sensor calibration is to determine the internal parameters and external parameters of the vision sensor. The sensor calibration method is the checkerboard calibration method to solve the internal and external parameters of the camera; Determine the target position: At the pixel coordinates in the image, solve the coordinates of the target in the camera coordinate system, and then according to the relative pose relationship between the camera and the robot, convert the coordinates of the target in the camera coordinate system to the robot coordinate system to determine the position of the target crop in the robot coordinate system.

5. The operation method of a humanoid agricultural robot based on path planning according to claim 1, characterized in that: The global path planning method in Step 5 is as follows: S1: Construct the search space: In the picking environment map, define the starting position of the robot, the position of the target crop, and the intermediate points as nodes. Each node is represented by its coordinates (x, y) in the map, and the line segment connecting two adjacent nodes is called an arc; S2: Create an open list and a closed list: Sort in ascending order according to the value of the evaluation function f(n). Initially, put the starting node S into the open list; During the execution of the algorithm, once a node is expanded, transfer it from the open list to the closed list; Put the starting node S into the open list, set g(S)=0 for the starting node, and h(S) is calculated according to the estimated distance from the starting node to the target node; S3: Select the node n with the minimum value of the evaluation function f(n) from the open list as the current expansion node. If the current expansion node n is the target node G, the optimal path is found and the algorithm ends. At this time, trace back from the target node step by step along the parent node pointer to the starting node to obtain the complete path. Otherwise, transfer the current expanded node n from the open list to the closed list; for each adjacent node n of the current expanded node n i , if n i is already in the closed list, skip this node; if n i is not in the open list, add it to the open list, and calculate its evaluation function f(n i ) = g(n i ) + h(n i ), and at the same time set its parent node pointer to point to the current node n. If n i is already in the open list, compare the new path cost with the original path cost, and loop until the open list is empty or the target node is found; S4: Path backtracking: If the target node is found, start from the target node and return to the previous node in sequence according to the parent node pointer until returning to the starting node to obtain the optimal path.

6. The operation method of a humanoid agricultural robot based on path planning according to claim 1, characterized in that: The steps of local path obstacle avoidance in Step Six are as follows: State - space representation: The state of the robot is represented by a vector where \((x,y)\) are the position coordinates of the robot in the two - dimensional plane, \(\theta\) is the orientation angle of the robot, \((v x ,v y ) are the velocity components of the robot in the \(x\) and \(y\) directions, and \(v θ \) is the angular velocity of the robot; Sampling: At each moment, according to the current state x of the robot t and the control input set, multiple next states are generated, and the sampling density is adjusted according to factors such as the speed of the robot and the environmental complexity; Prediction: According to the kinematic and dynamic models of the robot, predict the new states of each sampled state after a period of time. Collision detection: For each predicted state, check whether the robot collides with obstacles. Replace the obstacles with geometric shapes and judge whether there is a collision by calculating the distance between the robot and the obstacles. If the distance between the predicted state and any obstacle is less than the safety threshold, it is considered that a collision will occur.

7. The system of an operation method of a humanoid agricultural robot based on path planning according to claim 1, wherein: Including: Humanoid robot body: It includes a head, a torso, two arms, and two legs. Each part is connected by joints, and motors and reducers are equipped at the joints. Vision sensor: Installed on the robot's head or arms, used to collect image information of crops. Lidar and depth camera: Installed at different positions of the robot, including the head and shoulders, used to obtain the three-dimensional structure information of the environment. Inertial measurement unit: Installed at the center of gravity of the robot, used to monitor the attitude and motion state of the robot in real time. Auxiliary sensors: including contact sensors and pressure sensors, used to sense the contact situation between the robot and the surrounding environment and provide feedback information for the execution of picking actions. Actuator module: including robotic arms and leg walking mechanisms. The humanoid robot is equipped with two robotic arms, and the end effectors of the robotic arms replace different types of tools according to different picking task requirements, including grippers and suction cups. Control system: including a main controller and a motion controller. The main controller is responsible for the coordinated control and management of the entire robot system, and the motion controller is used to control the motion of the motors of each joint of the robot to achieve motion trajectory tracking. Data processing unit: responsible for collecting and processing data from various sensors and converting it into a format that can be recognized and used by the robot control system. Central processing unit: used to process data of computationally intensive tasks such as image recognition and path planning and various mathematical operations.

Images