Operation method and system of humanoid agricultural robot based on path planning

Abstract

The invention discloses an operation method of a humanoid agricultural robot based on path planning, which relates to the related technical field of agricultural robots and comprises the steps of multi-sensor data acquisition, data processing and map generation, map updating, target crop image recognition and positioning, global path planning, local path obstacle avoidance and picking action execution. The invention further discloses an operation system of the humanoid agricultural robot based on path planning. The operation system comprises a humanoid robot body, a visual sensor, a laser radar, a depth camera, an inertial measurement unit, an auxiliary sensor, an actuator module, a control system, a communication module and a power supply system. The inventor-shaped robot can recognize the positions and shapes of the tools in advance by using a sensitive visual perception system of the inventor-shaped robot, can skillfully cross over or bypass the tools by virtue of the flexibility of the inventor-shaped robot, can accurately determine the positions and distribution conditions of crops in a three-dimensional space, and generates a comprehensive picking environment map.

Description

Technical Field

[0001] The present invention relates to the technical field related to agricultural robots, and in particular to an operation method and system of a humanoid agricultural robot based on path planning. Background Art

[0002] With the continuous development of agricultural modernization, the automation and intelligence of agricultural picking operations have become the key direction to improve agricultural production efficiency and reduce labor costs. In traditional agricultural picking, manual operation is mainly relied on, but this method has many limitations. In order to overcome the shortcomings of manual picking, various agricultural picking machines have emerged. Among them, vehicles and crawler robots have been used in the field of agricultural picking. Vehicle robots usually have faster moving speeds and larger load capacities, and are suitable for operations in large areas of farmland. They can harvest and collect crops by carrying various picking devices.

[0003] However, vehicle-type robots have obvious deficiencies in adaptability to complex terrain. In farmland, tools such as hoes and shovels may be placed randomly, and when there are raised piles of soil on the ground, this will pose an obstacle to the robot's movement, making it difficult for the robot to cross some obstacles and perform flexible operations.

[0004] Moreover, the existing path planning methods of agricultural robots have certain limitations to varying degrees. Although some global path planning algorithms can find the optimal path from the starting point to the end point, in actual applications, due to the inaccuracy of the environmental model and lack of real-time performance, the planned path may be difficult to execute in complex environments.

[0005] Therefore, it is necessary to propose an operation method and system of a humanoid agricultural robot based on path planning to solve the above problems. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention provides an operating method and system of a humanoid agricultural robot based on path planning, which solves the problems that existing vehicle-type robots have shortcomings in adaptability to complex terrains, difficulty in crossing some obstacles for flexible operations, and difficulty in executing planned paths in complex environments due to inaccuracy and lack of real-time performance of environmental models.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: A method for operating a humanoid agricultural robot based on path planning comprises the following steps: Step 1: Multi-sensor data collection: The humanoid agricultural robot is equipped with a variety of sensors, including visual sensors, lidar, depth cameras, and inertial measurement units; Step 2: Data processing and map generation: pre-process and fuse the data collected by different sensors to generate a picking environment map; Step 3: Map update: Continuously collect new environmental data and update the map in real time; Step 4: Target crop image recognition and positioning: Use the convolutional neural network algorithm to recognize and classify the crop images collected by the visual sensor, and combine the map information to determine the location of the target crop; Step 5: Global path planning: Based on the constructed picking environment map and target crop location information, the A algorithm is used for global path planning; Step 6: Local path obstacle avoidance: Monitor environmental information, and when encountering sudden obstacles, adjust the robot's movement direction and speed to avoid the obstacles; Step 7: Execution of picking action: According to the position and posture of the target crops, the motion trajectory of the robotic arm is planned, the inverse kinematics algorithm is used to solve the angle values ​​of each joint of the robotic arm, and the motion parameters of the picking tool are adjusted to realize the picking of fruits.

[0008] Optionally, the target crop image recognition and positioning steps in step 4 are as follows: Training process: Using supervised learning, several images and their labels are input into the model, the model's predicted output is calculated through forward propagation, and the loss function is calculated based on the difference between the predicted output and the true label; through the back-propagation algorithm, the model parameters are adjusted according to the loss function to reduce the value of the loss function; Identification process: During the harvesting operation, the crop images collected in real time are input into the trained CNN model. The model calculates the probability distribution of the input image belonging to various types of crops through forward propagation, and determines the crop category to which the input image belongs based on the maximum value in the probability distribution, thereby obtaining the recognition result of the target crop; The method of obtaining the location information of the target in the image is as follows: after obtaining the recognition result, the region proposal network is introduced into the CNN model to generate region proposals containing the target crop in the image, and a score is assigned to each proposal. Then, these proposals are screened and adjusted according to the score to obtain the target location box; Determine the position of the target in the robot coordinate system: obtain the relative position and posture relationship between the visual sensor and the robot through sensor calibration. The purpose of sensor calibration is to determine the internal and external parameters of the visual sensor. The sensor calibration method is the checkerboard calibration method to solve the internal and external parameters of the camera; Determine the target position: Use the pixel coordinates in the image to solve the coordinates of the target in the camera coordinate system, and then convert the coordinates of the target in the camera coordinate system to the robot coordinate system based on the relative posture relationship between the camera and the robot to determine the position of the target crop in the robot coordinate system.

[0009] An operation system of a humanoid agricultural robot based on path planning, comprising: The humanoid robot body: includes the head, torso, arms and legs, each part is connected by joints, and the joints are equipped with motors and reducers; Visual sensor: installed on the robot head or arms to collect image information of crops; LiDAR and depth camera: installed at different locations on the robot, including the head and shoulders, to obtain three-dimensional structural information of the environment; Inertial measurement unit: installed at the center of gravity of the robot, used to monitor the robot's posture and motion status in real time; Auxiliary sensors: including contact sensors and pressure sensors, which are used to sense the contact between the robot and the surrounding environment and provide feedback information for the execution of picking actions; Actuator module: including robotic arms and leg walking mechanisms. The humanoid robot is equipped with two robotic arms. The end effector of the robotic arms can replace different types of tools, including grippers and suction cups, according to different picking task requirements. Control system: including main controller and motion controller. The main controller is responsible for the coordination, control and management of the entire robot system. The motion controller is used to control the movement of the motors of each joint of the robot to achieve motion trajectory tracking. Data processing unit: responsible for collecting and processing data from various sensors and converting them into a format that can be recognized and used by the robot control system; Central Processing Unit: Used to process data and various mathematical operations for computationally intensive tasks such as image recognition and path planning.

[0010] The present invention provides an operation method and system of a humanoid agricultural robot based on path planning, which has the following beneficial effects: 1. The humanoid robot of the present invention can use its keen visual perception system to identify the position and shape of tools in advance. It can rely on its own flexibility to cleverly step over or bypass these tools to avoid being tripped or damaging the tools. In addition, the humanoid robot can easily cross the small mounds in the field with its flexible leg structure, so that the humanoid robot can maintain efficient mobility in complex farmland terrain. During the movement, the humanoid robot can also dynamically adjust the path according to the changing environment. If a new small mound suddenly appears or someone moves the position of the hoe or shovel, it can respond quickly. By quickly processing the new environmental information, the humanoid robot can re-plan the path to ensure that it can always reach the target location smoothly and reduce the interference caused by terrain changes.

[0011] 2. The present invention combines the crop information obtained by visual sensors with the spatial information obtained by lidar and depth cameras, which can more accurately determine the position and distribution of crops in three-dimensional space. The generated comprehensive picking environment map contains information on topography, crop distribution, obstacle location, etc., which provides a comprehensive and accurate basis for the robot's path planning, task allocation and operation strategy formulation. Based on such a map, the robot can plan the optimal path more intelligently, avoid collisions with obstacles, efficiently complete the picking operation, and improve work efficiency and quality.

[0012] 3. The present invention designs special picking tools for different types of fruits, and adopts suction cups for soft fruits, which improves the adaptability of the picking tools to fruits of different shapes and textures, reduces the damage to the fruits during the picking process, ensures the integrity and quality of the fruits, and adjusts the action parameters of the picking tools according to factors such as the size, shape and maturity of the fruits. By setting a reasonable initial opening and closing degree, adsorption force and proportional coefficient, the picking force can be accurately controlled according to actual conditions to avoid picking too loosely or too tightly, thereby further improving the accuracy and efficiency of picking and reducing damage to fruit trees. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A schematic diagram of a flow chart of an operation method of a humanoid agricultural robot of the present invention; Figure 2 The figure is a schematic diagram of the modules of the operation system of the humanoid agricultural robot of the present invention. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Example 1

[0015] Please refer to FIG1 , a method for operating a humanoid agricultural robot based on path planning, comprising the following steps: Step 1: Multi-sensor data collection: The humanoid agricultural robot is equipped with a variety of sensors, including visual sensors, laser radar, depth cameras, and inertial measurement units (IMUs). Before entering the picking area, each sensor is activated to conduct a comprehensive scan and data collection of the surrounding environment. The visual sensor is used to obtain the position, color, and shape information of the crops; the laser radar and depth camera are used to obtain the three-dimensional structure information of the environment, which is the basis for building a high-precision point cloud map; the IMU is used to monitor the robot's posture and motion status in real time, and provide data support for positioning and navigation; Step 2: Data processing and map generation: pre-process the data collected by different sensors, and perform data fusion processing to generate a harvesting environment map containing comprehensive information on topography, crop distribution, and obstacle locations; Step 3: Map update: During the robot operation, new environmental data is continuously collected and the map is updated in real time to adapt to environmental changes and ensure the accuracy of path planning; Step 4: Target crop image recognition and positioning: Use the convolutional neural network (CNN) algorithm in deep learning to recognize and classify the crop images collected by the visual sensor. Train the CNN model through a number of crop sample images so that it can accurately identify crops of different types and different growth stages. During the picking operation, input the real-time crop images collected into the trained CNN model to identify the target crops and obtain their location information; according to the relative position relationship between the visual sensor and the robot body, convert the pixel coordinates of the target crops in the image into actual coordinates in the robot workspace, and combine the map information to determine the position of the target crop in the picking area, providing the target point for the robot's path planning and picking action; Step 5: Global path planning: Based on the constructed picking environment map and the target crop location information, the A algorithm is used for global path planning. The A algorithm comprehensively considers the length and cost of the path, and can ensure that the optimal path is found while taking into account the robot's motion energy consumption. During the planning process, the obstacle information in the map is used as a constraint to prevent the robot from colliding with obstacles. Through global path planning, the initial path from the robot's starting position to the location of the target crop is determined; Step 6: Local path obstacle avoidance: When the robot walks along the global path, it monitors the surrounding environment in real time. When encountering sudden obstacles, it starts local path planning and uses the dynamic window method (DWA) to perform local path planning. According to the robot's current speed, acceleration, angular velocity, and the position and speed information of surrounding obstacles, the robot's safe motion trajectory is calculated in real time. By continuously adjusting the robot's motion direction and speed, the robot can avoid obstacles while returning to the global path and continue to move towards the target crops. Step 7: Execution of picking action: When the robot reaches the vicinity of the target crop, it plans the motion trajectory of the robotic arm according to the position and posture of the target crop, and uses the inverse kinematics algorithm to solve the angle value of each joint of the robotic arm, so that the end effector of the robotic arm approaches the target crop. During the movement, the motion trajectory of the robotic arm is adjusted through real-time feedback control to ensure that it can complete the picking action smoothly and accurately. The end effector of the robotic arm is equipped with a special picking tool, such as a flexible gripper or a suction cup. When the end effector of the robotic arm approaches the fruit of the target crop, the action parameters of the picking tool are adjusted according to the size, shape and maturity information of the fruit to achieve the picking of the fruit. The picked fruit is transported to the storage container of the robot through the internal conveying device, completing a picking operation; Step 8: Cyclic operation: After completing a picking operation, the robot automatically searches for the next target crop based on the preset task list and the current completed picking status.

[0016] Visual sensors (such as cameras) acquire visual information by capturing images of the harvesting area. Cameras usually capture color or grayscale images at a certain resolution (such as 1920×1080 pixels) and frame rate (such as 30 frames per second). During the acquisition process, the lens captures the image formed by the reflection of light, in which each pixel contains color (for color cameras) or brightness (for grayscale cameras) information, which reflects the surface characteristics of crops, obstacles, and terrain. LiDAR measures distance by emitting laser beams and receiving reflected light. It emits a series of fast-pulsed laser beams to the surrounding environment. When the laser beam encounters an obstacle, it will be reflected back. The receiver calculates the distance based on the time difference between the emission and reception of the laser. Through rotation and pitch movement, LiDAR scans in two-dimensional or three-dimensional space to obtain point cloud data of the surrounding environment. Each point in the point cloud data contains the three-dimensional coordinate information of the point. ,in , Indicates the horizontal position. Indicates height information; LiDAR can provide high-precision distance measurement, accurately describe the outline and position of objects in the environment, and has a wide measurement range; Depth cameras use infrared light or structured light technology to obtain depth information of objects. Infrared-based depth cameras calculate the distance between the object and the camera by emitting infrared light and receiving reflected light. Structured light depth cameras project known light patterns (such as stripes and spots) and determine the depth of the object surface by analyzing the deformation of the light pattern. The data output by the depth camera is presented in the form of a depth image, and the grayscale value of each pixel represents the distance from the point to the camera. Depth cameras can provide relatively accurate depth information in indoor environments and can also measure the distance of objects with unclear textures. The inertial measurement unit (IMU) is a device that integrates accelerometers, gyroscopes, and magnetometer sensors. The accelerometer is used to measure the robot's linear acceleration in all directions, the gyroscope is used to measure the robot's angular velocity, and the magnetometer measures the direction of the geomagnetic field to determine the robot's orientation. Through the combination of these sensors, the IMU monitors the robot's posture (including roll, pitch, and yaw angles) and motion state (such as acceleration, angular velocity) in real time. IMU data has the characteristics of high frequency and low noise, and can provide accurate posture and motion information, which helps the robot maintain balance and determine its own position in a dynamic environment.

[0017] The data processing in step 2 and the data preprocessing in map generation are as follows: Visual sensor data preprocessing: Use filtering algorithms (such as Gaussian filtering and median filtering) to remove noise points in the image. Gaussian filtering performs weighted averaging operations on the convolution kernel and image pixels to smooth the image and reduce the impact of noise on subsequent analysis. Since uneven lighting may cause image color distortion, color correction is required. The grayscale histogram of the image is adjusted through histogram equalization to make the grayscale distribution of pixels uniform and enhance the contrast of the image. Use image segmentation algorithms (such as threshold segmentation and region growing) to separate crops, obstacles and background in the image. The image segmentation method based on threshold segmentation compares the grayscale value of the pixel with the set threshold to divide the pixel into different categories. LiDAR data preprocessing: Use statistical filtering (such as mean filtering and statistical outlier removal) to remove noise points in point cloud data; In order to reduce the amount of data and improve processing efficiency, downsample the point cloud data. The downsampling method is voxel grid downsampling. Voxel grid downsampling divides the point cloud space into regular three-dimensional grids (voxels) and replaces the points in the voxel with the average value of the voxel center point; Depth camera data preprocessing: Use bilateral filtering to filter the depth image and remove noise and outliers in the depth data. Bilateral filtering not only considers the spatial position relationship of pixels, but also the depth value difference of pixels. It can smooth the depth image while retaining edge information. Due to object occlusion or sensor measurement limitations, there may be holes in the depth image. The holes are filled by interpolation algorithms (such as nearest neighbor interpolation and linear interpolation). The nearest neighbor interpolation algorithm finds the depth value of the non-hole point closest to the hole point as the depth value of the hole point; IMU data preprocessing: Use low-pass filtering to remove high-frequency noise from IMU data. Low-pass filtering sets a cutoff frequency to allow signals below the cutoff frequency to pass through and prevent signals above the cutoff frequency from passing through, thereby smoothing the IMU data. Since the data acquisition frequencies of different sensors may be different, the IMU data is time-synchronized with other sensor data to ensure data consistency. Data synchronization is achieved through hardware clock synchronization or software algorithms (such as timestamp matching and linear interpolation).

[0018] The data processing in step 2 and the data fusion processing in map generation are as follows: Coordinate system conversion: The data of different sensors are integrated into a unified coordinate system, and the coordinate system of each sensor data is converted. The coordinate systems of the visual sensor, lidar, depth camera and IMU are , , and , the unified target coordinate system is , through the coordinate transformation matrix R and the translation vector , point From the coordinate system Convert to coordinate system : ,in, and From the coordinate system To coordinate system The rotation matrix and translation vector of the coordinate system conversion between multiple sensors are realized through cascade transformation; Fusion of visual and lidar data: Project the point cloud data acquired by the lidar onto the image plane, and use the feature points of the image (such as corner points and edge points) to align with the point cloud data. The alignment method is the iterative closest point algorithm (ICP). The iterative closest point algorithm solves the optimal coordinate transformation parameters (rotation matrix and translation vector) by minimizing the sum of square distances between corresponding points of the point cloud and the image feature points. After the alignment, combine the color information in the visual image with the three-dimensional coordinate information in the point cloud data to construct a fusion map. Each pixel in the fusion map contains color information and corresponding three-dimensional coordinate information, which can more comprehensively reflect the characteristics of the picking environment. Fusion of vision, lidar and depth camera data: The depth image acquired by the depth camera is fused with the point cloud data of the lidar to improve the accuracy and reliability of the depth information. Through weighted fusion, different weight coefficients are assigned to the depth camera and lidar according to their accuracy in different areas, and then the weighted average depth value is calculated. The semantic segmentation results in the visual image (classifying the pixels in the image into different categories such as crops and obstacles) are combined with the depth information to further fuse the content of the map, mark the boundaries and positions of objects of different categories in the fused map, and provide more accurate information for subsequent path planning and decision-making; Add inertial measurement unit data fusion: Use the posture information measured by the inertial measurement unit to correct the posture of feature points in the fusion map (such as the center position of crops and the edge of obstacles), convert the coordinates of the feature points into coordinates in the global coordinate system through the coordinate transformation matrix, and adjust them according to the posture information of the inertial measurement unit; combine the motion state information (acceleration, angular velocity) of the inertial measurement unit to optimize the motion trajectory of the robot in the picking environment, and adjust the robot's speed planning according to the acceleration information measured by the IMU to improve the picking efficiency while ensuring safety.

[0019] The map update method in step 3 is: New data collection: The sensor collects new environmental data at a set frequency and pre-processes the data; Change detection: Compare the newly collected data with the existing map data to detect whether the environment has changed. Change detection is achieved by calculating the difference in data (such as the change in the gray value of the pixel point and the change in the density of the point cloud data); Local update: If changes in the environment are detected, the local area of ​​the change is updated. Local update can reduce the amount of calculation and improve update efficiency. For example, when it is detected that crops in a certain area have been picked, only the map information of that area is updated (such as deleting the corresponding crop model); Global update: The entire map is updated globally every hour. The global update can reintegrate all sensor data and build a new fused map. The update algorithm is Kalman filtering: Kalman filtering is a recursive state estimation algorithm used for map updating. It takes the state variables of the map (such as the position and color of feature points) as the system state and the sensor measurement data as the observation value. It estimates the current state of the map through two steps: prediction and update. In the prediction step, the map state at the current moment is predicted based on the robot's motion model and the map state at the previous moment; in the update step, the predicted map state is corrected using the sensor measurement data to obtain the updated map state.

[0020] The global path planning method in step 5 is: The A algorithm is a centralized search strategy that combines the advantages of best-first search and Dijkstra's algorithm by evaluating the function To select the next expansion node, where represents the actual cost from the starting node to node n (such as path length, energy consumption), represents the estimated cost from node n to the target node (heuristic function); S1: Constructing the search space: In the picking environment map, the robot's starting position, the target crop position, and some key intermediate points (such as turning points and intersections) are defined as nodes. Each node is represented by its coordinates in the map. Indicates, for example, the starting node , target node ; The line segment connecting two adjacent nodes is called an arc. The cost of the arc is determined by the distance between the nodes, the complexity of the terrain (such as flat terrain, rugged terrain), and energy consumption factors. The determination method is based on the Euclidean distance calculation: in, Represents a slave node To Node The cost of the arc; S2: Create an open list and a closed list: The open list is used to store the nodes to be expanded, sorted from small to large according to the value of the evaluation function f(n). Initially, the starting node S is placed in the open list; the closed list is used to store the nodes that have been expanded. During the execution of the algorithm, once a node is expanded, it is transferred from the open list to the closed list; the starting node S is placed in the open list, and the starting node is set , Calculated based on the estimated distance from the starting node to the target node (using Euclidean distance); S3: Select an evaluation function from an open list The node n with the smallest value is used as the current expansion node. If the current expansion node n is the target node G, the optimal path is found and the algorithm ends. At this time, the parent node pointer is traced back from the target node to the starting node to obtain a complete path; otherwise, the current expansion node n is transferred from the open list to the closed list; for each adjacent node of the current expansion node n (i.e., nodes directly connected to n), if Already in the closed list, skip the node; if If it is not in the open list, add it to the open list and calculate its evaluation function At the same time, set its parent node pointer to point to the current node n. If Already in the open list, compare the new path cost with the original path cost, if the new path cost is smaller, update The evaluation function value and parent node pointer are looped until the open list is empty or the target node is found; S4: Path backtracking: If the target node is found, start from the target node and return to the previous node in sequence according to the parent node pointer until you return to the starting node and get the optimal path.

[0021] In this embodiment, the coordinated work of multiple sensors can obtain environmental information of the picking area from multiple dimensions. The visual sensor obtains the position, color, and shape information of the crops, which helps to accurately identify crops of different types and maturity. The three-dimensional structural information obtained by the laser radar and the depth camera can accurately construct a point cloud map of the environment, providing high-precision spatial data support for subsequent path planning and operation operations. The IMU monitors the posture and motion state of the robot in real time to ensure that the robot can operate stably in complex terrain and improve the accuracy of positioning and navigation. Different types of sensors can adapt to various complex agricultural environments and operation scenarios. Whether in outdoor farmland with changing lighting conditions or in a relatively closed and obstructed greenhouse environment, the advantages of each sensor can be complementary to each other to comprehensively and accurately collect environmental information, so that the robot can better cope with the operation requirements in different environments. Preprocessing and fusion processing of data collected by different sensors can integrate scattered and heterogeneous data, eliminate redundancy and contradictions between data, and extract more valuable information. For example, the crop information obtained by the visual sensor is combined with the spatial information obtained by the laser radar and the depth camera to more accurately determine the position and distribution of crops in three-dimensional space. The generated comprehensive picking environment map contains information on topography, crop distribution, and obstacle locations, providing a comprehensive and accurate basis for the robot's path planning, task allocation, and operation strategy formulation. Based on such a map, the robot can plan the optimal path more intelligently, avoid collisions with obstacles, and efficiently complete picking operations, thereby improving work efficiency and quality. During the operation, sensors continuously collect environmental information, and the data processing system can update the picking environment map in real time, enabling it to dynamically adapt to environmental changes. For example, when crops change position or new obstacles appear during their growth, the robot can adjust path planning and operation strategies in a timely manner to ensure the smooth progress of the operation.

[0022] Example 2 This embodiment is based on the embodiment 1 and is optimized as follows. Specifically, the architecture of the convolutional neural network (CNN) algorithm in the target crop image recognition and positioning in step 4 includes: Input layer: Receives crop image data that has been preprocessed (such as normalization). The image is a color image with a certain resolution (such as 640×480 pixels). Each pixel contains information of three RGB channels, representing the intensity values ​​of the three color components of red, green, and blue, with a value range of 0-255. The pixel value is mapped to the interval of 0-1 to accelerate model training and improve numerical stability. Convolutional layer: The convolutional layer is the core component of CNN. It extracts local features of the image by sliding the convolution kernel (also called filter) on the image. For example, for a 3×3 convolution kernel, it will perform a convolution operation on a 3×3 area of ​​the image and calculate a feature value. The parameters (weights) of the convolution kernel are continuously updated during the training process to learn the pattern that best represents the image features. Pooling layer (downsampling layer): The pooling layer usually follows the convolution layer. Its main function is to reduce the dimension of the output feature map of the convolution layer, reduce the amount of data, and retain important information. The pooling operation is maximum pooling and average pooling. Fully connected layer: After being processed by multiple convolutional layers and pooling layers, the feature map is flattened into a one-dimensional vector and input into the fully connected layer. Each neuron in the fully connected layer is connected to all neurons in the previous layer, and the previously extracted features are weighted summed and nonlinearly transformed (through activation functions). Output layer: Each neuron in the output layer represents a crop category. The output vector is converted into a probability distribution through the Softmax function, which indicates the probability that the input image belongs to each crop category. For example, for a recognition task of three types of crops (such as apples, oranges, and bananas), the output vector is processed by the Softmax function to obtain a probability vector of length 3. ,in , , denote the probability that the input image belongs to an apple, an orange, and a banana, respectively, and , the expression of Softmax function is: ,in: represents the probability that the input image belongs to the i-th type of crop; is the output value of the output layer neuron; is the number of crop categories; The steps for target crop image recognition and positioning in step 4 are as follows: Training process: In order for the CNN model to accurately identify crops of different types and different growth stages, a large number of crop sample images are needed to train the model. The supervised learning method is used, that is, for each training image, there is a corresponding label (indicating the category of the crop). During training, several images and their labels are input into the CNN model, and the predicted output of the model is calculated through forward propagation. Then, the loss function (such as the cross entropy loss function) is calculated based on the difference between the predicted output and the true label; through the back-propagation algorithm, the parameters of the model (including the weights of the convolution kernel, the weights of the fully connected layer, and the bias term) are adjusted according to the loss function to reduce the value of the loss function. The core of the back-propagation algorithm is to calculate the gradient of the loss function to the model parameters, and then update the parameters according to the principle of gradient descent; Identification process: During the harvesting operation, the crop images collected in real time are input into the trained CNN model. The model calculates the probability distribution of the input image belonging to each type of crop through forward propagation, and determines the crop category to which the input image belongs based on the maximum value in the probability distribution, thereby obtaining the recognition result of the target crop. For example, if the probability distribution output by the model is [0.1, 0.8, 0.1], it means that the image has the highest probability of belonging to the second type of crop (such as orange), so the recognition result is orange; The method of obtaining the location information of the target in the image is as follows: after obtaining the recognition result of the target crop, it is necessary to further determine the precise location of the target in the image. By introducing the region proposal network (RPN) into the CNN model, the RPN generates a series of region proposals that may contain the target crop in the image, and assigns a score to each proposal, indicating the possibility that the proposal contains the target. Then, these proposals are screened and adjusted according to the score to obtain the target location box; Determine the position of the target in the robot coordinate system: In order to convert the position of the target crop from the image coordinate system to the robot coordinate system, the relative position relationship between the visual sensor and the robot is obtained through sensor calibration. The purpose of sensor calibration is to determine the internal parameters (such as focal length, principal point, etc.) and external parameters (such as rotation matrix, translation vector) of the visual sensor. The sensor calibration method is the checkerboard calibration method, which takes a checkerboard image of known size and square spacing, and uses the geometric relationship of the checkerboard and the pixel coordinates in the image to solve the internal and external parameters of the camera; Determine the target position: Use the pixel coordinates in the image to solve the coordinates of the target in the camera coordinate system, and then convert the coordinates of the target in the camera coordinate system to the robot coordinate system based on the relative posture relationship between the camera and the robot to determine the position of the target crop in the robot coordinate system.

[0023] In this embodiment, by using a convolutional neural network (CNN) algorithm to identify and classify crop images collected by a visual sensor, and using a large number of crop sample images for training, the model can learn to distinguish between crops of different types and different growth stages, so as to accurately identify target crops in actual operations. As time goes by, new sample images can be continuously collected to update and optimize the CNN model to improve its ability to identify newly appearing or changing targets, ensuring that the recognition accuracy is always high; Convert the pixel coordinates of the target crop in the image to the actual coordinates in the robot's workspace. This step can match the information on the image with the position in the real world, so that the robot can know the position of the target crop more accurately. Using the constructed harvesting environment map, the converted coordinates can be further verified and corrected to ensure that the final target position is consistent with the image recognition result and coordinated with the surrounding environment, providing an accurate target point for subsequent path planning. Deep learning models, especially CNN, have performed well in the field of image recognition. They can handle complex background interference and effectively identify target crops even when there are multiple vegetation or other obstructions in the farmland. Through a large number of sample training, the CNN model has a certain robustness to changes in lighting, and can maintain good recognition results under different lighting conditions, reducing misidentification caused by the influence of light intensity. Accurate identification and positioning can directly provide clear target points for the robot's path planning and picking actions, avoiding unnecessary searches and explorations, saving time and resources, and improving overall operational efficiency.

[0024] Example 3 This embodiment is based on Example 1 or Example 2 and is optimized as follows. Specifically, the steps of local path planning in the local path obstacle avoidance in step 6 are as follows: DWA principle: The dynamic window method is a local path planning method based on velocity space sampling. It represents the robot's motion state as a state space (including position, velocity, acceleration). By sampling and predicting the state space, it evaluates whether each sampling point (i.e., possible action) will lead to a collision, and selects the optimal action to control the robot's motion. State space representation: The state of the robot is represented by a vector Indicates that is the position coordinate of the robot in the two-dimensional plane, is the orientation angle of the robot (relative to some reference direction), It's the robot and The velocity component in the direction, is the angular velocity of the robot; Sampling: At each moment, according to the current state of the robot and control input sets (including discrete combinations of acceleration and angular acceleration), generate multiple next states (i.e., predicted states), and the sampling density is adjusted according to factors such as the robot's speed and the complexity of the environment, for example, sparse sampling in simple environments and dense sampling in complex environments; Prediction: Based on the robot's kinematic and dynamic models, predict each sampled state over a period of time (e.g. ) after the new state; Collision Detection: For each predicted state, check whether the robot collides with an obstacle. Obstacles are replaced with geometric shapes and the distance between the robot and the obstacle is calculated to determine whether a collision occurs. If the distance between the predicted state and any obstacle is less than a safety threshold (considering the size of the robot and the buffer zone), a collision is considered to have occurred.

[0025] The local path obstacle avoidance in step 6 also includes obstacle identification and classification and dynamic crossing strategy. The steps of obstacle identification and classification are as follows: Data collection and annotation: Before actual application, a number of images, point cloud data and corresponding sensor data containing various types of obstacles under different terrain backgrounds are collected. Various types of obstacles include soil piles of different sizes, shapes and postures, hoes and shovels of various types and placement methods, and different terrain backgrounds include soil of different textures and whether there is crop cover. These data are annotated by manual or automatic annotation tools to mark the type and key features of the obstacles, including the height, slope, and volume of the soil piles, and the type, size, placement posture, and whether the hoes and shovels are blocked; Feature extraction and model training: Use deep learning algorithms to extract features from the collected data. For image data, CNN automatically learns the shape features and color and texture features of obstacles. For point cloud data, the PointNet network structure is used to extract three-dimensional shape features and spatial distribution features. These extracted features are used as inputs to train the model using support vector machines (SVM) or random forests. During the training process, the model parameters (such as the number of neural network layers) are adjusted to enable the model to classify different obstacles. Classification result output: In actual application, the trained model inputs the sensor data collected in real time into the model and outputs the classification result of obstacles in the form of height. Divide the soil pile into units of height , and For hoes and shovels, identify their type, size, placement posture, and whether they are blocked by crops or soil. The types include ordinary hoes, long-handled hoes, and small shovels. The sizes are divided according to the length, width, and weight parameters. The placement postures include upright, oblique, lying flat, and lodging. The steps of the dynamic crossing strategy for the soil pile are as follows: Scenario 1, for height Control mode when the soil pile is: Motion planning: When the robot recognizes that there is a mound in front of it, it plans the movement trajectory of the leg joints according to the height, slope and shape information of the small mound. Assuming that the height of the small mound is 4 cm, the robot first adjusts the center of gravity of the body to move forward so that the center of gravity is located in the front position between the two feet to increase the forward leaning tendency. Then, the leg joints are controlled to move according to the predetermined trajectory. The hip joint and knee joint work together. The hip joint flexes first to lift the leg and bend the knee. Then the hip joint extends and the knee joint extends at the same time so that the toes can smoothly pass the top of the mound and finally the sole of the foot lands smoothly. During the whole process, the movement speed of the leg joints is adjusted according to the actual situation of the mound. The crossing speed is 1 m / s to ensure the smoothness and accuracy of the movement. Balance control: During the leaping process, the inertial measurement unit is used to monitor the robot's body posture angular velocity and acceleration information in real time. Combined with the foot pressure changes fed back by the pressure sensor, the robot's center of mass position is adjusted through the PID controller. For example, when the robot starts to leap over a pile of dirt, the forward shift of the center of gravity and the lifting of the legs may cause the body to lean forward. At this time, the PID controller will automatically adjust the torque output of the hip and knee joints according to the pitch angle changes detected by the IMU and the pressure changes of the pressure sensor, so that the robot's center of mass remains within a stable range to avoid falling due to imbalance. Scenario 2, for height Control mode when the soil pile is: Body posture adjustment: The robot first lowers its body center of gravity by adjusting the angles of its leg joints. The hip and knee joints are flexed at the same time (hip flexion 2 degrees, knee flexion 3 degrees), so that the body's center of gravity moves down and closer to the support area of ​​the feet. At the same time, the upper body leans forward to move the body's center of mass further forward, so as to better cope with the terrain changes ahead. Step action planning: shorten the stride to increase the stability of the steps. When taking a step, the sole of the foot touches the ground first, gradually increasing the contact area to disperse the pressure. As the foot moves forward, the direction and strength of the foot are adjusted in real time according to the changes in the terrain. For example, when the ground slope is large, the robot will increase the friction between the sole and the ground by adjusting the anti-slip material on the sole or the special sole structure. At the same time, the arm swing is used to assist in maintaining balance. The swing amplitude and frequency of the arm are adaptively adjusted according to the body's movement state. During the entire leap process, the robot moves at a speed of 0.5 m / s to ensure that each step is smooth and reliable. Scenario 3, for height Control mode when the soil pile is: Path planning: The robot first starts the path planning module, which comprehensively considers the height, slope, volume of the large soil pile and the surrounding terrain information (such as whether there is a flat area that can be bypassed), and searches for a feasible path in the global terrain model. For example, if there is a relatively flat area on one side of the soil pile, the robot will plan a path to bypass the area. Obstacle avoidance and replanning: While walking along the planned path, the robot continuously monitors the surrounding terrain changes and other obstacles. If a new obstacle is found or the original path is not feasible, the robot will immediately stop the current action and re-run the path planning algorithm to find a new feasible path. When replanning the path, the robot will update the terrain model and obstacle information based on the newly acquired information to ensure that the re-planned path is more reasonable and safer; The steps for the dynamic spanning strategy for hoe and shovel tools are as follows: Scenario 1, mode for upright or obliquely inserted tools: Approach strategy: For upright or obliquely inserted tools, the robot first approaches the tool, maintaining a distance of 1 meter. During the approach process, the robot uses the end effector of the robot arm to make a preparatory movement to prepare to grasp the tool; Grasping and moving: When the robot is 0.5 meters away from the tool, the robot arm's motion trajectory and the position of the gripper or suction cup are adjusted according to the type and size of the tool. For example, for a longer hoe, the robot arm extends, the gripper opens, and grabs the hoe's shaft or neck. For tools inserted obliquely, the gripper's angle and grasping position need to be adjusted to adapt to the tool's tilt angle. After grabbing the tool, the robot arm lifts it up to get it off the ground. Then, the robot places the tool in an open area according to the terrain to avoid affecting the robot's walking. Scenario 2, mode for a lying or lying tool: Identification and judgment: For tools lying flat on the ground, the robot first identifies their type, size, and placement. It scans the tool's outline and size information through a stereo vision camera or lidar, and compares it with the tool model database to determine the tool's specific information. Crossing decision: If the tool width is less than 0.5 meters, the robot chooses to cross, and when crossing, it adopts the strategy of crossing the pile of soil. If the tool is larger than 0.5 meters, the robot stops moving, evaluates the surrounding terrain and environmental conditions, and chooses to bypass the tool or use the robotic arm to move it to a safe place before continuing to move forward.

[0026] In this embodiment, the DWA principle evaluates actions by sampling and predicting in the state space, and can quickly adapt to the complex and changeable farmland environment. The robot's motion state is represented as a state space including position, velocity, and acceleration, which can fully describe the robot's dynamic characteristics. In actual farmland operations, robot operations such as starting, stopping, and turning all involve these factors. DWA can make accurate plans based on this to ensure smooth and reliable movement. The sampling density is adjusted according to the robot speed and environmental complexity. Sparse sampling in simple environments improves computing efficiency, and dense sampling in complex environments enhances accuracy. For example, the sampling points can be reduced in open farmland and increased in areas with dense obstacles, ensuring planning quality while improving efficiency. Collect images, point clouds and sensor data containing various types of obstacles in different terrain backgrounds, and annotate the obstacle types and key features in detail, so that the model can learn rich features and accurately identify obstacles in different situations, such as different sizes of soil piles and tools placed in different postures; use deep learning algorithms to extract features, and then combine support vector machines (SVM) or random forests for model training to give full play to the advantages of each algorithm. CNN is good at processing image texture features, PointNet can extract point cloud three-dimensional shape features, SVM and random forests have their own advantages in classification, and jointly improve the accuracy and robustness of obstacle classification; the trained model can output obstacle classification results in real time to provide a basis for subsequent actions. For example, different crossing strategies can be selected according to the soil pile height classification results, and the tool can be determined after grasping or avoiding, so that the robot can make intelligent decisions in time and adapt to complex farmland scenes, which is impossible for vehicles / tracked robots; For soil piles less than 10 cm in height, the leg joint motion trajectory and balance control are reasonably planned to ensure smooth and accurate crossing movements, coordinated work of center of gravity adjustment and joints, and real-time adjustment based on sensor feedback by PID controller to avoid imbalance and fall, ensuring stable operation of the robot in the environment of small soil piles. For soil piles greater than 10 cm and less than 50 cm in height, the body posture is first adjusted to lower the center of gravity, the stride is shortened to increase stability, the footstep direction and strength are adjusted in real time according to terrain changes, and arm swing is used to assist balance. These strategies meet the stability requirements when crossing large soil piles. At the same time, the moving speed is adjusted to ensure that each step is reliable, which improves the crossing ability under complex terrain. For larger soil piles greater than 50 cm in height, the path planning module is started to find a feasible path, comprehensively considering the soil pile and surrounding terrain information, and continuously monitoring and replanning during walking to avoid collision with the soil pile. At the same time, the model is updated according to new information to ensure that the path is reasonable and safe, reflecting the adaptability and flexibility under complex terrain. Prepare when approaching the tool, adjust the robot arm's motion trajectory and the gripper or suction cup posture according to the type and size of the tool to ensure accurate grasping, and place the tool according to the terrain after grasping to avoid affecting walking. This achieves effective handling of such tools and improves farmland operation efficiency. Tool information is identified and judged through stereo vision cameras or lidar, and strategies such as crossing, bypassing, and moving are selected based on the width. This intelligent decision-making avoids interference of tools on the robot's walking and ensures the continuity and stability of operations.

[0027] Example 4 This embodiment is a further optimization based on the embodiment 3. Specifically, the picking action execution mode of step 7 is: Agricultural harvesting robots are an automated system that combines image processing, robot control, and artificial intelligence technologies to improve the efficiency and accuracy of crop harvesting. The system captures farmland images through cameras, uses image recognition algorithms to locate the position and posture of crops, plans the movement trajectory of the robotic arm, and performs precise harvesting actions. Camera imaging: realized through visual sensors, assuming a point in space The coordinates of are (X, Y, Z), and its projection point on the image plane The coordinates of , the focal length is , then the imaging model is expressed as: Among them, Z is the distance from point P to the optical center of the camera; Image preprocessing: In order to improve the accuracy of image recognition, the original image needs to be preprocessed, including denoising, white balance, sharpening and other operations. Gaussian filter is used for denoising; Feature extraction and target detection: Use deep learning algorithms (such as convolutional neural networks) to extract features and detect targets on preprocessed images. Let the input image be I. After a series of convolutional layers and pooling layers, a feature map is obtained. The target detection network outputs the category probability of each target. and bounding box coordinates ; Robotic arm motion planning: The motion planning of the robotic arm is based on the inverse kinematics algorithm. The algorithm solves the angle value of each joint according to the target position and posture of the end effector of the robotic arm. During the movement of the robotic arm, the actual position and speed are monitored in real time by sensors and compared with the planned trajectory. The PID controller is used to correct the error to ensure that the robotic arm can accurately track the planned trajectory. The end effector of the robot arm is equipped with a special picking tool, including a flexible gripper or a suction cup. The design of the picking tool should take into account the size, shape and maturity of the fruit. For example, for spherical fruits, an arc-shaped gripper can be used; for soft fruits, a suction cup can be used; According to the information of the fruit, the action parameters of the picking tool are adjusted, including the opening and closing degree of the clamp and the adsorption force of the suction cup. The fruit radius is set to , the maturity is (value range 0-1), the degree of opening and closing of the gripper And the suction cup's adsorption force Respectively expressed as: , ,in, , They are the initial opening and closing degree and the adsorption force. , is the proportionality coefficient; Fruit transportation and storage: The picked fruits are transported to the storage container of the robot through an internal transportation device. The transportation device can be in the form of a conveyor belt or a pipe. The storage container should have appropriate ventilation and preservation measures to extend the shelf life of the fruit.

[0028] In this embodiment, by imaging with a visual sensor and using a specific imaging model (such as the relationship between coordinates and focal length), the projection information of crops in space on the image plane can be accurately obtained, providing accurate basic data for subsequent image analysis, which is helpful to improve the accuracy of crop position and posture recognition. The original image is subjected to preprocessing operations such as denoising, white balancing, and sharpening. Among them, denoising uses a Gaussian filter to remove noise interference in the image, improve image quality, and make image features more obvious, which is conducive to subsequent feature extraction and target detection. Deep learning algorithms such as convolutional neural networks are used for feature extraction and target detection, which can automatically learn complex features in the image, accurately identify different types of crops and their positions, postures, and other information. A feature map is obtained through a series of convolutional layers and pooling layers, and the category probability and bounding box coordinates of the output target are output, which can achieve high-precision target detection and reduce misidentification and missed recognition. Based on the inverse kinematics algorithm, the angle value of each joint is solved according to the target position and posture of the end effector of the robot arm, which can ensure that the robot arm moves accurately to the position of the crop according to the planned trajectory and achieve accurate picking. During the movement of the robot arm, the actual position and speed are monitored in real time by sensors and compared with the planned trajectory. The PID controller is used to correct the error. This real-time feedback control mechanism can adjust the movement state of the robot arm in time, overcome the influence of external factors (such as wind force, robot vibration, etc.) on the movement of the robot arm, ensure that the robot arm can accurately track the planned trajectory, and improve the accuracy and stability of the picking action; Special picking tools are designed for different types of fruits, such as arc-shaped clamps for spherical fruits and suction cups for soft fruits. This improves the adaptability of picking tools to fruits of different shapes and textures, reduces damage to fruits during picking, and ensures the integrity and quality of fruits. The action parameters of picking tools are adjusted according to factors such as the size, shape and maturity of the fruits, such as the opening and closing degree of the clamps and the suction force of the suction cups. By setting a reasonable initial opening and closing degree, suction force and proportional coefficient, the picking force can be accurately controlled according to actual conditions to avoid picking too loose or too tight, further improving the accuracy and efficiency of picking, while reducing damage to fruit trees; The picked fruits are promptly transported to the robot's own storage container through an internal transport device (such as a conveyor belt or pipe), which ensures the continuity of the picking process, avoids damage and contamination caused by the fruits falling to the ground, and improves the efficiency of fruit collection. The storage container has proper ventilation and preservation measures, which can extend the shelf life of the fruits, maintain the freshness and quality of the fruits, reduce the loss of fruits due to deterioration due to improper storage, and increase the economic benefits of agriculture after harvest.

[0029] Please refer to Figure 2 The present invention also discloses an operation system of a humanoid agricultural robot based on path planning, comprising: Humanoid robot body: including head, torso, arms and legs, each part is connected by joints to simulate the movement structure of humans, and the joints are equipped with motors and reducers to realize the movement control of each joint; Visual sensor: installed on the robot head or arms to collect image information of crops; LiDAR and depth camera: installed at different locations on the robot, including the head and shoulders, to obtain three-dimensional structural information of the environment; Inertial measurement unit: installed at the center of gravity of the robot, used to monitor the robot's posture and motion status in real time; Auxiliary sensors: including contact sensors and pressure sensors, which are used to sense the contact between the robot and the surrounding environment and provide feedback information for the execution of picking actions; Actuator module: including robotic arms and leg walking mechanisms. The humanoid robot is equipped with two robotic arms, each with multiple degrees of freedom to simulate the movement of human arms. The end effector of the robotic arm can replace different types of tools, including grippers and suction cups, according to different picking task requirements. The leg walking mechanism adopts a humanoid gait design, which can walk stably on complex terrain and cross some obstacles. At the same time, the leg walking mechanism also has certain climbing and balancing capabilities to adapt to different agricultural operation environments. Control system: including main controller and motion controller. The main controller is responsible for the coordination, control and management of the entire robot system. The motion controller is used to control the movement of the motors of each joint of the robot to achieve motion trajectory tracking. Data processing unit: responsible for collecting and processing data from various sensors and converting them into a format that can be recognized and used by the robot control system; CPU: used to process data and various mathematical operations for computationally intensive tasks such as image recognition and path planning; Communication module: used for data transmission and communication between the robot and external devices; Power system: Lithium batteries are used as the power source of the robot to provide power support for each component of the robot.

[0030] In summary: the present invention can obtain environmental information of the picking area from multiple dimensions through the collaborative work of multiple sensors. The visual sensor obtains the position, color, and shape information of crops, which is helpful to accurately identify crops of different types and maturity. The three-dimensional structural information obtained by the laser radar and the depth camera can accurately construct a point cloud map of the environment, providing high-precision spatial data support for subsequent path planning and operation operations. The IMU monitors the posture and motion state of the robot in real time to ensure that the robot can operate stably in complex terrain and improve the accuracy of positioning and navigation. Different types of sensors can adapt to various complex agricultural environments and operation scenarios. Whether in outdoor farmland with changing lighting conditions or in a relatively closed and obstructed greenhouse environment, the advantages of each sensor can be complementary to each other to comprehensively and accurately collect environmental information, so that the robot can better cope with the operation requirements in different environments. The comprehensive picking environment map generated by the present invention contains information on topography, crop distribution, obstacle locations, etc., providing a comprehensive and accurate basis for the robot's path planning, task allocation, and operation strategy formulation. Based on such a map, the robot can more intelligently plan the optimal path, avoid collisions with obstacles, efficiently complete the picking operation, and improve the operation efficiency and quality. During the operation, the sensor continuously collects environmental information, and the data processing system can update the picking environment map in real time, so that it can dynamically adapt to environmental changes. The humanoid robot of the present invention can select different crossing strategies according to the soil pile height classification results, and decide to grasp or avoid the tool after identifying it, so that the robot can make intelligent decisions in time and adapt to complex farmland scenes.

[0031] Finally, it should be noted that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for operating a humanoid agricultural robot based on path planning, characterized in that: The following steps are involved: Step 1: Multi-sensor data collection: The humanoid agricultural robot is equipped with multiple sensors for data collection, including visual sensors, lidar, depth cameras, and inertial measurement units; Step 2: Data processing and map generation: pre-process and fuse the data collected by different sensors to generate a picking environment map; Step 3: Map update: Continuously collect new environmental data and update the map in real time; Step 4: Target crop image recognition and positioning: Use the convolutional neural network algorithm to recognize and classify the crop images collected by the visual sensor, and combine the map information to determine the location of the target crop; Step 5: Global path planning: Based on the constructed picking environment map and target crop location information, the A algorithm is used for global path planning; Step 6: Local path obstacle avoidance: Monitor environmental information, and when encountering sudden obstacles, adjust the robot's movement direction and speed to avoid the obstacles; Step 7: Execution of picking action: According to the position and posture of the target crops, the motion trajectory of the robotic arm is planned, the inverse kinematics algorithm is used to solve the angle values ​​of each joint of the robotic arm, and the motion parameters of the picking tool are adjusted to realize the picking of fruits.

2. The operation method of a humanoid agricultural robot based on path planning according to claim 1, characterized in that: The data processing in step 2 and the data fusion processing in map generation are as follows: Coordinate system conversion: integrate the data of different sensors into a unified coordinate system and perform coordinate system conversion on the data of each sensor; Fusion of visual and lidar data: Project the point cloud data acquired by the lidar onto the image plane, and use the feature points of the image to align with the point cloud data using the iterative closest point algorithm; combine the color information in the visual image with the three-dimensional coordinate information in the point cloud data to construct a fusion map. Each pixel in the fusion map contains color information and corresponding three-dimensional coordinate information. Fusion of vision, lidar and depth camera data: The depth image acquired by the depth camera is fused with the point cloud data of the lidar. Through weighted fusion, different weight coefficients are assigned according to the accuracy of the depth camera and lidar in different areas, and then the weighted average depth value is calculated; the semantic segmentation results and depth information in the visual image are combined to fuse the content of the map, and the boundaries and positions of objects of different categories are marked in the fused map; Add inertial measurement unit data fusion: Use the posture information measured by the inertial measurement unit to correct the posture of the feature points in the fusion map, convert the coordinates of the feature points into coordinates in the global coordinate system through the coordinate transformation matrix, and adjust them according to the posture information of the inertial measurement unit; combine the motion state information of the inertial measurement unit to optimize the motion trajectory of the robot in the picking environment.

3. The operation method of a humanoid agricultural robot based on path planning according to claim 1, characterized in that: The map update method in step 3 is: New data collection: The sensor collects new environmental data at a set frequency and pre-processes the data; Change detection: Compare newly collected data with existing map data to detect whether the environment has changed. Change detection is achieved by calculating the difference in data; Local update: If a change in the environment is detected, the changed local area is updated; Global update: Every hour, the entire map is globally updated to build a new fused map.

4. The operation method of a humanoid agricultural robot based on path planning according to claim 1, characterized in that: The steps for target crop image recognition and positioning in step 4 are as follows: Training process: Using supervised learning, several images and their labels are input into the model, the model's predicted output is calculated through forward propagation, and the loss function is calculated based on the difference between the predicted output and the true label; through the back-propagation algorithm, the model parameters are adjusted according to the loss function to reduce the value of the loss function; Identification process: During the harvesting operation, the crop images collected in real time are input into the trained CNN model. The model calculates the probability distribution of the input image belonging to various types of crops through forward propagation, and determines the crop category to which the input image belongs based on the maximum value in the probability distribution, thereby obtaining the recognition result of the target crop; The method of obtaining the location information of the target in the image is as follows: after obtaining the recognition result, the region proposal network is introduced into the CNN model to generate region proposals containing the target crop in the image, and a score is assigned to each proposal. Then, the proposals are screened and adjusted according to the score to obtain the target location box; Determine the position of the target in the robot coordinate system: obtain the relative position and posture relationship between the visual sensor and the robot through sensor calibration. The purpose of sensor calibration is to determine the internal and external parameters of the visual sensor. The sensor calibration method is the checkerboard calibration method to solve the internal and external parameters of the camera; Determine the target position: Use the pixel coordinates in the image to solve the coordinates of the target in the camera coordinate system, and then convert the coordinates of the target in the camera coordinate system to the robot coordinate system based on the relative posture relationship between the camera and the robot to determine the position of the target crop in the robot coordinate system.

5. The operation method of a humanoid agricultural robot based on path planning according to claim 1, characterized in that: The global path planning method in step 5 is: S1: Constructing the search space: In the picking environment map, the robot's starting position, target crop position, and intermediate points are defined as nodes, and each node is represented by its coordinates in the map. Indicates that the line segment connecting two adjacent nodes is called an arc; S2: Create an open list and a closed list: Sort the values ​​of the evaluation function f(n) from small to large. Initially, put the starting node S into the open list; during the algorithm execution, once a node is expanded, transfer it from the open list to the closed list; put the starting node S into the open list and set the starting node , Calculated based on the estimated distance from the starting node to the target node; S3: Select an evaluation function from an open list The node n with the smallest value is used as the current expansion node. If the current expansion node n is the target node G, the optimal path is found and the algorithm ends. At this time, the target node is traced back to the starting node along the parent node pointer to obtain the complete path. Otherwise, transfer the current expanded node n from the open list to the closed list; for each adjacent node of the current expanded node n ,if Already in the closed list, skip the node; if If it is not in the open list, add it to the open list and calculate its evaluation function At the same time, set its parent node pointer to point to the current node n. If Already in the open list, compare the new path cost with the original path cost, and execute in a loop until the open list is empty or the target node is found; S4: Path backtracking: If the target node is found, start from the target node and return to the previous node in sequence according to the parent node pointer until you return to the starting node and get the optimal path.

6. The operation method of a humanoid agricultural robot based on path planning according to claim 1, characterized in that: The steps of local path obstacle avoidance in step 6 are as follows: State space representation: The state of the robot is represented by a vector Indicates that is the position coordinate of the robot in the two-dimensional plane, is the robot's orientation angle, It's the robot and The velocity component in the direction, is the angular velocity of the robot; Sampling: At each moment, according to the current state of the robot and control input set, generate multiple next states, and the sampling density is adjusted according to factors such as the robot's speed and the complexity of the environment; Prediction: Based on the robot's kinematic and dynamic models, predict the new state of each sampled state after a period of time; Collision detection: For each predicted state, check whether the robot collides with an obstacle. Obstacles are replaced with geometric shapes and the distance between the robot and the obstacle is calculated to determine whether there is a collision. If the distance between the predicted state and any obstacle is less than the safety threshold, a collision is considered to occur.

7. The operation method of a humanoid agricultural robot based on path planning according to claim 1, characterized in that: The local path obstacle avoidance in step 6 also includes obstacle recognition and classification. The steps of obstacle recognition and classification are as follows: Data collection and annotation: Collect a number of images, point cloud data and corresponding sensor data containing various types of obstacles in different terrain backgrounds, and mark the types and key features of the obstacles; Feature extraction and model training: Use deep learning algorithms to extract features from the collected data, use the extracted features as input, and use support vector machines (SVMs) for model training to enable the model to classify different obstacles; Classification result output: In actual application, the trained model inputs the sensor data collected in real time into the model and outputs the classification result of obstacles in the form of height. Divide the soil pile into units of height and for hoes and shovels, identify their type, size, placement, and whether they are blocked by crops or soil.

8. The operation method of a humanoid agricultural robot based on path planning according to claim 7, characterized in that: The local path obstacle avoidance in step 6 also includes a dynamic crossing strategy; The steps of the dynamic crossing strategy for the soil pile are as follows: Balance control: During the leaping process, the inertial measurement unit is used to monitor the robot's body posture angular velocity and acceleration information in real time. Combined with the foot pressure changes fed back by the pressure sensor, the robot's center of mass position is adjusted through the PID controller. Body posture adjustment: The robot first lowers its body center of gravity by adjusting the angles of its leg joints, so that the body's center of gravity moves down and closer to the support area of ​​its feet. Obstacle avoidance and replanning: While walking along the planned path, the robot continuously monitors the surrounding terrain changes and other obstacles. If a new obstacle is found or the original path is not feasible, the robot will immediately stop the current action and re-run the path planning algorithm to find a new feasible path; The steps for the dynamic spanning strategy for hoe and shovel tools are as follows: Grasping and moving: When the robot is 0.5 meters away from the tool, the robot adjusts the movement trajectory of the robot arm and the posture of the gripper or suction cup according to the type and size of the tool. After grasping the tool, the robot arm lifts it up to make the tool off the ground. The robot places the tool in an open area according to the terrain conditions. Crossing decision: If the tool width is less than 0.5 meters, the robot chooses to cross, and when crossing, it adopts the strategy of crossing the pile of soil. If the tool is larger than 0.5 meters, the robot stops moving, evaluates the surrounding terrain and environmental conditions, and chooses to bypass the tool or use the robotic arm to move it to a safe place before continuing to move forward.

9. The operation method of a humanoid agricultural robot based on path planning according to claim 1, characterized in that: The picking action of step seven is performed as follows: Camera imaging: photographing crops through visual sensors; Image preprocessing: preprocess the original image; Feature extraction and target detection: Use deep learning algorithms to extract features and detect targets from preprocessed images; Robotic arm motion planning: The motion planning of the robotic arm is based on the inverse kinematics algorithm, which solves the angle value of each joint according to the target position and posture of the robot arm end effector, monitors its actual position and speed through sensors, and compares it with the planned trajectory, and uses a PID controller to correct the error; according to the information of the fruit, adjust the motion parameters of the picking tool, including the opening and closing degree of the gripper and the adsorption force of the suction cup.

10. An operation system of a humanoid agricultural robot based on path planning, characterized in that: include: The humanoid robot body: includes the head, torso, arms and legs, each part is connected by joints, and the joints are equipped with motors and reducers; Visual sensor: installed on the robot head or arms to collect image information of crops; LiDAR and depth camera: installed at different locations on the robot, including the head and shoulders, to obtain three-dimensional structural information of the environment; Inertial measurement unit: installed at the center of gravity of the robot, used to monitor the robot's posture and motion status in real time; Auxiliary sensors: including contact sensors and pressure sensors, which are used to sense the contact between the robot and the surrounding environment and provide feedback information for the execution of picking actions; Actuator module: including robotic arms and leg walking mechanisms. The humanoid robot is equipped with two robotic arms. The end effector of the robotic arms can replace different types of tools, including grippers and suction cups, according to different picking task requirements. Control system: including main controller and motion controller. The main controller is responsible for the coordination, control and management of the entire robot system. The motion controller is used to control the movement of the motors of each joint of the robot to achieve motion trajectory tracking. Data processing unit: responsible for collecting and processing data from various sensors and converting them into a format that can be recognized and used by the robot control system; Central Processing Unit: Used to process data and various mathematical operations for computationally intensive tasks such as image recognition and path planning.

Images