Field picking obstacle avoidance method and system based on active perception

Abstract

The invention relates to a field picking obstacle avoidance method and system based on active perception. The picking obstacle avoidance method comprises the steps that hardware and a mechanical arm control system are initialized; rGB image data and depth image data of crops are acquired and preprocessed; inputting the preprocessed RGB image data and depth image data of the crops into an improved YOLOv5 network with an attention mechanism; analyzing the pixels of the region of interest based on the color threshold of the crop target, and providing an obstacle avoidance basis; moving a mechanical arm based on the crop target information; if the shielding rate of the crop target reaches a threshold value or the crop target has no obstacle, and the depth information of the crop target meets the picking condition, the mechanical arm executes a picking task. According to the method provided by the invention, the color, the depth information and the spatial position of the crop are sensed in real time, and the target recognition and segmentation of deep learning are combined, so that the shielding, obstacle and target characteristics can be accurately sensed.

Description

Technical Field

[0001] The present invention relates to the technical field of agricultural automation control, and particularly relates to a method and system for obstacle avoidance in field picking based on active perception, which can be applied to complex and unstructured fields for crop picking and obstacle avoidance operations. Background Art

[0002] As a large agricultural country, China has always had the advantages of a large cultivated area and a variety of crop species. In the long-term development process of farming technology, agricultural production has gradually advanced from primitive slash-and-burn farming to agricultural mechanization and intelligence. With the development of smart agriculture, agricultural intelligence has become the theme of development. Realizing the mechanization and automation of fruit picking is beneficial to solving problems such as labor shortage, low productivity, and high production costs in agricultural production, and can improve efficiency and save labor costs at the same time.

[0003] In the field of agricultural automation, obstacle avoidance in field picking is one of the key technologies to ensure that the picking robotic arm can complete tasks efficiently and accurately. However, the existing picking obstacle avoidance methods have the following problems:

[0004] 1. Rough occlusion judgment: Most of the existing occlusion processing methods are based on simple threshold judgments of depth information, and cannot refine the analysis of the occlusion rate and position characteristics of the crop area, which easily leads to incorrect path planning or obstacle avoidance failure. For example, the existing methods may not be able to accurately distinguish between the occluded part and the non-occluded part of the crop, thus affecting the path planning of the picking robotic arm.

[0005] 2. Limited path planning: The existing methods rely on preset static path planning algorithms, lack real-time perception feedback, and cannot avoid obstacles dynamically. Once an obstacle moves (such as a leaf blown by the wind), the robotic arm may misjudge or lose the target, seriously reducing the picking efficiency and reliability.

[0006] 3. Slow response to environmental changes: There are uncontrollable real-time changes in the agricultural environment (such as suddenly moving occluders or non-target objects), and many solutions lack targeted optimization and cannot quickly adapt to dynamic changes.

[0007] In summary, the existing field picking obstacle avoidance technologies have many limitations in dealing with complex and dynamic agricultural environments. Summary of the Invention

[0008] In view of some or all of the problems in the prior art, the present invention provides a method for obstacle avoidance in field picking based on active perception, and the method includes the following steps:

[0009] Initialize the hardware and the robotic arm control system, where the hardware includes a camera and a robotic arm;

[0010] Obtain the RGB image data and depth image data of the crop, and perform preprocessing;

[0011] Input the RGB image data and depth image data of the preprocessed crop into the improved YOLOv5 network with an attention mechanism to locate the crop target and extract the spatial position of the crop target.

[0012] Detect that the crop target is within the target area, analyze the pixels in the region of interest based on the color threshold of the crop target, calculate the occlusion rate of the region of interest, and determine whether the target area is partially occluded to provide a basis for obstacle avoidance.

[0013] Move the robotic arm based on the crop target information. The robotic arm will continue to perform active perception during movement to ensure that obstacles on the path can be recognized and avoided in a timely manner. When the robotic arm approaches the crop target, a closed-loop control system is used for fine-tuning; and

[0014] If the occlusion rate of the crop target reaches the threshold or there are no obstacles for the crop target, and the depth information of the crop target meets the picking conditions, the robotic arm performs the picking task.

[0015] Further, initialize the hardware and the robotic arm control system. The hardware includes a camera and the robotic arm includes:

[0016] The hardware further includes an external sensor. Initialize the camera and the external sensor. The camera includes an RGB camera and a depth camera. Set the resolution, frame rate, and exposure parameters of the RGB camera and the depth camera.

[0017] Initialize the robotic arm control system, initialize the robotic arm, and set the motion range, joint angles, and load capacity parameters of the robotic arm; and

[0018] Ensure that the states of the hardware and the robotic arm control system meet the requirements.

[0019] Further, obtain the RGB image data and depth image data of the crop, and perform preprocessing including:

[0020] Use the RGB camera to obtain the RGB image data of the crop and the surrounding environment. The RGB image data includes appearance, color, and texture.

[0021] Use the depth camera to obtain the depth image data of the crop and the surrounding environment, and perform noise processing and filtering on the depth image data.

[0022] Perform data synchronization on the RGB image data and the depth image data through a timestamp, a trigger, or a software algorithm; and

[0023] Perform preprocessing on the RGB image data and the depth image data. The preprocessing includes image denoising, image cropping, color standardization, and scale correction.

[0024] Further, input the RGB image data and depth image data of the preprocessed crop into an improved YOLOv5 network with an attention mechanism to perform localization on the crop target and extract the spatial position of the crop target, including:

[0025] Input the RGB image data of the preprocessed crop into an improved YOLOv5 network with an attention mechanism. The improved YOLOv5 network with an attention mechanism includes CSP-Darknet for feature extraction, PA-Net for feature fusion, and a YOLO layer for outputting detection results.

[0026] Pair the resulting image data obtained by processing the RGB image data with the improved YOLOv5 network with an attention mechanism with the depth image data.

[0027] According to the class information, bounding box, and confidence of the detection results output by the improved YOLOv5 network with an attention mechanism, identify the crop target in the image and determine the position of the crop target; and

[0028] Calculate the three-dimensional coordinates of the crop target through the depth image data of the crop target and camera calibration data, and align them with the coordinate system of the robotic arm.

[0029] Further, detect that the crop target is within the target area, analyze the pixels in the region of interest based on the color threshold of the crop target, calculate the occlusion rate of the region of interest, and determine whether the target area is partially occluded, providing a basis for obstacle avoidance, including:

[0030] Extract the region of interest of the crop target from the recognition results output by the improved YOLOv5 network with an attention mechanism. Extract the shadow area and leaf area of the crop target through image processing methods, and combine the depth image data to obtain the average depth, area, and centroid coordinates in three-dimensional space of each region.

[0031] Analyze the pixels in the region of interest and convert them into pixels in the HSV color gamut; and

[0032] Calculate the occlusion rate of the region of interest and determine whether the target area is partially occluded;

[0033] The occlusion rate R of the region of interest O is calculated by the following formula

[0034]

[0035] where A f and A O represent the area of the region of interest of the crop target and the area of the shadow area of the crop target, respectively.

[0036] Further, based on the crop target information, the robotic arm is moved. During the movement, the robotic arm will continue to perform active perception to ensure that obstacles on the path can be recognized and avoided in a timely manner. When the robotic arm approaches the crop target, the use of a closed-loop control system for fine-tuning includes:

[0037] Based on the three-dimensional coordinates of the crop target, the robotic arm control system calculates the shortest path from the current position of the robotic arm to the target position. In path planning, considering the movement range, joint angles, and speed parameters of the robotic arm, a kinematic model is used to calculate accurate control instructions;

[0038] During the movement of the robotic arm, the robotic arm control system uses a depth camera to real-time monitor the images and depth data of the surrounding environment, and detects whether there are new obstacles or crops. If new obstacles or crops are detected, the robotic arm control system adjusts the path or re-plans the path to ensure that the robotic arm can avoid the obstacles and continue to approach the crop target; and

[0039] When the robotic arm approaches the target, a closed-loop control system is used for fine-tuning, including using a vision guidance system to guide the robotic arm to perform fine positioning on the crop target to ensure that the robotic arm accurately contacts the crop target.

[0040] Further, if the occlusion rate of the crop target reaches the threshold or there are no obstacles to the crop target, and the depth information of the crop target meets the picking conditions, the robotic arm performs the picking task including:

[0041] If the occlusion rate of the crop target reaches the threshold or there are no obstacles to the crop target, the robotic arm control system checks whether the distance between the crop target and the robotic arm is within the set safety range. If the distance is within the safety range, the robotic arm control system performs the picking task. If the distance is less than the safety range, the robotic arm control system issues a warning and stops moving; and

[0042] If the occlusion rate of the crop target does not reach the threshold or there are obstacles to the crop target, the robotic arm control system re-performs the active perception and obstacle avoidance operation based on the crop target, and the robotic arm control system re-detects the crop target and adjusts the path in real time according to the feedback information.

[0043] The present invention also provides a system for the active perception-based field picking and obstacle avoidance method described above. The system includes the following modules:

[0044] An initialization module, configured to initialize the hardware and the robotic arm control system, where the hardware includes a camera and a robotic arm;

[0045] An image data acquisition module, configured to acquire RGB image data and depth image data of the crop and perform preprocessing;

[0046] The YOLOv5 network module is configured to input the RGB image data and depth image data of the preprocessed crop into the improved YOLOv5 network with an attention mechanism, perform localization on the crop target, and extract the spatial position of the crop target;

[0047] The occlusion rate calculation module is configured to detect the crop target within the target area, analyze the pixels in the region of interest based on the color threshold of the crop target, calculate the occlusion rate, and determine whether the target area is partially occluded, providing a basis for obstacle avoidance;

[0048] The active perception module is configured to move the robotic arm based on the crop target information. The robotic arm will continue to perform active perception during the movement to ensure that obstacles on the path can be recognized and avoided in a timely manner. When the robotic arm approaches the crop target, a closed-loop control system is used for fine-tuning; and

[0049] The picking module is configured to, if the occlusion rate of the crop target reaches the threshold or there are no obstacles for the crop target, and the depth information of the crop target meets the picking conditions, then the robotic arm performs the picking task.

[0050] The present invention also provides a computer system, including:

[0051] A processor configured to execute machine-readable instructions;

[0052] A graphics card with an artificial intelligence chip configured to train the field picking obstacle avoidance method based on active perception; and

[0053] A memory configured to store machine-readable instructions, and the machine-readable instructions, when executed by the processor and / or the graphics card, execute the steps of the field picking obstacle avoidance method based on active perception.

[0054] The present invention also provides a computer-readable storage medium, on which machine-readable instructions are stored, and the machine-readable instructions, when executed by the processor, execute the steps of the field picking obstacle avoidance method based on active perception.

[0055] The technical solution provided by the present invention has the following advantages:

[0056] 1. The field picking obstacle avoidance method based on active perception proposed by the present invention uses an RGB camera and a depth camera to real-time sense the color, depth information and spatial position of the crop, combines object recognition and segmentation of deep learning, can accurately sense occlusion, obstacles and target characteristics, and the attention mechanism is used to improve the recognition accuracy, adapting to complex fruit backgrounds and field environments.

[0057] 2. The method for obstacle avoidance in field picking based on active perception proposed by the present invention introduces a real-time path planning and feedback mechanism, enabling the robotic arm to dynamically adjust its actions, bypass random or moving obstacles, achieve continuous operation, and flexibly respond to complex planting environments, such as problems where fruits are hidden behind branches and leaves and the occlusion dynamics change.

[0058] 3. The method for obstacle avoidance in field picking based on active perception proposed by the present invention adopts a closed-loop mechanism of active perception and dynamic adjustment to ensure that each action during the picking process is based on real-time updated information, improving the robustness and coherence of obstacle avoidance and picking.

[0059] 4. The method for obstacle avoidance in field picking based on active perception proposed by the present invention is also suitable for being deployed in agricultural production bases with unmanned operations. Combined with robotic arms with automatic navigation or unmanned driving systems, it can achieve large-area continuous picking under unmanned management conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] To further clarify the above and other advantages and features of the embodiments of the present invention, a more specific description of the embodiments of the present invention will be presented with reference to the accompanying drawings. It can be understood that these drawings only depict typical embodiments of the present invention and thus will not be considered as limiting its scope. In the drawings, for clarity, the same or corresponding components will be denoted by the same or similar reference numerals.

[0061] Figure 1 FIG. shows a schematic flow chart of the method for obstacle avoidance in field picking based on active perception according to an embodiment of the present invention;

[0062] Figure 2 FIG. shows a schematic overall control diagram of the method for obstacle avoidance in field picking based on active perception according to an embodiment of the present invention;

[0063] Figure 3 FIG. shows a schematic identification diagram of an improved YOLOv5 network with an attention mechanism according to an embodiment of the present invention;

[0064] Figure 4 FIG. shows a schematic diagram of the calculation method for the occlusion rate and occlusion orientation of the region of interest according to an embodiment of the present invention;

[0065] Figure 5 FIG. shows a schematic diagram of active perception according to an embodiment of the present invention; and

[0066] Figure 6 FIG. shows a schematic diagram of the system for obstacle avoidance in field picking based on active perception according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0067] In the following description, the present invention is described with reference to various embodiments. However, those skilled in the art will recognize that the embodiments can be implemented without one or more specific details or in conjunction with other alternative and / or additional methods or components. In other instances, well-known structures or operations are not shown or described in detail to avoid obscuring the inventive aspects of the present invention. Similarly, for purposes of explanation, specific numbers and configurations are set forth in order to provide a thorough understanding of the embodiments of the present invention. However, the present invention is not limited to these specific details.

[0068] In this specification, the reference to "an embodiment" or "the embodiment" means that the particular features, structures, or characteristics described in connection with the embodiment are included in at least one embodiment of the present invention. The phrase "in an embodiment" appearing throughout this specification does not necessarily all refer to the same embodiment.

[0069] It should be noted that the embodiments of the present invention describe the method steps in a specific order. However, this is only for the purpose of explaining the specific embodiment and does not limit the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to the actual requirements.

[0070] In the present invention, the various modules of the system according to the present invention can be implemented using software, hardware, firmware, or a combination thereof. When a module is implemented using software, the functions of the module can be achieved through a computer program flow. For example, the module can be implemented through a code segment (such as a code segment in languages like C, C++) stored in a storage device (such as a hard disk, memory, etc.), where when the code segment is executed by a processor, the corresponding functions of the module can be achieved. When a module is implemented using hardware, the functions of the module can be achieved by setting up the corresponding hardware structure. For example, the functions of the module can be achieved by hardware programming of a programmable device such as a field-programmable gate array (FPGA), or by designing an application-specific integrated circuit (ASIC) including multiple electronic devices such as transistors, resistors, and capacitors. When a module is implemented using firmware, the functions of the module can be written in a read-only memory such as an EPROM or EEPROM of the device in the form of program code, and when the program code is executed by a processor, the corresponding functions of the module can be achieved. Additionally, certain functions of the module may need to be implemented by separate hardware or in cooperation with the hardware. For example, the detection function is achieved through a corresponding sensor (such as a proximity sensor, an acceleration sensor, a gyroscope, etc.), the signal emission function is achieved through a corresponding communication device (such as a Bluetooth device, an infrared communication device, a baseband communication device, a Wi-Fi communication device, etc.), the output function is achieved through a corresponding output device (such as a display, a speaker, etc.), and so on.

[0071] The present invention proposes an active perception-based field picking obstacle avoidance method for path planning of picking obstacle avoidance in a complex field environment. By introducing active perception technology and combining dynamic obstacle avoidance with path optimization, the robotic arm can quickly identify and adapt to complex picking environments, significantly shortening the single picking time and improving the overall operation efficiency. There are many uncertainties in the field environment, such as crop occlusion, obstacle interference, crop variety diversity, etc. This technology improves the ability of the picking robotic arm to handle obstacles and occlusion situations in the natural environment through multi-modal perception and continuous feedback mechanisms. The key to the wide application of agricultural robotic arms lies in reducing the dependence on manual picking. Through autonomous obstacle avoidance and intelligent picking technologies, unmanned operation is maximally achieved, and the long-term labor cost is reduced.

[0072] Figure 1 FIG. 4 shows a schematic flow chart of an active perception-based field picking obstacle avoidance method according to an embodiment of the present invention; Figure 2 FIG. 5 shows an overall control schematic diagram of an active perception-based field picking obstacle avoidance method according to an embodiment of the present invention. The following combines Figure 1 and Figure 2 to illustrate the active perception-based field picking obstacle avoidance method proposed by the present invention. In an embodiment of the present invention, the active perception-based field picking obstacle avoidance method can be executed by a computer. As Figure 1 shown, the active perception-based field picking obstacle avoidance method includes the following steps:

[0073] Step 1: Initialize the hardware and the robotic arm control system. The hardware includes a camera, a robotic arm, and other external sensors. Step 1 specifically includes the following three sub-steps:

[0074] Step 1.1: Initialize the camera and external sensors. The camera includes an RGB camera and a depth camera. Set the resolution, frame rate, and exposure parameters of the RGB camera and the depth camera. To ensure that the acquired image quality is suitable for subsequent processing, the focal length and image clarity of the camera are usually adjusted to ensure the efficient synchronization of the RGB image and the depth image.

[0075] Step 1.2: Initialize the robotic arm control system, load the robotic arm control system, initialize the robotic arm, and set the motion range, joint angles, and load capacity parameters of the robotic arm to ensure that these parameters are within an acceptable range. The robotic arm control system needs to perform self-checks to ensure that the motors of each joint are running normally and that the sensor calibration is accurate.

[0076] Step 1.3: Check the status of each hardware and software to ensure no hardware failures. Through the self-calibration of the sensors, confirm the synchronization of the data stream, and provide a stable input signal for the subsequent steps. Ensure that the status of the hardware and the robotic arm control system meets the requirements.

[0077] In one embodiment of the present invention, the depth camera can be a Time of Flight (ToF) camera.

[0078] Initialization is a necessary step to ensure that the software and hardware systems enter the normal working state. The camera ensures the alignment of the RGB image and the depth image through internal and external parameter calibration, providing accurate spatial information support. The robotic arm uses motion control theory to reset and correct the initial position and movement range of the robotic arm, ensuring the safety and accuracy of subsequent actions.

[0079] It is achieved through the initialization of the camera and robotic arm programs: the system internal parameters are unified, avoiding spatial coordinate conversion errors and error accumulation, and through reset and status checks, ensuring that the hardware is in good operating condition, providing a stable platform for subsequent complex tasks.

[0080] Step 2: Obtain the RGB image data and depth image data of the crop and perform preprocessing. Step 2 specifically includes the following four sub-steps:

[0081] Step 2.1: Use the RGB camera to obtain the RGB image data of the crop and the surrounding environment. The RGB image data includes feature information such as appearance, color, and texture, facilitating subsequent crop target detection and classification. The image quality directly affects the detection accuracy of the subsequent YOLOv5 network.

[0082] Step 2.2: Use the depth camera to obtain the depth image data of the crop and the surrounding environment. The depth image data reflects the relative distance between each pixel point and the camera, which is crucial for obstacle detection and crop target positioning. Since the accuracy of the depth map is usually affected by environmental light, the depth camera may be distorted or have errors. Therefore, it is necessary to perform noise processing and filtering on the depth image data to remove invalid information.

[0083] Step 2.3: To ensure the subsequent processing effect, the RGB image data and the depth image data need to be precisely synchronized. Ensure the matching of the RGB image data and the depth image data through timestamps, triggers, or software algorithms, so that each frame of the RGB image and the depth image represents the same moment scene. Asynchronous data may cause spatial deviation during crop target detection, thus affecting the recognition and positioning accuracy.

[0084] Step 2.4: Perform preprocessing on the RGB image data and the depth image data. The preprocessing includes image denoising, image cropping, color normalization, and scale correction, etc. The scale correction is mainly for the depth map. This sub-step ensures that the data input into the subsequent YOLOv5 network is clear, stable, and of high quality.

[0085] Based on multi-modal sensing technology, the complementary nature of RGB image data and depth image data is utilized to enhance the environmental perception ability. RGB image processing provides rich crop texture and color information for crop target detection and area segmentation. Depth image data, on the other hand, calculates the relative distance between crop targets and obstacles and obtains three-dimensional positions through the pulsed light of a depth camera.

[0086] By perceiving the RGB image data and depth image data of the environment, the manipulator can judge the target position and posture more accurately. And by adding depth information derivation, it can avoid misjudgments caused by occlusion or lighting effects in the planar RGB image, thus improving the adaptation to complex scenarios.

[0087] Step 3: Input the preprocessed RGB image data and depth image data of the crop into an improved YOLOv5 network with an attention mechanism (hereinafter referred to as the YOLOv5 network) to locate the crop target and extract the spatial position of the crop target. Figure 3 The schematic diagram of the improved YOLOv5 network with an attention mechanism according to an embodiment of the present invention is shown. Figure 3 The crop in Figure 3 is a tomato, and the crop target is a tomato fruit. The following combines

[0088] Step 3.1: Input the preprocessed RGB image data of the crop into the improved YOLOv5 network with an attention mechanism. The YOLOv5 network has the advantages of real-time detection, high speed, and high accuracy. The input of the model is the preprocessed RGB image data and depth image data. After being processed by multiple convolutional networks and pooling layers, the output includes the category, bounding box position, and confidence of the crop target. By introducing the attention mechanism, the model can automatically focus on the key areas in the image, ignore irrelevant information, reduce environmental interference, and improve the recognition accuracy of the crop target. In the process of improving the YOLOv5 network, a channel and spatial attention module is introduced. This module consists of two sub-modules: 1) Channel attention, which extracts the importance of each channel through global average pooling and global maximum pooling, assigns weights to different channels, and highlights the features related to the target; 2) Spatial attention, which assigns weights to each position in the feature map, identifies local regions of interest, and focuses the network on the spatial regions that may contain the crop target.

[0089] The improved YOLOv5 network with an attention mechanism consists of three parts: (1) Backbone: CSP-Darknet (Cross-Stage Partial), (2) Neck: PA-Net (Path Aggregation), (3) Head: YOLO layer. The process of the improved YOLOv5 network with an attention mechanism is as follows: The data is first input into CSP-Darknet for feature extraction, then input into PA-Net for feature fusion, and finally the YOLO layer outputs the detection results (category, score, location, size).

[0090] As Figure 3 shown, the bounding boxes of tomato fruits detected by the improved YOLOv5 network with an attention mechanism are shown in red, while the bounding boxes of the pedicels detected are sky blue, and the pedicel regions that may be obtained using prior knowledge are represented by black rectangles. In all cases except the most extreme ones, the estimated regions are obtained because the stem of the tomato cluster must be within the upper half region of the center of the tomato body. Each region is described by its upper left corner coordinates (x, y) and its height H and width W. The following formula describes the relationship between the region positions based on the data obtained in the greenhouse.

[0091]

[0092]

[0093] where W t , H t , (x t , y t ) are the width, height, and upper left corner coordinates of the minimum bounding rectangle of the fruit region, respectively. W e , H e , (x e , y e ) are the estimated position values of the pedicel, respectively. In addition, the picking point coordinates based on the detected pedicel are defined as follows:

[0094]

[0095] where W p , H p , (x p , y p ) are the width, height, and upper left corner coordinates of the minimum bounding rectangle of the detected pedicel, respectively, and (x pp , y pp ) are the picking point coordinates.

[0096] Step 3.2: Pair the resulting image data obtained after processing the RGB image data by the improved YOLOv5 network with attention mechanism with the depth image data for subsequent processing. To improve the detection effect, in addition to the RGB image data, the depth image data is used as an additional information to supplement the output of the improved YOLOv5 network with attention mechanism. The spatial information provided by the depth image data helps to perform more accurate crop target localization when detecting crops. There are usually noises, errors or incomplete information in the depth image data, so operations such as filtering, interpolation and denoising need to be carried out in the processing of the depth image data. The processed depth image data can provide more accurate crop target positions.

[0097] Step 3.3: After the improved YOLOv5 network with attention mechanism is trained, identify the crop targets in the image according to the class information, bounding boxes and confidence levels output by the improved YOLOv5 network with attention mechanism, and determine the positions of the crop targets. At this time, the improved YOLOv5 network with attention mechanism marks the identified crop targets in the image with bounding boxes to form the two-dimensional coordinates of the targets. The recognition process of the improved YOLOv5 network with attention mechanism is to extract multi-scale image features through CNN (Convolutional Neural Network); based on the feature maps, predict the centers, widths and heights (relative to the grid) of multiple bounding boxes within each grid, as well as the confidence levels indicating whether the target is included; each bounding box further predicts the classification probability and obtains the specific class through Softmax; filter and optimize the overlapping boxes according to the confidence levels, and only retain the optimal bounding boxes as the region of interest (ROI).

[0098] Step 3.4: Calculate the three-dimensional coordinates (x, y, z) of the crop targets through the depth image data of the crop targets and the camera calibration data, and align them with the coordinate system of the robotic arm. Through multiple feedback adjustments of the positions of the target and the robotic arm, ensure that the robotic arm can process the target at the most accurate angle and distance to avoid deviation. The depth camera projects the pixel points in the RGB image into the three-dimensional space by using the depth map and the camera internal parameters. The specific process is as follows: Each pixel point (u, v) in the RGB image corresponds to a depth value d in the depth map, indicating the distance from this point to the camera; through the camera internal parameters (focal lengths fx, fy and principal point coordinates cx, cy) and the focal length formula, convert the pixel coordinates into camera coordinates.

[0099] Noise suppression is achieved by introducing an attention mechanism and depth pairing. In agricultural scenarios, there are many cluttered backgrounds and environmental interferences (such as branches, leaves, and soil). The attention mechanism can highlight crop features, suppress irrelevant information simultaneously, and reduce false detections. At the same time, by focusing on features and preferentially paying attention to the target area, the network can capture key information more quickly and accurately, improving the robustness of detection. In addition, the three-dimensional positioning ability is enhanced. The improved YOLOv5 network is combined with depth information, and the attention mechanism is used to optimize feature expression, improving the calculation accuracy of the three-dimensional spatial position of the target.

[0100] Step 4: Detect the crop target within the target area. Analyze the pixels of the region of interest (ROI) based on the color threshold of the crop target, calculate the occlusion rate of the region of interest, and determine whether the target area is partially occluded, providing a basis for obstacle avoidance. The occlusion rate is the Figure 3 occulusion ratio in Figure 4 shows a schematic diagram of the method for calculating the occlusion rate and occlusion orientation of the region of interest in an embodiment of the present invention. The following combines Figure 4 to illustrate Step 4. Step 4 specifically includes the following three sub-steps:

[0101] Step 4.1: Extract the ROI region (c) of the crop target, that is, the fruit, from the recognition result (a) output by the improved YOLOv5 network with an attention mechanism. Extract the regions of the tomato cluster and leaves (d) through image processing methods, and combine with the depth map (b) to obtain the average depth D, area A, and the centroid coordinates P in the three-dimensional space of each region. By fusing depth information, further confirm the three-dimensional boundary of this region in the depth map to ensure that the target area is accurately defined in the actual physical space.

[0102] Figure 4 This operation crops the recognition result (a) of the YOLOv5 network and extracts the mask template of the tomato cluster from the tomato region (c). Then, morphological opening and closing operations are used to fill small gaps and eliminate small noises. To remove tomatoes in the background, calculate the area of each independent region and eliminate regions with an area < 50 pixels (d). Then multiply the obtained tomato mask template by the depth map to retain only the depth information of the tomatoes in the foreground (e).

[0103] Step 4.2: Analyze the pixels of the ROI region, convert them into pixels in the HSV color gamut, and iterate through all the pixels within the fruit region, with the value range (H: 0 - 10, S: 43 - 255, V: 46 - 255).

[0104] Step 4.3: Calculate the occlusion rate of the ROI region and determine whether the target area is partially occluded.

[0105] The occlusion rate R of the ROI regionO Calculated by the following formula:

[0106]

[0107] Where, A f and A O represent the areas of the entire fruit region and the shaded region respectively. 0.14 is an empirical value used to supplement the reduction caused by the in-cluster stems. With this information, it is possible to decide whether to move the robotic arm up, down, left or right to avoid the view point where the leaves obscure the target stem.

[0108] When calculating the depth of each region, the area and the centroid are determined by the following formula:

[0109]

[0110] In the formula, A f , A O and A li represent the areas of the entire fruit region, the shaded region and the independent leaf l i region respectively; O i is a value that determines whether the leaf l i is regarded as an occluder.

[0111] Occlusion judgment is based on the color threshold method of image segmentation, combined with the concept of ROI. By statistically analyzing the occlusion ratio of the target region, it evaluates whether the picking path is unobstructed. Its underlying principles include: taking advantage of the high distinguishability of the colors of the target crop (such as the color differences between the green leaves, brown trunks and the target parts), performing color feature segmentation in the image with the color domain as the threshold; quantifying the occlusion situation in the spatial dimension by calculating the pixel ratio, and statistically analyzing the occlusion information to clarify the interference degree of the obstacles to the target.

[0112] The accurate quantification of the occlusion situation is realized by introducing the color threshold occlusion judgment based on the ROI region. Compared with the simple detection based only on depth information, the color segmentation method combined with statistical analysis can accurately judge the type and degree of occlusion of the obstacles in the target region, improving the scientificity of the picking judgment; at the same time, the multi-modal fusion enhances the robustness. With the dual support of color information and depth data, the robotic arm is more adaptable to the recognition of occlusion and can effectively solve the complex occlusion problems in natural scenes; finally, real-time judgment and dynamic adjustment are carried out. By continuously calculating the occlusion rate, the robotic arm can dynamically plan the path or adjust the picking target to avoid failed picking due to obstacles or occlusion.

[0113] Step 5: Move the robotic arm based on the crop target information and actively sense new data. Once the crop target is located, the robotic arm needs to move to the appropriate picking position according to the position of the crop target. The robotic arm will continue to actively sense during the movement to ensure that obstacles on the path can be recognized and avoided in a timely manner. When the robotic arm approaches the crop target, a closed-loop control system is used for fine-tuning. Figure 5 shows a schematic diagram of active sensing in an embodiment of the present invention. The following will describe Step 5 in conjunction with Figure 5 , as follows. Step 5 specifically includes the following three sub-steps:

[0114] Step 5.1: Based on the three-dimensional coordinates of the crop target, the robotic arm control system calculates the shortest path from the current position of the robotic arm to the target position. In path planning, considering the movement range, joint angles, and speed parameters of the robotic arm, a kinematic model is used to calculate accurate control instructions. In an embodiment of the present invention, the robotic arm can be a Jaka Zu 5 robotic arm, and the kinematic model used is RRTconnect.

[0115] Step 5.2: During the movement of the robotic arm, the robotic arm control system uses a depth camera to monitor the images and depth data of the surrounding environment in real time. Whether there are new obstacles or crops is detected through the following steps: The depth camera continuously captures the images and depth data of the surrounding environment. The system processes the collected data, filters out noise and irrelevant information, extracts useful features, and then performs target detection and classification; According to the processed data compared with the previous data, it is analyzed whether there are new obstacles or target crops in the surrounding environment, and whether there are newly emerging objects or changes in the positions of the original objects; When the system detects a new obstacle, it will re-plan the movement path of the robotic arm according to the position and shape of the obstacle to ensure that the robotic arm can avoid the obstacle and continue to approach the crop target; During the entire movement of the robotic arm, the system will continuously update the data and monitor the changes in the surrounding environment in real time to adjust the path planning in a timely manner.

[0116] Step 5.3: When the robotic arm approaches the target, a closed-loop control system is used for fine-tuning. For example, using a vision guidance system, the robotic arm can perform fine positioning on the crop target to ensure precise contact with the crop. Actively sensing new data in real time and correcting the movement trajectory according to the new environmental information are the keys to ensuring that the robotic arm can stably reach the target position.

[0117] Figure 5Among them, the confidence level represents the degree of confidence of the model in the existence of a certain target. The YOLOv5 network will output a confidence value for each detected bounding box, which represents the probability of the target's existence and the accuracy of the bounding box localization. The YOLOv5 network will dynamically adjust the confidence level of the target according to environmental changes and perceptual data (such as depth information or subsequent detections). In step 5, the confidence level provided by YOLOv5 and the occlusion rate obtained in step 4 are used to drive the motion trajectory and picking decision of the robotic arm. At this time, the target position and the confidence level occlusion rate will affect the control strategy of the robotic arm. As the robotic arm approaches the target, the real-time feedback data will be re-input into the YOLOv5 network to update the target detection result and the confidence level.

[0118] Active perception is a dynamic perception and decision-making mechanism for path optimization based on real-time data feedback during the movement of the robotic arm. It uses a ToF camera and the YOLOv5 network to monitor obstacles on the path in real time, and feeds the environmental perception information back to the motion control system to form a closed-loop control and achieve real-time path adjustment; and realizes the combination of global and local planning. The global path is based on the pre-planned target position, and the local path is corrected according to the real-time perceived data to ensure adaptation to unexpected obstacles in a dynamic environment; it also realizes real-time optimization in picking obstacle avoidance. The robotic arm adjusts its position and posture when approaching the target, and actively perceives the depth, pose, and occlusion situation of the target to ensure picking accuracy.

[0119] By introducing active perception and occlusion orientation judgment, the dynamic adaptation ability of obstacle avoidance is improved. Active perception allows the robotic arm to adjust its strategy according to real-time environmental changes, such as temporarily avoiding obstacles or changing the picking angle, significantly improving adaptability and success rate; at the same time, the safety of obstacle avoidance is improved. The real-time detection of obstacles and path optimization avoid accidental collisions or failed pickings caused by obstacles, enhancing the reliability of operations; and reducing the interference between the robotic arm and the environment. By adjusting the action details in real time, the probability of misoperation of the robotic arm is reduced, and the target crop is better protected from damage.

[0120] Step 6: If the occlusion rate of the crop target reaches the threshold or the crop target has no obstacles, and the depth information of the crop target meets the picking conditions, the robotic arm performs the picking task. Step 6 specifically includes the following two sub-steps:

[0121] Step 6.1: If the requirements for picking are met after active perception and obstacle avoidance, that is, if the occlusion rate of the crop target reaches the threshold or the crop target has no obstacles, the robotic arm control system checks whether the distance between the crop target and the robotic arm is within the set safety range. If the distance is within the safety range, the robotic arm control system performs the picking task. If the distance is less than the safety range, the robotic arm control system issues a warning and stops moving.

[0122] Due to poor lighting at night and on cloudy and rainy days, there are shadow occlusions, resulting in poor object recognition effects. The calculated occlusion rate and confidence level deviate significantly from the actual situation. Even when supplementary lighting is carried out in case of insufficient lighting, shadows will be formed on the objects, increasing the recognition difficulty. Judging from the results, the occlusion rate and confidence level thresholds need to be divided into two categories: sunny days and nights or cloudy and rainy days. For sunny days, the occlusion rate threshold is set as a variable value. If the change value of the occlusion rate < -0.04 and the confidence level > 0.6, it is acceptable; for nights or cloudy and rainy days, the occlusion rate threshold is also set as a variable value. If the change value of the occlusion rate < -0.05 and the confidence level > 0.4, it is acceptable.

[0123] Active perception is mainly completed by a depth camera and a robotic arm. The depth camera perceives image and position information, and the robotic arm moves actively. The combination of the two forms an active perception strategy.

[0124] Step 6.2: If the occlusion rate of the crop target does not reach the threshold or there are obstacles to the crop target, the robotic arm control system re-executes the active perception and obstacle avoidance operation based on the crop target. The robotic arm control system continuously detects the crop target and adjusts the path according to the feedback information in real time.

[0125] Step 6 aims to establish a closed-loop control mechanism for continuous perception and action adjustment by repeating Steps 2 to 5. The dynamic process of obstacle avoidance depends on real-time feedback. The data currently sensed by the robotic arm is compared with the preset threshold to determine whether the environment is suitable for continued picking operations.

[0126] By repeating Steps 2 to 5 and based on the results of Step 5, the following are achieved: through repeated perception and dynamic adjustment, avoid failures caused by incomplete obstacle avoidance in the picking action, thus ensuring a smooth connection between obstacle avoidance and picking; multiple perception-adjustment closed-loop processes ensure that each operation decision is based on the latest environmental data, improving the response speed of the robotic arm to complex environments; continuous comparison and judgment mechanisms reduce the possibility of misjudgment, providing robust support for precise picking in complex environments.

[0127] The method for field picking obstacle avoidance based on active perception proposed by the present invention is suitable for complex field environments with obstacles (such as branches, vines) or severe target occlusions. For example: scenarios where the crop planting density is relatively high and there is an easy overlap between the target and non-target areas; scenarios where prismatic fruits (such as tomatoes, sweet peppers) are in the middle of the branches and obstacle avoidance is required to reach them.

[0128] The field picking obstacle avoidance method based on active perception proposed by the present invention uses an RGB camera and a depth camera to perceive the color, depth information and spatial position of crops in real time. Combining object recognition and segmentation of deep learning, it can accurately perceive occlusion, obstacles and object characteristics, and improve the recognition accuracy with an attention mechanism, adapting to complex fruit backgrounds and field environments. A real-time path planning and feedback mechanism is introduced, enabling the robotic arm to dynamically adjust its actions, bypass random or moving obstacles, and achieve continuous operation, thus flexibly coping with complex planting environments, such as problems where fruits are hidden behind branches and leaves or the occlusion dynamically changes. A closed-loop mechanism of active perception and dynamic adjustment is adopted to ensure that each action during the picking process is based on real-time updated information, improving the robustness and coherence of obstacle avoidance and picking. At the same time, it is suitable for deployment in agricultural production bases for unmanned operations, combined with robotic arms with automatic navigation or unmanned driving systems, to achieve large-area continuous picking under unmanned management conditions.

[0129] In an embodiment of the present invention, the present invention also provides a field picking obstacle avoidance system based on active perception. Figure 6 The schematic diagram of the field picking obstacle avoidance system based on active perception according to an embodiment of the present invention is shown. As Figure 6 shown, the system includes the following modules:

[0130] An initialization module, configured to initialize the hardware and the robotic arm control system, where the hardware includes a camera and a robotic arm;

[0131] An image data acquisition module, configured to acquire the RGB image data and depth image data of the crops and perform preprocessing;

[0132] A YOLOv5 network module, configured to input the preprocessed RGB image data and depth image data of the crops into an improved YOLOv5 network with an attention mechanism, perform positioning on the crop target and extract the spatial position of the crop target;

[0133] An occlusion rate calculation module, configured to detect the crop target within the target area, analyze the pixels in the region of interest based on the color threshold of the crop target, calculate the occlusion rate and determine whether the target area is partially occluded, providing a basis for obstacle avoidance;

[0134] An active perception module, configured to move the robotic arm based on the crop target information. The robotic arm will continue to perform active perception during the movement to ensure that obstacles on the path can be recognized and avoided in a timely manner. When the robotic arm approaches the crop target, a closed-loop control system is used for fine-tuning; and

[0135] A picking module, configured to if the occlusion rate of the crop target reaches the threshold or there are no obstacles for the crop target, and the depth information of the crop target meets the picking conditions, then the robotic arm performs the picking task.

[0136] In one embodiment of the present invention, the present invention further provides a computer system, which includes a processor, a graphics card with an artificial intelligence chip, and a memory. The memory is configured to store machine-readable instructions, the graphics card is configured to train the field picking obstacle avoidance method based on active perception, and the processor is configured to execute the machine-readable instructions. When the processor and / or the graphics card execute the machine-readable instructions, the following processing steps are implemented: initialize the hardware and the robotic arm control system, where the hardware includes a camera and a robotic arm; obtain the RGB image data and depth image data of the crop and perform preprocessing; input the preprocessed RGB image data and depth image data of the crop into an improved YOLOv5 network with an attention mechanism to locate the crop target and extract the spatial position of the crop target; detect that the crop target is within the target area, analyze the pixels in the region of interest based on the color threshold of the crop target, calculate the occlusion rate of the region of interest and determine whether the target area is partially occluded to provide a basis for obstacle avoidance; move the robotic arm based on the crop target information, and the robotic arm will continue to perform active perception during the movement to ensure that obstacles on the path can be recognized and avoided in a timely manner. When the robotic arm approaches the crop target, a closed-loop control system is used for fine-tuning; and if the occlusion rate of the crop target reaches the threshold or there are no obstacles for the crop target, and the depth information of the crop target meets the picking conditions, the robotic arm performs the picking task.

[0137] The graphics card may preferably be a graphics card with a GPU computing power higher than model 5.0. Since the amount of data to be trained is large, providing a graphics card configuration can significantly improve the training speed.

[0138] The memory includes various media that can store machine-readable instructions, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disc.

[0139] It can be understood that in addition to the memory and the processor described above, the above computer system further includes other software and hardware components not listed in this specification. Specifically, it can be determined according to the model of the specific data processing device in different application scenarios, and this specification will not list and elaborate one by one.

[0140] In one embodiment of the present invention, the present invention further provides a computer-readable storage medium, on which machine-readable instructions are stored, and the machine-readable instructions, when executed by a processor, implement the following processing steps: initializing the hardware and the robotic arm control system, where the hardware includes a camera and a robotic arm; acquiring RGB image data and depth image data of the crop and performing preprocessing; inputting the preprocessed RGB image data and depth image data of the crop into an improved YOLOv5 network with an attention mechanism to perform localization on the crop target and extract the spatial position of the crop target; detecting that the crop target is within the target area, analyzing the pixels in the region of interest based on the color threshold of the crop target, calculating the occlusion rate of the region of interest and determining whether the target area is partially occluded to provide an obstacle avoidance basis; moving the robotic arm based on the crop target information, and the robotic arm will continue to perform active perception during the movement to ensure that obstacles on the path can be recognized and avoided in a timely manner. When the robotic arm approaches the crop target, a closed-loop control system is used for fine-tuning; and if the occlusion rate of the crop target reaches the threshold or there are no obstacles for the crop target, and the depth information of the crop target meets the picking conditions, the robotic arm performs the picking task.

[0141] Although the embodiments of the present invention have been described above, it should be understood that they are presented only as examples and not as limitations. It will be apparent to those skilled in the relevant art that various combinations, variations, and changes can be made to them without departing from the spirit and scope of the present invention. Therefore, the width and scope of the present invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined according to the technical solution of the present invention and its equivalent replacements.

Claims

1. A field picking obstacle avoidance method based on active perception, characterized in that, Including the following steps: Initialize the hardware and the robotic arm control system, where the hardware includes a camera and a robotic arm; Obtain the RGB image data and depth image data of the crop, and perform preprocessing; Input the preprocessed RGB image data and depth image data of the crop into an improved YOLOv5 network with an attention mechanism, perform localization on the crop target, and extract the spatial position of the crop target; Detect whether the crop target is within the target area, analyze the pixels in the region of interest based on the color threshold of the crop target, calculate the occlusion rate of the region of interest, and determine whether the target area is partially occluded, providing a basis for obstacle avoidance; Move the robotic arm based on the crop target information. The robotic arm will continue to perform active perception during the movement to ensure that obstacles on the path can be recognized and avoided in a timely manner. When the robotic arm approaches the crop target, a closed-loop control system is used for fine-tuning; And If the occlusion rate of the crop target reaches the threshold or there are no obstacles for the crop target, and the depth information of the crop target meets the picking conditions, the robotic arm performs the picking task.

2. The method for avoiding obstacles in field picking based on active perception according to claim 1, characterized in that, Initializing the hardware and the robotic arm control system, where the hardware includes a camera and a robotic arm includes: The hardware further includes an external sensor. Initialize the camera and the external sensor. The camera includes an RGB camera and a depth camera. Set the resolution, frame rate, and exposure parameters of the RGB camera and the depth camera; Initialize the robotic arm control system, initialize the robotic arm, and set the motion range, joint angles, and load capacity parameters of the robotic arm; and Ensure that the states of the hardware and the robotic arm control system meet the requirements.

3. The method for avoiding obstacles in field picking based on active perception according to claim 1, characterized in that Obtaining the RGB image data and depth image data of the crop, and performing preprocessing includes: Use the RGB camera to obtain the RGB image data of the crop and the surrounding environment. The RGB image data includes appearance, color, and texture; Use the depth camera to obtain the depth image data of the crop and the surrounding environment, and perform noise processing and filtering on the depth image data; Perform data synchronization on the RGB image data and the depth image data through timestamps, triggers, or software algorithms; and Perform preprocessing on the RGB image data and the depth image data. The preprocessing includes image denoising, image cropping, color normalization, and scale correction.

4. The method for avoiding obstacles in field picking based on active perception according to claim 1, characterized in that Inputting the preprocessed RGB image data and depth image data of the crop into an improved YOLOv5 network with an attention mechanism, performing localization on the crop target, and extracting the spatial position of the crop target includes: Input the preprocessed RGB image data of the crop into an improved YOLOv5 network with an attention mechanism. The improved YOLOv5 network with an attention mechanism includes CSP-Darknet for feature extraction, PA-Net for feature fusion, and a YOLO layer for outputting detection results; Pair the result image data obtained after the improved YOLOv5 network with an attention mechanism processes the RGB image data with the depth image data; Identify the crop target in the image according to the class information, bounding box, and confidence of the detection results output by the improved YOLOv5 network with an attention mechanism, and determine the position of the crop target; and Based on the depth image data and camera calibration data of the crop target, calculate the three-dimensional coordinates of the crop target and align them with the coordinate system of the robotic arm.

5. The method for avoiding obstacles during field picking based on active perception according to claim 1, characterized in that, Detect that the crop target is within the target area, analyze the pixels in the region of interest based on the color threshold of the crop target, calculate the occlusion rate of the region of interest, and determine whether the target area is partially occluded. The basis for obstacle avoidance is provided as follows: Extract the region of interest of the crop target from the recognition results output by the improved YOLOv5 network with an attention mechanism. Extract the shadow area and leaf area of the crop target through image processing methods, and combine the depth image data to obtain the average depth, area, and centroid coordinates in three-dimensional space for each region; Analyze the pixels in the region of interest and convert them into pixels in the HSV color gamut; and Calculate the occlusion rate of the region of interest and determine whether the target area is partially occluded; Occlusion rate R of the region of interest O which is calculated by the following formula Among them, A f and A O respectively represent the area of the region of interest of the crop target and the shadow region of the crop target.

6. The method for avoiding obstacles in field picking based on active perception according to claim 1, wherein Move the robotic arm based on the crop target information. The robotic arm will continue to perform active perception during the movement to ensure that obstacles on the path can be recognized and avoided in a timely manner. When the robotic arm approaches the crop target, a closed-loop control system is used for fine-tuning, including: Based on the three-dimensional coordinates of the crop target, the robotic arm control system calculates the shortest path from the current position of the robotic arm to the target position. In path planning, considering the movement range, joint angles, and speed parameters of the robotic arm, a kinematic model is used to calculate accurate control instructions; During the movement of the robotic arm, the robotic arm control system uses a depth camera to monitor the image and depth data of the surrounding environment in real time, detect whether there are new obstacles or crops. If new obstacles or crops are detected, the robotic arm control system adjusts the path or re-plans the path to ensure that the robotic arm can avoid obstacles and continue to approach the crop target; and When the robotic arm approaches the target, a closed-loop control system is used for fine-tuning, including using a vision guidance system to guide the robotic arm to perform fine positioning on the crop target to ensure that the robotic arm precisely contacts the crop target.

7. The method for avoiding obstacles in field picking based on active perception according to claim 1, wherein If the occlusion rate of the crop target reaches the threshold or there are no obstacles for the crop target, and the depth information of the crop target meets the picking conditions, the robotic arm performs the picking task, including: If the occlusion rate of the crop target reaches the threshold or there are no obstacles for the crop target, the robotic arm control system checks whether the distance between the crop target and the robotic arm is within the set safety range. If the distance is within the safety range, the robotic arm control system performs the picking task. If the distance is less than the safety range, the robotic arm control system issues a warning and stops the movement; and If the occlusion rate of the crop target does not reach the threshold or there are obstacles for the crop target, the robotic arm control system re-performs the active perception and obstacle avoidance operation based on the crop target. The robotic arm control system re-detects the crop target and adjusts the path in real time according to the feedback information.

8. A system for the active perception-based field picking obstacle avoidance method according to any one of claims 1-7, characterized in that, It includes the following modules: An initialization module configured to initialize the hardware and the robotic arm control system, where the hardware includes a camera and a robotic arm; An image data acquisition module configured to acquire the RGB image data and depth image data of the crop and perform preprocessing; The YOLOv5 network module is configured to input the RGB image data and depth image data of the preprocessed crop into the improved YOLOv5 network with an attention mechanism, perform localization on the crop target, and extract the spatial position of the crop target; The occlusion rate calculation module is configured to detect the crop target within the target area, analyze the pixels in the region of interest based on the color threshold of the crop target, calculate the occlusion rate, and determine whether the target area is partially occluded, providing a basis for obstacle avoidance; The active perception module is configured to move the robotic arm based on the crop target information. The robotic arm will continue to perform active perception during the movement to ensure that obstacles on the path can be recognized and avoided in a timely manner. When the robotic arm approaches the crop target, a closed-loop control system is used for fine-tuning; And The picking module is configured to, if the occlusion rate of the crop target reaches the threshold or there are no obstacles for the crop target, and the depth information of the crop target meets the picking conditions, the robotic arm performs the picking task.

9. A computer system, characterized in that, Including: A processor configured to execute machine-readable instructions; A graphics card with an artificial intelligence chip, configured to train the field picking obstacle avoidance method based on active perception; And A memory configured to store machine-readable instructions, the machine-readable instructions, when executed by the processor and / or the graphics card, perform the steps of the method according to one of claims 1-7.

10. A computer-readable storage medium, characterized in that, On which machine-readable instructions are stored, the machine-readable instructions, when executed by the processor, perform the steps of the method according to one of claims 1-7.

Images