Multi-view target active sensing method and system, and storage medium

Abstract

The invention provides a multi-view target active sensing method and system, and a storage medium, and the method comprises the steps: S1, collecting an RGBD image of a target scene, and determining a target initial position through a target detection model; s2, constructing a picking task sequence according to the initial position of each target, and planning a motion path of a mechanical arm; and S3, according to the planned motion path of the mechanical arm, carrying the view finding device to a target position, collecting an RGBD image of the target, and executing target key point shielding judgment and a view angle active optimization strategy until an optimal target RGBD image is obtained. Therefore, the problem that good depth information of the target cannot be accurately extracted due to target shielding is solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a multi-perspective target active perception method and system for a hand-eye system, and a storage medium. Background Art

[0002] In modern agricultural production, as people's requirements for quality and yield continue to increase, and labor costs gradually rise, how to achieve automated harvesting of agricultural targets such as fruits and flowers has become an important direction for the development of agricultural automation.

[0003] Currently, in the field of agricultural automation, when dealing with stacked targets such as wolfberry picking and small shrub and flower picking, in irregular and dynamic environments, due to the mutual stacking and occlusion between targets, the traditional agricultural robot's single-view single-shot target detection method cannot accurately extract the best target depth information, which in turn affects the subsequent 3D position detection of target key points and interferes with the accuracy of picking point generation.

[0004] For example, existing keypoint detection-based fruit picking localization solutions typically rely on deep learning models to identify key points of the fruit in an image (such as the stem and top) and integrate them with a depth camera for 3D localization. For example, by processing the RGB image, the 2D coordinates of the key points are directly regressed, and then the 3D pose is estimated by combining the depth map information. (Common models include the YOLOv5 / YOLOv8-Pose series and RT-DETR, whose feature extraction backbones are often based on ResNet, EfficientNet, or Transformer architectures.)

[0005] However, this approach relies heavily on a single-view image for keypoint recognition, making it susceptible to factors such as leaf occlusion, uneven lighting, and background interference. Detection accuracy decreases significantly when the fruit stem is partially obscured. Consequently, the approach lacks a supplementary perception mechanism for uncertain areas, significantly impacting the accuracy of picking point generation. Summary of the Invention

[0006] To this end, the main purpose of the present invention is to provide a multi-perspective target active perception method and system, and a storage medium to solve the problem that target occlusion makes it impossible to accurately extract the target's optimal depth information.

[0007] To achieve the above object, according to one aspect of the present invention, a multi-view target active perception method is provided, the steps of which include:

[0008] Step S1 collects an RGBD image of the target scene and determines the initial position of the target through the target detection model;

[0009] Step S2 constructs a picking task sequence based on the preliminary positions of each target and plans the motion path of the robotic arm;

[0010] In step S3, according to the planned motion path of the robotic arm, the robot carries the viewfinder to the target position, collects the RGBD image of the target, and performs target key point occlusion judgment and active viewing angle optimization strategy until the best target RGBD image is obtained.

[0011] In a possible preferred embodiment, the steps of constructing a picking task sequence and planning a robot arm motion path include:

[0012] Step S21 generates a preliminary picking sequence as the initial path using a hierarchical task sequence optimization algorithm based on the obtained preliminary target position;

[0013] Step S22 defines the node connection lines in the initial path as flight segments, and creates ellipses with the manipulator's operable width of each flight segment as the minor axis length of the ellipse; deletes the overlapping areas between all ellipses to create a multi-ellipse space;

[0014] Step S23: Based on the multi-elliptical spatial area restriction, the RRT algorithm is used to adjust the initial path to obtain the planned robot arm motion path.

[0015] In a possible preferred embodiment, the steps of target key point occlusion judgment and viewpoint active optimization strategy include:

[0016] Step S31 uses the target detection model to identify the target key points in the RGBD image and detect their edge sharpness and confidence;

[0017] Step S32 adjusts the robot arm posture, switches the viewing angle, collects RGBD images, and executes step S31 to obtain the edge sharpness and confidence of the target key points under different viewing angles;

[0018] Step S33 compares the edge sharpness and confidence of each perspective and selects the best target RGBD image.

[0019] In a possible preferred embodiment, the target key points include: the pedicel and flower top of small shrub flowers, or the fruit stalk and fruit top of small fruits.

[0020] To achieve the above object, according to another aspect of the present invention, a multi-view target active perception method is provided, the steps of which include:

[0021] Step S1 collects an RGBD image of the target scene and determines the initial position of the target through the target detection model;

[0022] Step S2 constructs a picking task sequence based on the preliminary positions of each target and plans the motion path of the robotic arm;

[0023] Step S3: According to the planned motion path of the robotic arm, the robot moves to the target location with the viewfinder, collects the RGBD image of the target, and performs target key point occlusion judgment and active view angle optimization strategy until the best target RGBD image is obtained;

[0024] Step S4 detects the best target RGBD image through the image analysis model, obtains the two-dimensional coordinates of at least any one of the target and its key points, and obtains the corresponding spatial coordinates according to the back-projection method.

[0025] In a possible preferred embodiment, the steps of constructing a picking task sequence and planning a robot arm motion path include:

[0026] Step S21 generates a preliminary picking sequence as the initial path using a hierarchical task sequence optimization algorithm based on the obtained preliminary target position;

[0027] Step S22 defines the node connection lines in the initial path as flight segments, and creates ellipses with the manipulator's operable width of each flight segment as the minor axis length of the ellipse; deletes the overlapping areas between all ellipses to create a multi-ellipse space;

[0028] Step S23: Based on the multi-elliptical spatial area restriction, the RRT algorithm is used to adjust the initial path to obtain the planned robot arm motion path.

[0029] In a possible preferred embodiment, the steps of target key point occlusion judgment and viewpoint active optimization strategy include:

[0030] Step S31 uses the target detection model to identify the target key points in the RGBD image and detect their edge sharpness and confidence;

[0031] Step S32 adjusts the robot arm posture, switches the viewing angle, collects RGBD images, and executes step S31 to obtain the edge sharpness and confidence of the target key points under different viewing angles;

[0032] Step S33 compares the edge sharpness and confidence of each perspective and selects the best target RGBD image.

[0033] In a possible preferred embodiment, the target key points include: the pedicel and flower top of small shrub flowers, or the fruit stalk and fruit top of small fruits.

[0034] In order to achieve the above-mentioned purpose, according to one aspect of the present invention, a multi-perspective target active perception system is provided, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the system implements the steps of any of the multi-perspective target active perception methods described above.

[0035] In order to achieve the above-mentioned purpose, according to one aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed, the steps of any of the multi-perspective target active perception methods described above are implemented.

[0036] Through the multi-perspective target active perception method, system and storage medium provided by the present invention, the linkage of target key point occlusion judgment and perspective active optimization strategy is cleverly designed to actively adjust the robot arm posture and obtain the optimal depth map of the target. Compared with the traditional single-time, single-perspective target detection and positioning scheme, this has stronger robustness and high precision, and can still stably identify and locate targets in natural occlusion and high-density scenarios. In addition, this scheme also cleverly establishes a unified task coordination mechanism between the detection of target key points and the picking path planning, so it can cope with the rapid changes in target posture and occlusion conditions in complex environments. The key point deviation caused by this further amplifies the defect of path error, thereby effectively solving the positioning interference problem caused by target density. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0038] Figure 1 Schematic diagram of the steps of the multi-view target active perception method of the present invention;

[0039] Figure 2 This is a schematic diagram of the process of coarse positioning recognition and semantic mapping in the multi-view target active perception method of the present invention;

[0040] Figure 3 Schematic diagram of the process of solving the task sequence optimization problem in the multi-view target active perception method of the present invention;

[0041] Figure 4 Schematic diagram of the fine positioning and occlusion perception process in the multi-view target active perception method of the present invention;

[0042] Figure 5 This is a schematic diagram of the process of using the multi-view active target perception method of the present invention for greenhouse tomato picking tasks;

[0043] Figure 6 This is a schematic diagram of the structure of the multi-view target active perception system of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the technical solutions of the present invention, the specific technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments to help those skilled in the art further understand the present invention. Obviously, the embodiments described in this case are only embodiments of a part of the present invention, rather than all embodiments. It should be pointed out that, for those of ordinary skill in the art, the embodiments in this application and the features in the embodiments can be combined with each other without departing from the concept of the present invention and without conflicting with each other. Based on the embodiments in the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative work should fall within the disclosure and protection scope of the present invention.

[0045] In addition, the terms "first," "second," "S1," "S2," and the like in the specification, claims, and drawings of the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential sequence. It should be understood that the features used in this manner are interchangeable where appropriate, such that the embodiments of the present invention described herein can be implemented in an order other than that described herein. Furthermore, the stages described in each step are not necessarily implemented in the same step. It should be understood that the order of implementation of the contents of each step can be adjusted and interchanged without violating the inventive concept, such that the step embodiments of the present invention described herein can be implemented in an order other than that described herein. Furthermore, the terms "including," "having," and any variations thereof, as used herein, are intended to cover non-exclusive inclusions. Unless otherwise expressly specified or limited, the terms "disposed," "arranged," "installed," "connected," and "connected" should be interpreted broadly, for example, to mean fixed, removable, or integral; mechanical or electrical; direct or indirect through an intermediary; or internal communication between two components. Those skilled in the art will understand the specific meanings of these terms in this case based on specific circumstances and in conjunction with prior art.

[0046] In traditional agricultural automation, when dealing with tiny stacked targets such as wolfberry picking and small shrub and flower picking, in irregular and dynamic environments, due to the stacking of targets, the traditional agricultural robot's single-view single-shot target detection scheme cannot extract the best target depth information, which in turn affects the detection of the target's 3D key point position (such as the pedicel and flower top of small shrub and flower, or the pedicel and fruit top of small fruit) and 6D pose estimation, and interferes with the accuracy of picking point generation.

[0047] Therefore, in order to solve the problem that the target is blocked and the target's best depth information cannot be accurately extracted, such as Figures 1 to 5 As shown, the present invention proposes a multi-view target active perception method, the exemplary steps of which include:

[0048] Step S1 collects the RGBD image of the target scene and determines the initial position of the target through the target detection model.

[0049] Specifically, this step is mainly aimed at coarse perception of the environment. Based on the rapid and real-time target detection of multiple targets within the field of view, the system locates the target position of the preliminary recognition. In this embodiment, the example uses an out-of-hand depth camera to complete coarse perception of the environment to collect a panoramic RGBD image of the environment, and then combines it with a deep learning target detection model to complete preliminary target detection. It also constructs an environmental semantic map through pixel-level semantic segmentation to obtain global semantic information and preliminary target location.

[0050] For example, based on the target detection model fine-tuned in agricultural target scenes, the RGB information is annotated with contours and detection frames. By performing pixel-level classification and segmentation on the image, a detailed environmental semantic map is constructed to quickly locate the approximate position of the target in the image, guiding the construction of subsequent task maps. For example, the yolov8m model is used for pre-training on images of the overall crop growth environment taken by an off-hand RGBD camera. During pre-training, the image data is appropriately enhanced, such as by modifying brightness and contrast, to simulate changes in lighting conditions, in order to enhance the versatility of the system. Furthermore, the position of the target in the image is appropriately clipped and reorganized to expand the adaptability of the pre-training.

[0051] This part of the network outputs the coordinates of the target's detection frame center. After hand-eye calibration with the eye outside the hand, it selects the coordinates of the detection frame center point and its k nearest neighbors within an appropriate range. The average 3D coordinates of these coordinates, projected onto a 3D point cloud with the robot base as the origin and after outlier removal, are calculated. Filtering and data correction techniques are used for smoothing and noise reduction to ensure the stability and accuracy of the coarse positioning information. The target position is initially located. The system uses a rigid coordinate transformation method to map the target center point and depth information in the camera coordinate system to the robotic arm's workspace. It then outputs the target's position in three-dimensional space and uploads it to the ROS system for subscription, providing data support for subsequent robotic arm motion planning and precise operation.

[0052] Considering that in the prior art, the detection of target key points and downstream tasks such as picking path planning are usually performed separately, and no unified task coordination mechanism has been established, it is difficult to cope with the rapid changes in target posture and occlusion conditions in complex environments. As a result, key point deviations further amplify path errors, resulting in a reduction in the overall operating efficiency of the system. Positioning interference is particularly prominent in areas with dense fruit targets. To this end, in the following example, the present invention specifically establishes a unified task coordination mechanism between the detection of target key points and picking path planning to address the positioning interference problem caused by dense targets.

[0053] Step S2 constructs a picking task sequence based on the preliminary positions of each target and plans the motion path of the robotic arm.

[0054] Specifically, this step mainly constructs a picking task graph based on the coarse perception semantic graph of the environment, applies a genetic algorithm to generate a picking sequence, and combines the RRT algorithm to optimize the path to achieve obstacle avoidance and efficient planning. Figure 2 As shown, the example constructs a picking task graph based on the target semantic environment graph and preliminary target positions (e.g., 3D center point location information) given in step S1. This example abstracts the traversal problem of multiple picking points in the picking task into a task sequence optimization problem (TSP). This problem considers multiple factors, including the time window for the task, path continuity, and robotic arm motion constraints. An optimization model is established that incorporates target positions, task costs, and motion constraints, thereby achieving hierarchical optimization from coarse sorting to refined path refinement. In the coarse sorting stage, a genetic algorithm is used to rapidly generate a preliminary picking sequence. In the fine sorting stage, each path segment is refined and adjusted using algorithms such as the RRT algorithm, and local motion planning is performed to ensure optimal obstacle avoidance and picking efficiency.

[0055] For example, by subscribing to coarse positioning information from the ROS system, a hierarchical task sequence optimization algorithm is used to generate an optimal picking path, guiding the robot to efficiently reach the target area. This example assumes a greenhouse environment where the robot primarily travels in a straight line, with fruit plants arranged in two rows on either side of the agricultural robot. This simplifies the path sorting of multiple 3D coordinates.

[0056] In this stage of the rough sorting process, the example uses a genetic algorithm to quickly generate a preliminary picking sequence, such as Figure 3 As shown, the process example is as follows:

[0057] 1) First, only keep the coordinates within the robot's workspace , generate a number i for each coordinate. And generate the coordinates of the observation point for fine positioning based on the optimal pose estimation .

[0058] 2) Initialize the population, fix the coordinate farthest from the base as the end point of the path, and randomly generate paths for the remaining coordinates. That is, path = [p1, p2, ..., pn-1, pn], where pn is the farthest point (fixed), but the GA optimizes the order of [p1, ..., pn-1].

[0059] 3) For the fitness function: fitness(path) = total path length (start → obs1 → tgt1 →obs2 → tgt2 → ... → obsn → tgtn) (Euclidean distance, including the full cost of the farthest point) + penalty term (increased each time the left or right row is switched;).

[0060] 4) Iterate through sequential crossover (OX) and mutation to obtain a roughly sorted sequence, that is, generate a preliminary picking sequence as the initial path.

[0061] The following is an example process of fine-tuning each path segment using the RRT algorithm:

[0062] 1) After obtaining a roughly sorted sequence as the initial path, a multi-ellipse region is created as the designated space. The connecting lines of the nodes in the initial path are defined as segments, and ellipses are created with the minimum width of each segment (i.e., the robot arm's operating width) as the minor axis length. By removing the overlapping areas between all ellipses, a multi-ellipse space is created, which serves as the designated space for the RRT algorithm to search for the final path.

[0063] 2) Initialize the RRT tree, that is, create an empty path tree with the starting point as the root node, and set the algorithm parameters, such as the maximum number of iterations, connection distance r, obstacle radius, etc. Set the starting point and target point, using tgt1 and tgtn in the initial path as the starting point and target point of the task, respectively.

[0064] 3) Randomly select a point within the specified space in step 1 as the sampling point qrand. Find the node (qnear) closest to qrand in the path tree. Select a point between qnear and qrand with a distance r from qnear (i.e., the connection distance parameter r requires a pre-set connection distance) as a new node (qnew). Check whether the line between qnew and qnear overlaps with an obstacle. If so, discard qrand and select a new sampling point. If not, add qnew to the path tree and set its parent node to qnear.

[0065] 4) Among the nodes within a certain radius from qnew, find the node with the minimum cost as the best parent node. Compare the cost of the nodes within this radius with the cost of connecting them as children of qnew. If the cost of connecting a node as a child is lower, disconnect the existing link to the parent node and set it as a child of qnew.

[0066] 5) Repeat steps 3) and 4) until the target point is reached. Generate a graph connecting the nodes generated by the RRT algorithm and remove any overlap with obstacles to obtain a collision-free spatial graph. Search the collision-free spatial graph and use the Dijkstra algorithm to find the shortest path connecting the starting and ending points. This path is then used as the final path.

[0067] In step S3, according to the planned motion path of the robotic arm, the robot carries the viewfinder to the target position, collects the RGBD image of the target, and performs target key point occlusion judgment and active viewing angle optimization strategy until the best target RGBD image is obtained.

[0068] Specifically, such as Figure 4 As shown in the figure, this step is used for optimal perspective estimation and active perspective optimization. For example, according to the picking order planned in the picking task graph, the optical center optimization model is used to obtain the best shooting pose for the target to be picked, guide the system robot arm to move to the specified pose, and use the RGBD camera on hand to collect fine images. When occlusion is identified, the active perception module is called to switch to the alternative pose.

[0069] This step focuses on guiding the robotic arm to sequentially approach the target, following the picking sequence and plan outlined in step S2. Based on the optical center optimization model, the optimal shooting pose is determined for the current target. The optimal camera shooting coordinates are calculated using a back-projection formula. The optimal pose for the hand camera is then determined, taking into account the target pose and the reach of the robotic arm. The robotic arm is guided to approach this pose for refined capture, obtaining optimal RGBD information from the optimal perspective and performing detailed recognition. When occlusion is detected, the active perception module is invoked to switch to an alternative pose.

[0070] It's important to note that this example utilizes a fixed-focus 3D camera lens, characterized by a fixed imaging plane. This characteristic optimizes depth measurement accuracy at the camera's optical center under unobstructed conditions. Therefore, using the back-projection formula to calculate the optimal camera coordinates outside the target coordinates, and combining the target pose and the reach of the robotic arm, we can determine the optimal pose for the handheld camera, optimizing perception accuracy.

[0071] For occlusion recognition, if the above-mentioned target and key point recognition results in reduced detection accuracy and confidence due to occlusion, an active perspective adjustment strategy can be adopted to obtain an unobstructed perspective through alternative perception postures. Based on the real-time feedback of the target detection system, the system automatically switches to different perception postures to obtain the highest target detection confidence, thereby achieving accurate capture of the target key points.

[0072] For example, the aforementioned optical center optimization model implements a strategy for determining target key point occlusion and actively optimizing the viewing angle. This strategy aims to select a pose in which the target is unobstructed and a clear and complete image is captured. For example, based on coarse positioning information, the robotic arm moves the hand-mounted camera to a position approximately 0.6 meters from the target on the line connecting the external camera and the target, placing the target in the center of the hand-mounted camera image. For any obscured target captured in this shot, the robot moves the camera upwards by 30 degrees (maintaining the 0.6-meter mark, with the image centered) and re-images until a complete image is captured or multiple candidate poses (multi-directional 30-degree angle movements) have been explored.

[0073] This example uses the following criteria to determine whether an image is occluded: Occlusion often causes unclear edges, so the Laplacian operator or LoG operator in OpenCV is used to detect edge sharpness of the target within the detection frame. Several viewpoints with the highest confidence and edge sharpness, such as those from Yolov8, are selected. If the confidence at a single viewpoint is lower than the expected value, such as 0.4, and the calculated edge sharpness is lower than that of other viewpoints, the target is considered occluded. Alternative viewpoints are obtained by actively adjusting the camera position, and this process is repeated iteratively. The new local image information is combined to re-execute the viewpoint optimization and subsequent fine-grained positioning process.

[0074] Furthermore, in order to achieve spatial positioning of the target and its key points after active multi-view target perception and provide spatial positioning information for automated picking, in an optional embodiment, the method steps of this example further include:

[0075] Step S4 detects the best target RGBD image through the image analysis model, obtains the two-dimensional coordinates of at least any one of the target and its key points, and obtains the corresponding spatial coordinates according to the back-projection method.

[0076] Specifically, this step mainly targets the obtained optimal viewing angle RGBD information to complete fine recognition and positioning. Conceptually, this image analysis model combines target detection and key point detection modules to analyze target key points, and combines the depth map to accurately locate the target and key points. Here, the image analysis model for fine recognition preferably uses the Yolov8-segment pre-trained model based on Yolov8x.pt, and is fine-tuned based on the RGB dataset captured at close range, marking relevant key point data such as the target stem cutting point (including the pedicel and flower top of small shrubs and flowers, or the fruit pedicel and fruit top of small fruits).

[0077] The acquired RGB detection results are then used to calculate the target's center pixel coordinates and keypoint coordinates. For the depth map, median filtering or local mean calculation is used to suppress noise and obtain a robust depth estimate. Combining this depth information with camera intrinsic parameters (including focal length, principal point, and distortion parameters), the two-dimensional pixel coordinates are mapped to three-dimensional spatial coordinates based on the back-projection principle to guide subsequent 3D pose estimation and visual servoing.

[0078] For example, after obtaining the optimal shooting angle, the handheld RGBD camera is used to segment and locate the object again, obtaining complete target features and key point information. The system uses the eye-on-hand back-projection formula, which maps 2D pixel coordinates to 3D spatial coordinates to guide subsequent 3D pose estimation and visual servoing.

[0079] In this step, it is preferred to deploy and train on Yolov8-seg, based on the YOLOv8x model, to detect contours and key points in near-field target images. During pre-training, the LabelMe tool can be used to finely label the dataset, outlining the target contours and key point information. Data augmentation is performed using the same pre-processing methods as in the previous step. The output contour edge information or detection box information is projected onto a 3D point cloud as the edge of the viewing cone. Outlier removal and point cloud clustering are then performed to extract a detailed 3D point cloud of the visible portion of the target and its key points.

[0080] Example

[0081] In order to verify the application effect of the present invention in actual agricultural scenes, the following Figure 5 The greenhouse tomato picking task shown is used to specifically illustrate the multi-view target active perception method of the present invention.

[0082] At a certain agricultural base, rows of tomato plants were planted in a greenhouse. The fruits grew in clusters lateral to the main stem, forming a dangling pattern. The fruits were heavily obscured from one another, with the stalks hidden behind leaves or upper fruit. Furthermore, the cluster structure posed challenges for identifying continuity and separating objects.

[0083] The robot platform configuration is as follows:

[0084] 1. Mobile platform: tracked chassis with autonomous navigation function;

[0085] 2. Dual-DOF picking robot: RGBD camera (hand-mounted camera) installed at the end;

[0086] 3. Handheld RGBD camera: installed above the vehicle body, used for large-scale panoramic perception.

[0087] Stage 1: Coarse Perception of the Environment

[0088] After the robot starts, the handheld RGBD camera captures panoramic RGBD images of the fruit plants on both sides. A fine-tuned YOLOv8m model is used to identify the tomato bunches, extracting the overall outline and approximate center coordinates of each tomato bunch.

[0089] Semantic segmentation was combined to construct a semantic map of the environment, and preliminary 3D coordinates were obtained through backprojection. This batch detected 14 objects with an average confidence level of 0.74. The coordinates were then mapped to the robot base coordinate system using external camera calibration.

[0090] Phase 2: Construction of the picking task graph

[0091] The system combines the target location with the workspace constraints to form a picking task map. The path optimization is as follows:

[0092] 1. Generate an initial picking sequence with the center of the fruit bunch as the node using a genetic algorithm;

[0093] 2. Use the RRT algorithm under multi-elliptical area constraints to perform local path adjustment and generate the optimal obstacle avoidance path.

[0094] The final path contains 14 targets, the path length is about 5.2 meters, the generation time is less than 2 seconds, and there is no obstacle overlap in the path.

[0095] Phase 3: Optimal View Estimation and Active Optimization

[0096] As the robotic arm approached the target C4 tomato bunch, the system used an optical center optimization strategy to plan a top-down view approximately 0.6 meters from the target, aligned with the normal of the fruit stem. The initial image showed the fruit stem was obscured, resulting in a detection confidence of 0.46 and insufficient edge sharpness.

[0097] The system automatically switches to the 30° direction above as an alternative viewing angle. After shooting, the confidence level increases to 0.79, and the detection frame shows the entire fruit stalk area. The system confirms this viewing angle as the final perception viewing angle.

[0098] Stage 4: Fine positioning and key point extraction

[0099] The fine-tuned YOLOv8x-seg model is used to perform fine-grained object boundary and key point detection on unobstructed images, and to identify the top stalk and cutting point of the tomato.

[0100] The depth map and camera intrinsic parameters are combined for back projection to obtain the three-dimensional coordinates of the key points, which are then transmitted to the visual servo system for posture guidance and precise picking.

[0101] The center point positioning error of target C4 is 0.92mm, which meets the picking requirements. The system ensures stable recognition of key points throughout the entire process and guides the robot arm to accurately approach.

[0102] Experimental results

[0103] During this test, 14 bunches of tomatoes were picked, all of which were effectively detected and path generated. The system achieved a 100% recognition success rate, with 10 of these being recognized at the initial viewing angle and 4 being recognized through active viewing angle adjustment to compensate for occlusion.

[0104] The system's average single-target perception time is 0.81 seconds, reducing overall picking time by approximately 28% compared to traditional fixed-viewing strategies. This method demonstrates excellent robustness and real-time performance on clustered tomatoes, a typical target with dense occlusion.

[0105] On the other hand, corresponding to the above method example, Figure 6 As shown, the present invention also provides a multi-perspective target active perception system, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the system implements the steps of the multi-perspective target active perception method as described in any one of the above examples.

[0106] On the other hand, corresponding to the above method examples, the present invention also provides a computer-readable storage medium, which stores a computer program, wherein when the computer program is executed, the steps of the multi-perspective target active perception method as described in any of the above examples are implemented.

[0107] In summary, the multi-perspective target active perception method, system and storage medium provided by the present invention cleverly adjust the robot arm posture through the linkage of target key point occlusion judgment and perspective active optimization strategy to obtain the optimal depth map of the target. Compared with the traditional single-time, single-perspective target detection and positioning scheme, this has stronger robustness and high precision, and can still stably identify and locate targets in natural occlusion and high-density scenes. At the same time, this scheme also cleverly establishes a unified task coordination mechanism between the detection of target key points and the picking path planning, so it can cope with the rapid changes in target posture and occlusion conditions in complex environments. The key point deviation caused by this further amplifies the defect of path error, thereby effectively solving the positioning interference problem caused by target density.

[0108] In addition, it is worth mentioning that through the design of the present invention, the concept is no longer focused on the idea of ​​three-dimensional reconstruction and re-perception such as Nerf, which has large computational complexity, complex structure, and cannot be executed in real time. Instead, when precisely identifying a specific target, it estimates the optimal viewing angle with the least occlusion based on prior knowledge, and then determines whether to adjust the posture according to the degree of occlusion under the viewing angle. Therefore, it has the advantage of being more lightweight in computational complexity, which facilitates real-time detection and processing of the system.

[0109] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made based on the contents of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can better understand and utilize the present invention. The present invention is only limited by the claims and their full scope and equivalents. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0110] Those skilled in the art will understand that, in addition to implementing the systems, devices, units, and their respective modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same programs in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, etc. by logically programming the method steps. Therefore, the systems, devices, and their respective modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0111] In addition, all or part of the steps in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a program. The program is stored in a storage medium and includes a number of instructions for causing a single-chip microcomputer, chip, or processor to execute all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., various media that can store program code.

[0112] In addition, various implementations of the embodiments of the present invention may be arbitrarily combined, and as long as they do not violate the concept of the embodiments of the present invention, they should also be regarded as the contents disclosed in the embodiments of the present invention.

Claims

1. A multi-view target active perception method, comprising the following steps: Step S1 collects an RGBD image of the target scene and determines the initial position of the target through the target detection model; Step S2 constructs a picking task sequence based on the preliminary positions of each target and plans the motion path of the robotic arm; In step S3, according to the planned motion path of the robotic arm, the robot carries the viewfinder to the target position, collects the RGBD image of the target, and performs target key point occlusion judgment and active viewing angle optimization strategy until the best target RGBD image is obtained.

2. The multi-view active target perception method according to claim 1, wherein the steps of constructing a picking task sequence and planning a robot arm motion path include: Step S21 generates a preliminary picking sequence as the initial path using a hierarchical task sequence optimization algorithm based on the obtained preliminary target position; Step S22 defines the node connection lines in the initial path as flight segments, and creates ellipses with the manipulator's operable width of each flight segment as the minor axis length of the ellipse; deletes the overlapping areas between all ellipses to create a multi-ellipse space; Step S23: Based on the multi-elliptical spatial area restriction, the RRT algorithm is used to adjust the initial path to obtain the planned robot arm motion path.

3. The multi-view active target perception method according to claim 1, wherein the steps of target key point occlusion judgment and viewpoint active optimization strategy include: Step S31 uses the target detection model to identify the target key points in the RGBD image and detect their edge sharpness and confidence; Step S32 adjusts the robot arm posture, switches the viewing angle, collects RGBD images, and executes step S31 to obtain the edge sharpness and confidence of the target key points under different viewing angles; Step S33 compares the edge sharpness and confidence of each perspective and selects the best target RGBD image.

4. The multi-view active target perception method according to claim 1, wherein the target key points include: The pedicel or flower top of a small shrub, or the fruit stalk or fruit top of a small fruit.

5. A multi-view target active perception method, comprising the following steps: Step S1 collects an RGBD image of the target scene and determines the initial position of the target through the target detection model; Step S2 constructs a picking task sequence based on the preliminary positions of each target and plans the motion path of the robotic arm; Step S3: According to the planned motion path of the robotic arm, the robot moves to the target location with the viewfinder, collects the RGBD image of the target, and performs target key point occlusion judgment and active view angle optimization strategy until the best target RGBD image is obtained; Step S4 detects the best target RGBD image through the image analysis model, obtains the two-dimensional coordinates of at least any one of the target and its key points, and obtains the corresponding spatial coordinates according to the back-projection method.

6. The multi-view active target perception method according to claim 5, wherein the steps of constructing a picking task sequence and planning a robot arm motion path include: Step S21 generates a preliminary picking sequence as the initial path using a hierarchical task sequence optimization algorithm based on the obtained preliminary target position; Step S22 defines the node connection lines in the initial path as flight segments, and creates ellipses with the manipulator's operable width of each flight segment as the minor axis length of the ellipse; deletes the overlapping areas between all ellipses to create a multi-ellipse space; Step S23: Based on the multi-elliptical spatial area restriction, the RRT algorithm is used to adjust the initial path to obtain the planned robot arm motion path.

7. The multi-view active target perception method according to claim 5, wherein the steps of target key point occlusion judgment and viewpoint active optimization strategy include: Step S31 uses the target detection model to identify the target key points in the RGBD image and detect their edge sharpness and confidence; Step S32 adjusts the robot arm posture, switches the viewing angle, collects RGBD images, and executes step S31 to obtain the edge sharpness and confidence of the target key points under different viewing angles; Step S33 compares the edge sharpness and confidence of each perspective and selects the best target RGBD image.

8. The multi-view active target perception method according to claim 5, wherein the target key points include: The pedicel or flower top of a small shrub, or the fruit stalk or fruit top of a small fruit.

9. A multi-view target active perception system comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the system implements the steps of the multi-perspective target active perception method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, wherein when the computer program is executed, the steps of the multi-view target active perception method according to any one of claims 1 to 8 are implemented.

Images