Multi-view target active perception method and system based on multi-modal information fusion

Abstract

The application discloses a multi-view target active perception method and system based on multi-modal information fusion, and the method comprises the following steps: collecting an RGBD image of a target scene, determining a target preliminary position through a target detection model; constructing a picking task sequence according to each target preliminary position, and planning a mechanical arm motion path; going to the target position according to the planned mechanical arm motion path, collecting the RGBD image of the target, and executing target key point occlusion judgment and view active optimization strategy until the best target RGBD image meeting a preset threshold is obtained; when the target cannot obtain the best target RGBD image meeting the preset threshold after the view active optimization strategy, the target is marked as a target to be revisited, and the execution order of the target in the picking task sequence is adjusted. Through the linkage of occlusion judgment and view active optimization, and the feedback of the perception result to the task sequence, the problem that the target occlusion leads to the inability to accurately extract the better depth information of the target is solved.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and robot control technology, and in particular to a method and system for active perception of multi-view targets based on multimodal information fusion. Background Technology

[0002] In modern agricultural production, with the increasing demands for quality and yield, and the gradual rise in labor costs, how to achieve automated harvesting of agricultural targets such as fruits and flowers has become an important direction for the development of agricultural automation.

[0003] Currently, in the field of agricultural automation, when dealing with stacked targets such as goji berry picking and small shrub flower picking, in irregular and dynamic environments, due to the mutual stacking and occlusion between targets, the traditional single-view, single-detection target detection method of agricultural robots cannot accurately extract the best target depth information, which in turn affects the subsequent 3D position detection of key points of the target and interferes with the accuracy of picking point generation.

[0004] Existing fruit picking and localization schemes based on keypoint detection typically rely on deep learning models to identify key points of the fruit in an image (such as the stem and apex) and combine this with a depth camera for 3D localization. For example, by processing RGB images, the 2D coordinates of the key points are directly regressed, and then the 3D spatial pose is estimated using depth map information. However, this type of scheme heavily relies on a single-view image for keypoint identification and is easily affected by factors such as leaf occlusion, uneven lighting, and background interference. The detection accuracy drops significantly, especially when the stem is partially obscured, and it lacks a supplementary sensing mechanism for uncertain areas.

[0005] Existing technologies also propose observation pose estimation schemes based on dual-modal fusion semantic segmentation and point cloud geometric calculation (such as patent document CN119563455B). These schemes identify fruit clusters and stem masks, extract point clouds of individual fruit clusters through clustering, and fit straight lines along the stems. The observation pose of the end-camera is estimated in one go based on the relative position of the center of the fruit cluster point cloud and the perpendicular point of the fitted straight line. Such schemes rely on the integrity of the fruit cluster point cloud and the reliability of the stem straight line fitting. However, when the target is heavily obscured by leaves or upper fruits, the incomplete point cloud results in unverifiable pose estimation results, and there is still a risk of poor observation angle and failure to identify key points.

[0006] Existing technologies also propose viewpoint planning schemes based on deep reinforcement learning (such as patent document CN119489434A), which iteratively adjust the camera viewpoint based on the occlusion region detection results by training an action policy network. However, such schemes require pre-collection of scene data and training of the policy network; their generalization ability depends on the training scene distribution; and the viewpoint adjustment process is separate from the picking task sequence, meaning the perception results cannot be fed back to the task planning layer.

[0007] Furthermore, existing technologies include schemes that pre-plan the harvesting order based on shading conditions (such as patent document CN118700142A). These schemes pre-sort the harvesting queue by judging the spatial interference relationship between the target fruit and the branch and calculating the multi-fruit collision coefficient. However, the harvesting order planning of such schemes is based on static geometric relationships before harvesting. They lack a dynamic scheduling mechanism for newly perceived but unharvested targets during the harvesting process and typically directly remove unharvested targets, resulting in missed harvests.

[0008] Therefore, how to solve the problem of being unable to accurately extract the best depth information of a target due to target occlusion, while improving the robustness of target perception, the efficiency of viewpoint optimization, and the synergy between perception and task planning in occluded scenarios, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0009] In view of this, the main objective of the present invention is to provide a multi-view active target perception method and system based on multimodal information fusion, so as to solve the problem that target occlusion leads to the inability to accurately extract the best depth information of the target.

[0010] To achieve the above objectives, according to one aspect of the present invention, a multi-view target active perception method is provided, the steps of which include: Acquire RGBD images of the target scene and determine the initial location of the target using a target detection model; Based on the initial locations of each target, a harvesting task sequence is constructed, and the movement path of the robotic arm is planned; According to the planned robotic arm movement path, the camera is carried to the target position to collect the RGBD image of the target, and the target key point occlusion judgment and view active optimization strategy are executed until the best target RGBD image that meets the preset threshold is obtained. When a target fails to meet the preset threshold for keypoint confidence and / or edge sharpness under each viewpoint after being optimized by the aforementioned viewpoint proactive strategy, the target is marked as a target to be revisited, and the execution order of the target to be revisited in the picking task sequence is adjusted.

[0011] In a possible preferred embodiment, the step of constructing the picking task sequence and planning the robotic arm motion path includes: Based on the obtained preliminary target location, a hierarchical task sequence optimization algorithm is used to generate a preliminary picking sequence as the initial path; Define the node connection lines in the initial path as flight segments, and create ellipses with the robotic arm running width of each flight segment as the minor axis length of the ellipse. Delete the overlapping areas between all ellipses to create a multi-ellipse space. Based on the constraints of a multi-elliptical space region, the RRT algorithm is used to adjust the initial path to obtain the planned robotic arm motion path. By combining hierarchical task sequence optimization and RRT path adjustment under multi-elliptical space constraints, the joint optimization of the picking sequence and motion path is achieved, improving path planning efficiency and obstacle avoidance capabilities.

[0012] In a possible preferred embodiment, the steps of the target key point occlusion determination and active viewpoint optimization strategy include: Using an image analysis model, target key points in RGBD images are identified, and their edge sharpness and confidence are detected. Adjust the robotic arm posture, switch the viewing angle, acquire RGBD images, and re-identify the target key points to obtain the edge sharpness and confidence of the target key points under different viewing angles; By comparing the edge sharpness and confidence levels from different viewpoints, the best RGBD image of the target is selected. Selecting the optimal viewpoint image through multi-view comparison improves the accuracy and robustness of target keypoint detection.

[0013] In a possible preferred embodiment, the step of adjusting the robotic arm posture and switching the framing perspective includes: When the edge sharpness or confidence of the target key point does not meet the preset threshold, the gradient magnitude and direction histogram are calculated for the image within the target key point detection box. The main direction of the gradient direction histogram is used to determine the main direction of the occlusion boundary. The vertical direction of the main direction is used as the extension direction of the occlusion boundary. The occlusion source direction is confirmed based on the spatial distribution of low confidence in the area where the key point is located within the detection box. Based on the location of the occlusion source, alternative viewing angles are planned preferentially within the opposite direction of the occlusion source. RGBD images of the alternative viewing angles are acquired, and the occlusion judgment of target key points is re-executed. By guiding the planning of alternative viewing angles through occlusion source location perception, the number of viewing angle attempts is reduced, and the time consumption for single-target perception is shortened.

[0014] In a possible preferred embodiment, the step of adjusting the execution order of the target to be revisited in the picking task sequence includes: The target to be revisited is moved to the end of the picking task sequence, and the picking tasks of the remaining targets in the picking task sequence are executed first. After the remaining targets have been captured, the target to be revisited is subjected to a new strategy of judging target key point occlusion and actively optimizing the viewpoint until the best target RGBD image that meets the preset threshold is obtained. By rearranging the task sequence, targets that failed to be perceived are given a second chance to be perceived, avoiding missed captures and improving the capture completeness rate.

[0015] In a possible preferred embodiment, after obtaining the optimal target RGBD image, the method further includes: Simultaneously, fine-grained recognition and key point detection are performed on other targets within the field of view of the optimal target RGBD image, obtaining the two-dimensional coordinates of at least one of the other targets and its key points. By sharing the sensing pose, multiple targets can be sensed in one go, reducing the number of back-and-forth movements of the robotic arm and improving sensing efficiency.

[0016] In a possible preferred embodiment, the method further includes the following steps: By using an image analysis model, the optimal RGBD image of the target is detected, and the two-dimensional coordinates of the target and / or key points are obtained. The corresponding spatial coordinates are then obtained using a back-projection method. This back-projection method provides spatial positioning information for automated harvesting.

[0017] To achieve the above objectives, according to another aspect of the present invention, a multi-view target active perception system is also provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the system implements the steps of any of the above-described multi-view target active perception methods.

[0018] To achieve the above objectives, according to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium storing a computer program, wherein when the computer program is executed, it implements the steps of the multi-view target active perception method as described above.

[0019] The multi-view active target perception method, system, and storage medium provided by this invention, through the linkage of target key point occlusion judgment and active view optimization strategy, actively adjusts the posture of the robotic arm to obtain the optimal depth map of the target. Compared with traditional single-shot, single-view target detection and localization schemes, this approach has stronger robustness and higher accuracy, and can still stably identify and locate targets in natural occlusion and high-density scenes. In addition, this solution establishes a unified task collaboration mechanism for target key point detection and harvesting path planning. Targets that fail to be perceived are given a second perception opportunity through task sequence rearrangement. Therefore, it can cope with the defects of path error further amplified by the rapid changes in target posture and occlusion in complex environments, which lead to key point deviations. This effectively solves the localization interference problem caused by target density. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram illustrating the steps of the multi-view target active perception method of the present invention; Figure 2 This is a flowchart illustrating the task sequence optimization and robotic arm path planning in the multi-view target active perception method of the present invention. Figure 3 This is a schematic diagram of the structure of the multi-view target active perception system of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions of the present invention, the specific technical solutions of the present invention will be clearly and completely described below in conjunction with embodiments, so as to help those skilled in the art further understand the present invention. Obviously, the embodiments described in this application are merely some embodiments of the present invention, and not all embodiments. It should be noted that, for those skilled in the art, the embodiments and features in the embodiments of this application can be combined with each other without departing from the concept of the present invention and without conflict. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the disclosure and protection scope of the present invention.

[0022] Furthermore, the terms "first," "second," "S1," "S2," etc., used in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such features can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those described herein. At the same time, the stages described in each step are not necessarily to be implemented in the same step; it should be understood that the implementation order of the contents of each step stage can be adjusted and interchanged without violating the inventive concept, so that embodiments of the invention described herein can be implemented in orders other than those described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. Unless otherwise expressly specified and limited, the terms "set," "arrange," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; a mechanical connection or an electrical connection; a direct connection or an indirect connection through an intermediate medium; or a connection within two elements. Those skilled in the art can understand the specific meaning of the above terms in this case based on the specific circumstances and in conjunction with existing technology.

[0023] In traditional agricultural automation, when dealing with small, stacked targets such as goji berries and small shrubs, the single-view, single-shot target detection methods of traditional agricultural robots fail to extract optimal target depth information in irregular and dynamic environments due to the overlapping of targets. This affects the detection of 3D key points (such as the flower stalks and tips of small shrubs, or the fruit stalks and tips of small fruits) and 6D pose estimation, and interferes with the accuracy of harvest point generation. Therefore, to address the problem of inaccurate target depth information extraction caused by target occlusion, such as... Figure 1 As shown, this invention proposes a multi-view target active perception method, the example steps of which include: Step S1: Acquire RGBD images of the target scene and determine the initial location of the target using a target detection model.

[0024] This step primarily addresses coarse environmental perception, enabling rapid real-time target detection based on multiple targets within the field of view, and allowing the system to pinpoint the initially identified target locations. In this embodiment, an external depth camera is used for coarse environmental perception to acquire panoramic RGBD images of the environment. Then, a deep learning target detection model is combined to perform initial target detection. Finally, pixel-level semantic segmentation is used to construct an environmental semantic map, obtaining global semantic information and the initial target locations.

[0025] For example, based on a target detection model fine-tuned in an agricultural scenario, contours and bounding boxes are labeled using RGB information. By performing pixel-level classification and segmentation of the image, a detailed environmental semantic map is constructed, quickly locating the approximate position of the target within the image and guiding subsequent task map construction. For instance, the YOLOv8m model is fine-tuned on images of the overall growth environment of crops taken with an external RGBD camera. During fine-tuning, image data is appropriately enhanced, such as by modifying brightness and contrast, to simulate changes in lighting conditions and enhance the system's versatility. Furthermore, the target's position within the image is appropriately cropped and reassembled to expand the adaptability of the fine-tuning training.

[0026] This part of the network outputs the center coordinates of the target's detection box. After hand-eye calibration (eye-to-hand), it selects the k-nearest neighbor coordinates within a set range around the center point of the detection box, calculates the 3D point cloud projected onto the robot base as the origin, and then performs outlier removal on the mean 3D coordinates. Filtering and data correction techniques are used for smoothing and noise reduction to ensure the stability and accuracy of the coarse localization information. The system utilizes a rigid coordinate transformation method to map the target's center point and depth information from the camera coordinate system to the robot arm's workspace, outputting the target's position information in 3D space. This information is then uploaded to the ROS system for subscription, providing data support for subsequent robot arm motion planning and precise operation.

[0027] In existing technologies, downstream tasks such as target key point detection and harvesting path planning are typically executed separately without a unified task coordination mechanism. This makes it difficult to cope with rapid changes in target posture and occlusion in complex environments, leading to key point deviations further amplifying path errors and reducing overall system efficiency. Location interference is particularly prominent in areas with densely packed multiple fruit targets. Therefore, in the following example, this invention establishes a unified task coordination mechanism between target key point detection and harvesting path planning to address location interference caused by target density.

[0028] Step S2: Based on the initial positions of each target, construct the picking task sequence and plan the movement path of the robotic arm.

[0029] This step primarily involves constructing a harvesting task graph based on a coarse-grained environmental perception semantic map, applying a genetic algorithm to generate the harvesting order, and combining this with the RRT algorithm to optimize the path for obstacle avoidance and efficient planning. In this implementation, as... Figure 2 As shown, the example constructs a harvesting task graph using the target semantic environment graph and the target's initial position (such as 3D center point position information) given in step S1. This harvesting task graph uses the target's initial position as nodes and the operational relationships between nodes as edges, and is used to generate a harvesting task sequence. Addressing the traversal problem of multiple harvesting points in the harvesting task, this example abstracts it as a Task Sequence Optimization (TSP) problem, considering multiple factors such as operational time windows, path continuity, and robotic arm motion constraints. An optimization model incorporating target position, operational cost, and motion constraints is established to achieve hierarchical optimization from coarse sorting to refined path optimization. In the coarse sorting stage, a genetic algorithm is used to quickly generate the initial harvesting sequence; in the refined sorting stage, algorithms such as the RRT algorithm are combined to refine and adjust each path segment, performing local motion planning to ensure that the path reaches its optimal state in terms of obstacle avoidance and harvesting efficiency.

[0030] For example, coarse localization information can be obtained by subscribing to the ROS system, and an optimal harvesting path can be generated using a hierarchical task sequence optimization algorithm to guide the robot to efficiently approach the target area. In this example, it is assumed that in a greenhouse environment, the robot mainly travels in a straight line, with fruit plants arranged in two rows on either side of the agricultural robot. This reduces the difficulty of sorting paths from multiple 3D coordinates.

[0031] Hierarchical task sequence optimization algorithms are heuristic optimization algorithms that perform hierarchical optimization of task sequences, including genetic algorithms, ant colony algorithms, simulated annealing algorithms, etc. In this embodiment, the genetic algorithm is used to quickly generate the preliminary picking sequence in the coarse sorting stage. The process is exemplified by the following steps: First, only the coordinates within the robotic arm's workspace are retained, and an index i is generated for each coordinate. Based on the optimal pose estimation, the coordinates of the observation points for fine-tuning are generated. The population is initialized, with the coordinate furthest from the base fixed as the endpoint of the path, and the remaining indexes are used to randomly generate paths. That is: path = [p1, p2, ..., pn-1, pn], where pn is the furthest point (fixed), but GA optimizes the order of [p1, ..., pn-1].

[0032] For the fitness function: fitness(path) = total path length (start→obs1→tgt1→obs2→tgt2→...→obsn→tgtn) (Euclidean distance, including the total cost of the farthest point) + penalty term (increases with each switch between left and right rows).

[0033] By iterating through sequential crossover (OX) and mutation, a coarsely sorted sequence is obtained, which generates a preliminary picking sequence as the initial path.

[0034] The following is an example of a process for refining each path segment using algorithms such as RRT: After obtaining the coarsely sorted sequence as the initial path, a multi-elliptical region is created as the designated space. The connecting lines of the nodes in the initial path are defined as segments, and ellipses are created with the minimum width of each segment (i.e., the operating width of the robotic arm) as the minor axis length. By deleting the overlapping areas between all ellipses, a multi-elliptical space is created as the designated space for the RRT algorithm to search for the final path.

[0035] Initialize the RRT tree, i.e., create an empty path tree and set the starting point as the root node. Set the algorithm parameters, such as the maximum number of iterations, connection distance r, obstacle radius, etc. Set the starting point and target point, using tgt1 and tgtn in the initial path as the starting point and target point of the task, respectively.

[0036] Within the specified space, randomly select a point as the sampling point qrand. Find the node qnear that is closest to qrand in the path tree. Between qnear and qrand, select a point with a distance r from qnear (i.e., the pre-defined connection distance parameter r) as the new node qnew. Check if the line between qnew and qnear overlaps with an obstacle. If it overlaps, discard qrand and reselect a sampling point; otherwise, add qnew to the path tree and set its parent node to qnear.

[0037] Among nodes within a radius of qnew, find the node with the lowest cost and designate it as the optimal parent node. Compare the cost of each node within that radius with the cost of connecting it as a child node of qnew. If connecting a node as a child node has a lower cost, break the existing link with the parent node and set it as a child node of qnew.

[0038] Repeat the above steps until the target point is reached. Generate a graph connecting the nodes generated by the RRT algorithm, and remove parts that overlap with obstacles to obtain a collision-free spatial graph. Search the collision-free spatial graph and use Dijkstra's algorithm to find the shortest path connecting the start and end points, which is taken as the final path.

[0039] Step S3: According to the planned robotic arm movement path, carry the camera to the target position, collect the target's RGBD image, and execute the target key point occlusion judgment and view active optimization strategy until the best target RGBD image that meets the preset threshold is obtained.

[0040] Step S3 is used for optimal viewpoint estimation and active viewpoint optimization. For example, according to the picking sequence planned by the picking task sequence, the best shooting pose is obtained based on the optical center optimization model for the target to be picked, and the system robotic arm is guided to move to the designated pose. The handheld RGBD camera is used to collect fine images. When occlusion is detected, the active perception module is called to switch to the alternative pose.

[0041] Step S3 primarily focuses on the picking sequence and planning given in step S2, guiding the robotic arm to approach the target to be picked sequentially. For the current target, the optimal shooting pose is obtained based on the optical center optimization model. Outside the target coordinates, the optimal shooting coordinates of the camera are calculated using the back projection formula. Combining the target pose and the reachable range of the robotic arm, the optimal pose of the handheld camera is determined. The robotic arm is then guided to approach this pose for refined shooting to obtain the optimal RGBD information of the viewing angle, completing fine recognition. When occlusion is detected, the active perception module is invoked to switch to an alternative pose.

[0042] It should be noted that, since this example preferably uses a fixed-focus 3D camera lens, its characteristic of a fixed imaging plane ensures optimal depth measurement accuracy at the camera's optical center position under unobstructed conditions. Therefore, the optimal shooting coordinates of the camera can be calculated using the back-projection formula outside the target coordinates. By combining the target pose and the reachability of the robotic arm, the optimal pose of the handheld camera can be determined, thereby optimizing the perception accuracy.

[0043] For occlusion-based recognition, if the detection accuracy and confidence of the aforementioned target and key point recognition decrease due to occlusion, an active perspective adjustment strategy can be adopted. By obtaining an unobstructed perspective through alternative perception poses, and based on the real-time feedback of the target detection system, the system automatically switches between different perception poses to obtain the highest target detection confidence, thereby achieving accurate capture of target key points.

[0044] For example, the aforementioned optical center optimization model implements a target key point occlusion judgment and active viewpoint optimization strategy. The purpose of this strategy is to select a pose where the target is not occluded and can be clearly and completely imaged. For instance, according to the coarse positioning information, the robotic arm moves the handheld camera to a position about 0.6m away from the target on the line connecting the handheld camera and the target, and makes the target appear in the center of the handheld camera image; for the target that was occluded in the previous shot, it moves upward 30° (keeping above 0.6m, center imaging) and takes another shot, until a complete image is captured or multiple alternative poses (30° angular movements in multiple directions) have been traversed.

[0045] To determine whether occlusion exists in the image, this example uses the following criteria: Occlusion usually causes unclear edges, so the Laplacian operator or LoG operator in OpenCV is used to perform edge sharpness detection on the target within the target keypoint detection box; several viewpoints with the highest target detection confidence and edge sharpness, such as those in YOLOv8, are selected. If the confidence of a single viewpoint is lower than expected (e.g., 0.4), and the calculated edge sharpness is lower than other viewpoints, then the target is judged to be occluded. Alternative viewpoints are obtained by actively adjusting the camera pose. This process is iterated, and combined with new local image information, viewpoint optimization and subsequent fine-tuning localization are performed again.

[0046] Furthermore, this embodiment provides a viewpoint switching method guided by occlusion direction perception. When the edge sharpness or confidence of a target key point does not meet a preset threshold and the target is determined to be occluded, the occlusion source direction is estimated based on the principal direction of the edge gradient and the spatial distribution of confidence within the target key point detection box. According to the occlusion source direction, alternative viewpoints are planned in the opposite direction region of the occlusion source, RGBD images under the alternative viewpoints are acquired, and the occlusion judgment is re-executed. Compared with the viewpoint switching method that iterates step by step at a fixed angle, occlusion direction perception guidance reduces the number of viewpoint attempts required for a single target to obtain an unoccluded viewpoint, shortening the perception time for a single target.

[0047] For example, when the fruit stalk is obscured by a leaf on the left, the principal direction of the gradient within the detection box points towards the leaf edge, and the confidence level in the keypoint area is low. Based on this, the source of the occlusion is estimated to be on the left. Therefore, alternative viewpoints shifted to the right (e.g., shifting 15° to 45° to the right) are prioritized for planning, rather than trying multiple fixed directions sequentially. If the keypoint is still obscured after shifting to the right, the direction of the occlusion source is re-estimated based on the new confidence spatial distribution. The next alternative viewpoint is then planned in the opposite direction until the best image that meets the threshold requirement is obtained or all alternative viewpoints are explored.

[0048] The estimation of the occlusion source location can be implemented using the following example: Calculate the Sobel gradient of the grayscale image within the target keypoint detection box to obtain the gradient magnitude and direction angle of each pixel. Construct a gradient direction histogram from 0 degrees to 180 degrees. Take the direction angle corresponding to the peak value of the histogram as the principal direction of the occlusion boundary. The vertical direction of this principal direction is the extension direction of the occlusion boundary. Divide the detection box into several sub-regions and calculate the mean of the target keypoint detection confidence in each sub-region. Take the side of the sub-region where the mean confidence is lower than the threshold of the overall box mean as the occlusion source location. Based on this, plan alternative viewing angles with offsets of 15° to 45° on a sphere approximately 0.6m away from the target in the opposite direction of the occlusion source, and center the target in the image.

[0049] This implementation also provides a task sequence reordering mechanism for feedback of perception results. When a target still cannot obtain the best RGBD image that meets the threshold requirements after traversing the alternative viewpoints, the target is marked as a target to be revisited, and its execution order in the picking task sequence is adjusted to the end of the sequence. The picking tasks of the other targets in the sequence are executed first. After the other targets are picked, the environmental occlusion structure changes (such as the removal of adjacent fruits or the displacement of occluding leaves). The target key point occlusion judgment and viewpoint active optimization strategy are re-executed for the target to be revisited until the best target RGBD image that meets the preset threshold is obtained. This mechanism feeds back the perception results of single target viewpoint optimization failure to the task sequence optimization layer, forming a closed loop between perception and planning: target perception failure is no longer an irrecoverable loss, but a second perception opportunity is obtained through task sequence reordering. At the same time, because adjacent targets are picked first, the sources of occlusion are reduced during revisiting, and the success rate of active viewpoint optimization is improved.

[0050] Furthermore, this embodiment provides a method for implementing multi-target shared perception pose. After acquiring the optimal RGBD image of the current target, fine recognition and key point detection are simultaneously performed on other candidate targets within the field of view of the optimal RGBD image. This reuses a single-position perception of multiple targets, reducing the number of times the robotic arm needs to move back and forth to perceive different targets. For example, under the current optimal field of view, if there are other candidate targets within the field of view besides the target to be picked, target and key point detection are simultaneously performed on them. If the key point detection of other candidate targets meets the threshold requirements, their spatial coordinates are directly generated, and subsequent execution according to the picking sequence does not require re-moving to the position for perception.

[0051] Furthermore, in order to achieve spatial localization of the target and its key points after active perception from multiple perspectives, and to provide spatial positioning information for automated harvesting, in an optional implementation, the method steps of this example further include: Step S4: Detect the best target RGBD image through the image analysis model, obtain the two-dimensional coordinates of the target and / or key points, and obtain the corresponding spatial coordinates according to the back projection method.

[0052] This step primarily utilizes the obtained optimal RGBD information to perform fine-grained recognition and localization. Conceptually, this image analysis model combines target detection and keypoint detection modules to analyze target keypoints, and uses depth maps to accurately locate the target and keypoints. Here, the image analysis model for fine-grained recognition preferably uses the YOLOv8-seg model based on YOLOv8x, and is fine-tuned based on a close-up RGB dataset, annotating relevant keypoint data such as the cutting points of the target stem (including the pedicel and flower apex of small shrubs, or the pedicel and fruit apex of small fruits).

[0053] Subsequently, the center pixel coordinates and keypoint coordinates of the target are calculated from the acquired RGB detection results. For the depth map, median filtering or local mean calculation is used to suppress noise and obtain robust depth estimates. Combining the depth information, camera intrinsic parameters (including focal length, principal point, and distortion parameters) are used to map the two-dimensional pixel coordinates to three-dimensional spatial coordinates based on the back projection principle to guide subsequent 3D pose estimation and visual servoing.

[0054] For example, after obtaining the optimal shooting angle, the system uses a handheld RGBD camera for further segmentation and localization to acquire complete target features and key point information. The back-projection formula used by the system is the eye-on-hand back-projection formula: mapping two-dimensional pixel coordinates to three-dimensional spatial coordinates to guide subsequent 3D pose estimation and visual servoing.

[0055] In this step, YOLOv8-seg, based on the YOLOv8x model, is preferred for deployment and training to achieve contour detection and keypoint detection of near-field target images. During fine-tuning, the LabelMe tool can be used to finely annotate the dataset, outlining the target contours and keypoint information, and data augmentation is performed using the same preprocessing methods as in step S1. The output contour edge information or detection box information is projected onto the 3D point cloud as the edge of the view frustum information, and outlier removal, point cloud clustering, and other processing are performed to extract the refined 3D point cloud of the visible target and the keypoints on the point cloud.

[0056] To verify the application effect of the present invention in actual agricultural scenarios, the following describes the multi-view target active perception method of the present invention in detail, taking the greenhouse bunch tomato picking task as an example.

[0057] In a certain agricultural base, multi-row tomato crops are grown in a greenhouse. The fruits grow in clusters along the sides of the main stem, forming a hanging pattern. The fruits severely obscure each other, with the fruit stalks hidden behind leaves or upper fruits. This clustered structure poses a challenge to recognizing continuity and separating the target.

[0058] The robot platform is configured as follows: the mobile platform is a tracked chassis with autonomous navigation; an RGBD camera (handheld camera) is installed at the end of the dual-degree-of-freedom picking robotic arm; an external RGBD camera is installed on the top of the vehicle body for wide-range panoramic perception.

[0059] Step S1, Coarse Environmental Perception: After the robot starts, the external RGBD camera acquires panoramic RGBD images of the left and right fruit plants. A finely tuned YOLOv8m model is used to identify clusters of tomatoes, extracting the overall outline and approximate center coordinates of each cluster. An environmental semantic map is constructed using semantic segmentation, and preliminary 3D coordinates are obtained through backprojection. This batch detected 14 targets with an average confidence level of 0.74. Combined with the external camera calibration, the coordinates are mapped to the robot's base coordinate system.

[0060] Step S2: Harvesting Task Graph Construction: The system combines target locations and workspace constraints to form a harvesting task graph. A genetic algorithm is used to generate an initial harvesting sequence with the fruit bunch center as the node; the RRT algorithm under multi-elliptical region constraints is used for local path adjustment to generate the optimal obstacle avoidance path. The final path contains 14 targets, has a path length of approximately 5.2 meters, a generation time of less than 2 seconds, and has no overlapping obstacles.

[0061] Step S3, Optimal Viewpoint Estimation and Active Optimization: When the robotic arm approaches the target bunch of tomatoes C4, the system uses an optical center optimization strategy to plan a top-down viewpoint approximately 0.6m from the target and aligned with the stem normal. The initial image shows the stem is occluded, with a detection confidence of 0.46 and insufficient edge sharpness. Based on the principal direction of the edge gradient within the detection box, the occlusion source is estimated. In this example, the estimated occlusion source is upward, so a 30° upward shift is used as a candidate viewing angle. After shooting, the confidence increases to 0.79, the detection box fully displays the stem area, and the system confirms this viewpoint as the final perceived viewpoint. Notably, after adopting the viewpoint switching guided by occlusion direction perception, the average number of viewpoint attempts per target decreases from 3-4 times under the fixed-direction traversal method to 1-2 times.

[0062] Step S4, Fine-tuned localization and key point extraction: A finely tuned YOLOv8x-seg model is used to perform fine-grained target boundary and key point detection on the unobstructed image, identifying the stem and cutting point at the top of the tomato bunch. Back-projection is performed using the depth map and camera intrinsic parameters to obtain the 3D coordinates of the key points, which are then transmitted to the vision servo system for posture guidance and precise harvesting. The center point localization error of target C4 is 0.92mm, meeting the harvesting requirements.

[0063] In this implementation, targets that cannot be clearly imaged after traversing the initial and alternative viewing angles are marked and moved to the end of the picking sequence. Of the 14 targets in this batch, one target was marked as a target to be revisited because its causal stem was doubly occluded by upper fruits and leaves, and an image meeting the threshold requirement could not be obtained after traversing alternative orientations. After the remaining 13 targets in the sequence were picked, the occlusion structure around this target changed, and during the revisit, a single viewpoint optimization yielded the best image with a confidence level of 0.81, ultimately completing the picking without any missed picking. Simultaneously, fine recognition was performed on other candidate tomato bunches within the current optimal viewpoint, with three bunches of tomatoes being perceived in one go within the shared viewpoint, reducing the robotic arm's round trips by three times.

[0064] Test results show that all 14 bunches of tomatoes in this batch were effectively detected and paths were generated, achieving a 100% success rate. Nine were identified from the initial viewpoint, four were identified through active viewpoint adjustment to compensate for occlusion, and one was identified through a revisit after task sequence rearrangement. The system's average single-target perception time was 0.72 seconds, and the overall harvesting time was reduced by approximately 31% compared to the traditional fixed-viewpoint strategy.

[0065] On the other hand, corresponding to the above method examples, such as Figure 3As shown, the present invention also provides a multi-view target active perception system, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the system implements the steps of any of the multi-view target active perception methods described in the above examples.

[0066] On the other hand, corresponding to the above method examples, the present invention also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed, it implements the steps of any of the multi-view target active perception methods described in the above examples.

[0067] In summary, the multi-view active target perception method, system, and storage medium provided by this invention, through the linkage of target key point occlusion judgment and active view optimization strategy, actively adjust the posture of the robotic arm to obtain the optimal depth map of the target. Compared with traditional single-shot, single-view target detection and localization schemes, this approach has stronger robustness and higher accuracy. It can still stably identify and locate targets in natural occlusion and high-density scenes. At the same time, this solution also establishes a unified task coordination mechanism for target key point detection and picking path planning. Therefore, it can cope with the defects of path error further amplified by the rapid changes in target posture and occlusion in complex environments, which can lead to key point deviations. This effectively solves the localization interference problem caused by dense targets.

[0068] Furthermore, through the design of this invention, the concept no longer focuses on the computationally intensive, structurally complex, and unreal-time-executable 3D reconstruction and re-sensing approach such as NeRF. Instead, when performing fine identification of specific targets, it estimates the optimal viewpoint with the least occlusion based on prior knowledge, and then determines whether to adjust the pose based on the degree of occlusion under the viewpoint. Therefore, it has a more lightweight advantage in terms of computational complexity, which facilitates real-time detection and processing of the system.

[0069] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The present invention is limited only by the claims and their full scope and equivalents. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

[0070] Those skilled in the art will understand that, besides implementing the system, apparatus, unit, and its modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and its modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0071] Furthermore, all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a microcontroller, chip, or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0072] Furthermore, various different implementations of the present invention can be combined arbitrarily, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed in the present invention.

Claims

1. A multi-view target active perception method, characterized in that, Includes the following steps: Acquire RGBD images of the target scene and determine the initial location of the target using a target detection model; Based on the initial locations of each target, a harvesting task sequence is constructed, and the movement path of the robotic arm is planned; According to the planned robotic arm movement path, the camera is carried to the target position to collect the RGBD image of the target, and the target key point occlusion judgment and view active optimization strategy are executed until the best target RGBD image that meets the preset threshold is obtained. When a target fails to meet the preset threshold for keypoint confidence and / or edge sharpness under each viewpoint after being optimized by the aforementioned viewpoint proactive strategy, the target is marked as a target to be revisited, and the execution order of the target to be revisited in the picking task sequence is adjusted.

2. The multi-view target active perception method according to claim 1, characterized in that, The steps of constructing the harvesting task sequence and planning the robotic arm motion path include: Based on the obtained preliminary target location, a hierarchical task sequence optimization algorithm is used to generate a preliminary picking sequence as the initial path; Define the node connection lines in the initial path as flight segments, and create ellipses with the robotic arm running width of each flight segment as the minor axis length of the ellipse. Delete the overlapping areas between all ellipses to create a multi-ellipse space. Based on the constraints of the multi-elliptical space region, the RRT algorithm is used to adjust the initial path and obtain the planned robotic arm motion path.

3. The multi-view target active perception method according to claim 1, characterized in that, The steps of the target key point occlusion judgment and active viewpoint optimization strategy include: Using an image analysis model, target key points in RGBD images are identified, and their edge sharpness and confidence are detected. Adjust the robotic arm posture, switch the viewing angle, acquire RGBD images, and re-identify the target key points to obtain the edge sharpness and confidence of the target key points under different viewing angles; By comparing the edge sharpness and confidence level from different perspectives, the best target RGBD image is selected.

4. The multi-view target active perception method according to claim 3, characterized in that, The steps of adjusting the robotic arm's posture and switching the framing perspective include: When the edge sharpness or confidence of the target key point does not meet the preset threshold, the gradient magnitude and direction histogram are calculated for the image within the target key point detection box. The main direction of the gradient direction histogram is used to determine the main direction of the occlusion boundary. The vertical direction of the main direction is used as the extension direction of the occlusion boundary. The occlusion source direction is confirmed based on the spatial distribution of low confidence in the area where the key point is located within the detection box. Based on the location of the occlusion source, alternative viewing angles are planned in the opposite direction of the occlusion source, RGBD images under the alternative viewing angles are acquired, and the target key point occlusion judgment is re-executed.

5. The multi-view target active perception method according to claim 1, characterized in that, The step of adjusting the execution order of the target to be revisited in the harvesting task sequence includes: The target to be revisited is moved to the end of the picking task sequence, and the picking tasks of the remaining targets in the picking task sequence are executed first. After the remaining targets have been captured, the target key point occlusion judgment and viewpoint active optimization strategy are re-executed for the target to be revisited until the best target RGBD image that meets the preset threshold is obtained.

6. The multi-view target active perception method according to claim 1, characterized in that, After obtaining the optimal target RGBD image, the following is also included: Simultaneously perform fine recognition and key point detection on other targets within the field of view of the optimal target RGBD image, and obtain the two-dimensional coordinates of at least one target and its key points.

7. The multi-view target active perception method according to claim 1, characterized in that, It also includes the following steps: By using an image analysis model, the optimal target RGBD image is detected, the two-dimensional coordinates of the target and / or key points are obtained, and the corresponding spatial coordinates are obtained by back projection method.

8. The multi-view target active perception method according to any one of claims 1 to 7, characterized in that, The target key points are the flower stalks and flower tips of small shrubs, or the fruit stalks and fruit tips of small fruits.

9. A multi-view target active perception system, characterized in that, The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the system implements the steps of the multi-view target active perception method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed, it implements the steps of the multi-view target active perception method as described in any one of claims 1 to 8.

Images