A vision algorithm guided robotic sorting method and system

Abstract

The application discloses a kind of visual algorithm guided robot sorting method and system, belong to industrial automation and machine vision technical field.The method is by multi-task visual detection synchronous output material's boundary frame, instance mask and root key point, according to root key point and instance mask generation with unidirectionality direction vector to determine the head and tail semantics of material, and generate grasping pose accordingly;The system contains the visual detection module and pose generation module of executing the above method.The application solves the 180 degree direction ambiguity problem in the rotating target detection by using the root key point as the direction anchor point, and overcomes the interference of multiple defect features by attribute decoupling learning, which is suitable for pipeline scenes with directional sorting and defect identification requirements.

Description

Technical Field

[0001] This invention relates to the fields of industrial automation and machine vision technology. Specifically, it relates to a robot sorting method and system guided by visual algorithms for target materials that are irregular, easily deformable, and have directional and multi-attribute detection requirements. It is applicable to assembly line scenarios with directional sorting and defect identification needs, such as flexible industrial parts assembly and post-harvest processing of agricultural products. Background Technology

[0002] Robotics technology is widely used in industrial sorting, and it is an effective way to replace human labor with machines. These robots are usually guided by computer vision to detect and classify the items to be sorted. After coordinate system transformation, the robotic arm performs the sorting action.

[0003] In current automated sorting systems, machine vision-based guidance technologies are widely used. For standard industrial parts with regular shapes, existing visual sorting technologies demonstrate high maturity. These systems typically rely on template matching or a certain level of expert knowledge for positioning and classification. For materials with fixed colors, there are also HSV-based segmentation methods, such as the blob method, to extract the material's morphological features. However, when the sorting objects become irregularly shaped, reflective, and unstructured materials, traditional visual methods are limited by surface reflection and material shape distribution, resulting in unstable visual detection results. To address this, industry and academia have gradually introduced deep learning technologies with strong generalization capabilities. For example, various object detection models based on the YOLO architecture are widely used in the positioning, feature recognition, and quality grading of agricultural products, providing a more flexible perception solution for sorting materials with complex shapes.

[0004] While deep learning has made significant progress in basic classification and localization tasks, it still faces considerable challenges when dealing with unstructured flexible materials. Taking agricultural product processing and sorting as an example, robots are required to grade and orient materials. Such tasks not only require the system to identify defects with multiple different attributes but also to strictly distinguish the head and tail directions of the materials. Faced with such complex tasks, existing orientation target detection technologies cannot simultaneously meet these requirements. On the one hand, existing orientation bounding box algorithms can only extract undirected geometric principal axes, resulting in inherent 180-degree directional ambiguity. They cannot identify the true head and tail semantics of the materials and are easily affected by flexible deformations such as random leaf unfolding, causing the robotic arm to fail to place materials due to the loss of unidirectional orientation information. On the other hand, traditional single-label mutually exclusive classification mechanisms are difficult to effectively decouple superimposed heterogeneous defects, and the competition between deep features reduces the detection stability of composite defect samples. End-to-end grasping pose estimation techniques have emerged, but general grasping models only optimize stable grasping as the sole objective, lacking mechanical constraints on the materials. They are prone to mistakenly grasping fragile and vulnerable parts such as leaf crowns while failing to lock onto rigid stems. Summary of the Invention

[0005] This invention aims to provide a vision algorithm-guided robot sorting method and system, mainly addressing the following technical problems: This invention addresses the 180° semantic orientation ambiguity problem in existing rotating target detection for the orientation and sorting of irregular objects, and achieves unambiguous orientation pose estimation. Overcome the feature interference problem of multi-defect attribute targets in the classification process and improve the recognition accuracy of composite defect samples; Establish a complete robotic sorting framework from visual perception to mechanical constraint grasping to achieve high-speed sorting of flexible and irregular materials.

[0006] To achieve the above objectives, the present invention provides a method and system for guiding a robot to sort items, as well as a robot sorting system and a robot sorting method including the system.

[0007] In a first aspect, the present invention provides a method for guiding a robot to sort items, comprising the following steps: acquiring an image containing the material to be sorted; performing multi-task visual detection on the image and simultaneously outputting the bounding box, instance mask, and root key points of the material to be sorted; generating a unidirectional direction vector of the material to be sorted based on the root key points and the instance mask; and generating a grasping pose based on the unidirectional direction vector and a preset grasping strategy.

[0008] This method synchronously outputs root keypoints and instance masks through multi-task visual detection, uses root keypoints as visual anchors for direction perception, and combines instance masks to generate unidirectional direction vectors, thereby overcoming the 180° direction ambiguity problem of traditional directed bounding boxes.

[0009] The multi-task visual detection specifically includes: performing multi-scale feature extraction and feature fusion on the image to obtain a fused feature map at multiple scales; performing bounding box regression and mask prediction based on the fused feature map to output the bounding box and the instance mask; and predicting the root keypoints based on the fused feature map. Through multi-scale feature extraction and fusion, both shallow spatial details and deep semantic information of the material can be captured simultaneously, adapting to the material detection needs of different scales and shapes.

[0010] In a preferred embodiment, the root key point is corrected according to the instance mask so that the corrected root reference point is located within the material area corresponding to the instance mask. A unidirectional direction vector is generated based on the root reference point and the center point of the material to be sorted. The gripping point is set in the direction from the root reference point to the center point, and the distance between the gripping point and the root reference point is a preset ratio of the distance between the root reference point and the center point. Based on the pixel ratio coefficient and the angle difference between the image coordinate system and the robot coordinate system, the position and orientation angle of the gripping point in the image coordinate system are converted into the gripping pose in the robot coordinate system.

[0011] In a preferred embodiment, the multi-task visual detection further includes an attribute-based feature enhancement method: an attribute prediction branch predicts whether the material to be sorted has a first attribute defect and whether it has a second attribute defect; a first defect enhancement branch aggregates local and contextual features of the fused feature map based on a color defect prior, generating a first defect attention map and a first attribute response; a second defect enhancement branch extracts high-frequency structural information from the fused feature map and generates a second defect attention map and a second attribute response based on the high-frequency structural information; residual modulation is applied to the input features of the classification branch based on the first and second defect attention maps; and the classification branch determines the defect state category of the material to be sorted based on the modulated input features. This method maps four instance states to two non-mutually exclusive binary attributes, and through explicit attribute supervision, guides the network to model different types of defect cues, effectively decoupling defect features of different dimensions.

[0012] Secondly, the present invention provides a system for guiding a robot to sort items, comprising: a vision detection module for performing multi-task vision detection on an image containing the material to be sorted, and simultaneously outputting the bounding box, instance mask and root key points of the material to be sorted; and a pose generation module for generating a unidirectional direction vector of the material to be sorted based on the root key points and the instance mask, and generating a grasping pose based on the unidirectional direction vector and a preset grasping strategy.

[0013] The visual detection module includes: a feature extraction unit for performing multi-scale feature extraction and feature fusion on the image to obtain a fused feature map at multiple scales; a segmentation unit for performing bounding box regression and mask prediction based on the fused feature map to output the bounding box and instance mask of the material to be sorted; and a key point localization unit for predicting the root key points of the material to be sorted based on the fused feature map.

[0014] In a preferred embodiment, the pose generation module includes: an anchor point correction unit, used to correct the root key point according to the instance mask, so that the corrected root reference point is located in the material area corresponding to the instance mask; a direction vector generation unit, used to generate a unidirectional direction vector based on the root reference point and the center point of the material to be sorted; a gripping point calculation unit, used to set the gripping point in the direction from the root reference point to the center point, and the distance between the gripping point and the root reference point is a preset ratio of the distance between the root reference point and the center point; and a coordinate transformation unit, used to convert the position and direction angle of the gripping point in the image coordinate system into a gripping pose in the robot coordinate system based on a pixel ratio coefficient and the angle difference between the image coordinate system and the robot coordinate system.

[0015] Thirdly, the present invention provides a robot sorting system, comprising: an image acquisition device for acquiring an image containing materials to be sorted; a visual guidance device including a visual inspection module and a pose generation module; the visual inspection module for performing multi-task visual inspection on the image, and simultaneously outputting the bounding box, instance mask, root key points, and multi-attribute defect status of the materials to be sorted; the pose generation module for generating a unidirectional direction vector of the materials to be sorted based on the root key points and the instance mask, and generating a grasping pose based on the unidirectional direction vector and a preset grasping strategy; and a robot execution module for performing grasping and sorting operations on the materials to be sorted based on the grasping pose.

[0016] In a preferred embodiment, the system further includes: a conveying device for conveying the material to be sorted; an encoder mounted on the conveying device for outputting pulse signals; a control cabinet with a built-in PLC for acquiring the pulse signals and converting them into the displacement and linear velocity of the conveying device, and writing the converted motion information into control variables for the robot execution module to read in real time, so that the robot execution module can track and grasp the material to be sorted on the conveying device; the robot execution module includes a pneumatic flexible gripper, the center point of which is defined at the midpoint between two of its fingers, and the other two fingers are used for lateral support; the robot execution module controls the robot to move along a gantry-shaped motion trajectory and places the material above the target workstation in a vertical release manner; the pose generation module generates the grasping pose and sends control commands only when the grasping point enters a preset trigger window.

[0017] Fourthly, the present invention provides a robot sorting method, comprising the following steps: Step S1, acquiring an image containing materials to be sorted; Step S2, performing multi-task visual detection on the image, and simultaneously outputting the bounding box, instance mask, root key points, and multi-attribute defect status of the materials to be sorted; Step S3, generating a unidirectional direction vector of the materials to be sorted based on the root key points and the instance mask, and generating a grasping pose based on the unidirectional direction vector and a preset grasping strategy; Step S4, controlling the robot to grasp the materials to be sorted based on the grasping pose; Step S5, performing task diversion based on the multi-attribute defect status to determine a target workstation, and controlling the robot to place the materials to be sorted at the target workstation.

[0018] This method directly uses visual inspection results for task routing, and determines the target station (processing station or no-processing station) of the material based on the defect status, realizing a complete automated process from perception to decision-making to execution.

[0019] In a preferred embodiment, in step S1, the acquired image is sent to a frame queue with a frame queue size of 1. When a new image arrives before the previous frame has been processed, the old frame is discarded and the latest frame is retained. The multi-task visual detection in step S2 also includes the aforementioned attribute-based feature enhancement method. In step S3, the root keypoints are corrected based on the instance mask. The gripping point is set in the direction from the root reference point to the center point, and the gripping point position is determined according to a preset ratio. Coordinate transformation is performed based on the pixel ratio coefficient and the angle difference between the image coordinate system and the robot coordinate system.

[0020] Beneficial effects The present invention has the following beneficial effects: First, by using the root key point as the visual anchor point for direction perception and combining it with instance mask for correction, this invention can generate a unidirectional direction vector with clear head and tail semantics, fundamentally solving the 180° direction ambiguity problem in existing rotating target detection methods, enabling robots to accurately distinguish the bottom and top of materials and achieve precise orientation placement.

[0021] Second, by mapping multi-attribute defect states to multiple non-mutually exclusive binary attributes and using an attribute-guided feature enhancement module to introduce attribute responses into classification features, this invention effectively decouples defect features of different dimensions, reduces mutual competition between features, and improves the accuracy and stability of identifying composite defect samples.

[0022] Third, the present invention sets the gripping point in the direction from the root reference point to the center of the material and close to the rigid base area, which takes into account both stable gripping and mechanical constraints, avoids damage to vulnerable parts, and realizes a complete automated process from visual perception to gripping pose generation to robot execution.

[0023] Fourth, this invention achieves integrated perception of multiple tasks by synchronously outputting bounding boxes, instance masks, root key points, and multi-attribute defect states through a visual inspection module, reducing computational redundancy and latency in the system, and is suitable for high-speed assembly line sorting scenarios. Attached Figure Description

[0024] Figure 1 This is a flowchart of the sorting system of the present invention; Figure 2 This is a schematic diagram of the hardware structure of the sorting system of the present invention; Figure 3 This is a top view of the hardware of the sorting system of the present invention; Figure 4 This is a schematic diagram of the material defect attribute decoupling method of the present invention; Figure 5 This is a schematic diagram of the structure of the visual inspection module of the present invention; Figure 6 This is a schematic diagram of the pose generation process of the present invention; Figure 7 This is a schematic diagram of the parallel robotic arm portal motion control according to the present invention.

[0025] The meanings of the markings in the attached diagram are as follows: 1—Four-DOF parallel robotic arm; 2—Pneumatic flexible gripper; 3—Industrial camera; 4—Feeding conveyor belt; 5—Incoming material conveyor belt; 6—Pneumatic circuit; 7—Control cabinet; 8—Frame. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0027] The vision algorithm-guided robot sorting system proposed in this invention includes an image acquisition device, a vision guidance device, and a robot execution module, such as... Figure 1 As shown, the system encompasses four stages: visual perception, task decision-making, motion control, and robot execution. The image acquisition device acquires images containing the materials to be sorted; the visual guidance device includes a visual inspection module and a pose generation module; the visual inspection module performs multi-task visual inspection on the images, simultaneously outputting the bounding box, instance mask, root key points, and multi-attribute defect states of the materials to be sorted; the pose generation module generates a unidirectional direction vector of the materials to be sorted based on the root key points and instance mask, and generates a grasping pose based on the unidirectional direction vector and a preset grasping strategy; the robot execution module performs grasping and sorting operations on the materials to be sorted according to the grasping pose.

[0028] The correspondence between its functional modules and hardware devices is as follows: The image acquisition device corresponds to industrial camera 3 (see...). Figure 2 , Figure 3 The system is used to acquire RGB images containing materials to be sorted. The vision detection module and pose generation module run in a host computer. The host computer uses the RGB images from the industrial camera 3 as input, detects the materials using the proposed vision detection algorithm, and generates gripping and placement position commands using a decision algorithm. The robot execution module corresponds to the four-degree-of-freedom parallel robot 1 and its end-effector pneumatic flexible gripper 2. The parallel robot, combined with conveyor belt tracking information, completes line tracking gripping, root-end redirection, and sorting placement through the pneumatic flexible gripper 2. The conveying device corresponds to the incoming material conveyor belt 5 and the feeding conveyor belt 4. The encoder is installed on the incoming material conveyor belt 5 to acquire conveyor belt motion information. The control cabinet 7 has a built-in PLC for acquiring encoder pulse signals and converting them into conveyor belt displacement and linear velocity.

[0029] The following detailed description, in conjunction with the accompanying drawings, illustrates the specific implementation methods of each component of the system in the application of vegetable directional sorting.

[0030] (1) Specific implementation method of the image acquisition device: like Figure 2 and Figure 3As shown, this invention selects a Hikvision industrial GigE camera MV-CS020-10GC as the image acquisition device, i.e., industrial camera 3. Camera 3 is located inside a vision lightbox and uses a hardware-triggered mode for image acquisition, ensuring synchronization between the image capture process and the conveyor loading process. The camera and lens are mounted inside a lightbox above the conveyor belt, which provides supplementary illumination to the detection area. After the camera enters the acquisition state, the system receives images through an independent acquisition thread and decodes the original Bayer RG image into an RGB image. The converted image is sent to a frame queue, awaiting subsequent visual inference.

[0031] To reduce system latency during continuous operation, the frame queue size is set to 1. When a new image arrives before the previous frame has been processed, the system automatically discards the old frame and retains the latest one. In this way, the perception program always performs detection and pose generation based on the latest image, thereby improving real-time performance in conveyor belt dynamic grasping scenarios.

[0032] (2) Specific implementation of the visual inspection module: The visual inspection module runs on the host computer, taking the RGB image captured by the industrial camera 3 as input, and executes a multi-task visual inspection algorithm.

[0033] Unlike conventional detection models that only focus on target localization and category prediction, this method explicitly introduces directional visual anchors and multi-attribute feature enhancement mechanisms. The root keypoints not only serve for spatial localization but also provide a semantically informative directional reference for subsequent robot repositioning; see [link to relevant documentation]. Figure 4 The attribute branch decouples the complex post-collection state into two non-mutually exclusive binary attributes, thereby guiding the classification branch to focus more on the local discriminative region.

[0034] In terms of overall architecture, the visual inspection module extracts features through a multi-task decoupled prediction head and enhances the network's learning of local discrimination regions using an attribute-guided mechanism. Building on this, the network combines root visual anchors to achieve orientation perception, ultimately transforming visual information into task-driven grasping poses, thus completing the connection from front-end perception to back-end execution.

[0035] like Figure 5As shown, the visual detection module follows a single-stage multi-scale detection framework, retaining the backbone and neck of YOLO11-seg to provide effective multi-scale feature extraction and fusion capabilities. Task-related improvements are concentrated in the decoupling prediction head and attribute-guided feature enhancement modules. The input RGB image first passes through the backbone network to extract hierarchical visual features, and then the neck structure fuses shallow spatial details and deep semantic information in the form of a feature pyramid. The fused multi-scale feature maps are labeled P3, P4, and P5, corresponding to different receptive fields, used to perceive the differences in scale, posture, and blade unfolding morphology of irregular materials on the conveyor belt.

[0036] For agricultural product processing and sorting tasks, it is necessary to identify defects in the materials. Take vegetables containing yellow leaves and post-harvest root residues as an example. Yellowed leaf defects are mainly manifested as abnormal color cues, while root residue defects are more often expressed as small-scale structural cues near the root tip. Based on this characteristic, the visual detection module of this invention constructs a multi-task decoupled prediction head on the basis of shared multi-scale features, integrating the above-mentioned perception tasks into a unified network through task branches.

[0037] Specifically, the multi-task decoupled prediction head includes five task branches: segmentation, keypoint localization, feature enhancement for yellowing leaf defect detection, feature enhancement for root residual defect detection, and multi-attribute classification. The segmentation branch includes bounding box regression and mask prediction. The bounding box provides target location information, and the instance mask predicts mask coefficients based on multi-scale features, while prototype mask features are generated from P3 features. The keypoint branch provides visual anchor points for orientation perception. The multi-attribute feature enhancement module is responsible for the multi-attribute classification branch, predicting the final four types of defect states.

[0038] (3) Specific implementation methods for attribute-based feature enhancement: The attribute enhancement module includes an attribute prediction head to predict two binary attributes: yellow leaf defect attribute and root residual defect attribute. It then generates corresponding attribute response maps using a yellow leaf defect attention map and a root residual defect heatmap. Subsequently, the attribute enhancement module incorporates these attribute responses into classification features, enabling the classification head to enhance its perception of local discrimination cues such as yellow leaf defect areas and root residual defect areas when distinguishing between the four defect states.

[0039] like Figure 4As shown, in the task scenario proposed in this invention, the four defect states of the target material are not simply mutually exclusive visual categories. Specifically, the yellowing leaf defect manifests as pigment degradation and color deterioration near the base of the petiole in terms of appearance; the root residue defect manifests as residual root and stem structures that were not completely removed in terms of physical structure. These two abnormal states are not mutually exclusive, but rather exhibit independent or co-existing distributions. If the traditional single-label mutually exclusive classification paradigm is directly adopted, the network will be forced to forcibly couple the two different dimensions of visual cues—color abnormalities and structural abnormalities—to a single category label. This paradigm not only intensifies the competition between different features but also hinders the detection network from decoupling and modeling composite state samples.

[0040] To alleviate the aforementioned feature coupling problem, this invention introduces an attribute decoupling learning strategy, mapping the four types of instance states to two non-mutually exclusive binary attributes. The attribute vector is represented as follows: ,in, This indicates whether there are defects such as yellowed leaf remnants at the base. This indicates whether root residual defects exist. It shows the mapping relationship between the four defect states and this binary attribute vector.

[0041] During network training, the attribute prediction branch is independently supervised using binary cross-entropy loss. For a given single material instance, its attribute decoupling loss function is defined as:

[0042] in, This represents the probability value of the corresponding attribute predicted by the network.

[0043] The role of the attribute branch is to provide explicit attribute supervision for yellow leaf defects and root residue defects, enabling the network to model color anomaly cues and local structural anomaly cues separately based on shared features. The attribute responses generated by the attribute branch will serve as the basis for feature enhancement in the sub-attribute feature enhancement module, used to modulate the input features of the classification branch. This allows the main classification branch to pay more attention to local discrimination cues such as yellow leaf regions and root residue regions during the discrimination of the four types of defect states. Through this joint learning approach, the visual detection module proposed in this invention can reduce the feature coupling between yellow leaf attributes and root residue attributes, and provide clearer supervisory constraints for the stable identification of composite state samples.

[0044] Although attribute decoupling learning provides explicit supervision for yellowing leaf defects and root residue defects, attribute labels themselves can only characterize the presence or absence of a certain attribute and cannot directly indicate the specific location of the discrimination region in the spatial dimension. The visual distribution of yellowing leaf defects and root residue defects differs significantly. The former is mostly distributed near the base of the material, with highly irregular shapes and easily affected by lighting, reflection, and occlusion; the latter typically manifests as small-scale fine structures near the bottom, with a small target area and is easily occluded. During continuous downsampling and multi-scale feature aggregation in deep networks, these weak and scale-specific local cues are easily weakened by the responses of large normal areas, leading to insufficient attention from classification features to yellowing leaf defect areas and root residue defect areas. Therefore, the visual detection module proposed in this invention further designs an attribute-guided feature enhancement module. Through a feature guidance path from the attribute branch to the classification branch, the classification branch pays more attention to the local discrimination regions of material color defects and structural defects.

[0045] For the yellow leaf defect detection enhancement branch, a color defect prior is introduced as an additional single-channel input. For each scale feature... First, the color defect prior is scaled to the spatial resolution of the corresponding feature map, and then... Convolution and SiLU activation are used for projection, aligning it with the feature space representation. Simultaneously, the input features... After processing using two complementary paths: one local path adopts... Convolution is used to capture fine-grained color defect responses; another context path is followed by average pooling. Convolution is used to aggregate global contextual information. Subsequently, local features, average pooling-based contextual features, and projected color defect priors are concatenated and then... 3. Convolutional processing is used for fusion. Based on the fused color defect perception features, the attribute-based feature enhancement module uses... Convolution generates yellow leaf attention logits, which are then activated by a sigmoid function to obtain a yellow leaf defect attention map. Meanwhile, another... The convolutional branch is used to predict the yellow leaf defect attribute response. In this way, the color defect prior provides weak spatial guidance for potential yellowing regions, while the local feature path and contextual feature path enable the yellow leaf defect detection enhancement branch to focus on color degradation regions rather than producing a uniform response across the entire material.

[0046] For root residual defect attributes, the sub-attribute feature enhancement module introduces a root residual defect detection enhancement branch to capture small-scale structural responses near the root tip. For each scale feature... This branch first subtracts the average pooling feature from the original feature, i.e. To extract high-frequency structural information, and then through Convolution is used for projection. This high-frequency representation can highlight local edge-class information and residual details of local structural features, which are easily weakened in high-level semantic features. Subsequently, the projected high-frequency features are further processed... Convolution and Convolution generates root response logits, which are then converted into root defect attention maps using sigmoid activation. Simultaneously, root attribute responses are predicted by fusing local root features, average pooling-based contextual features, and high-frequency structural features, enabling this branch to utilize both fine-grained bottom-level structural information and global material context.

[0047] During the training phase, the labeled local keypoints are converted into Gaussian heatmaps to provide spatial supervision for the root residual defect response branches. The root keypoints have a dual role in this invention. During the inference phase, they serve as root-end visual anchors for root-end orientation perception; during the training phase, they are used to construct a two-dimensional Gaussian heatmap to constrain the root defect response branches to focus on small-scale structural regions near the root end. Let the root keypoints be... The corresponding root-end Gaussian heatmap is defined as

[0048] in, Controlling the diffusion range of root-end spatial supervision. Considering that root remnants are small-scale structural targets, root spatial supervision is mainly applied to high-resolution feature layers. and This is to preserve local details near the root tip. The model retains the yellow leaf defect response and global semantic enhancement, but does not apply root residual defect Gaussian heatmap supervision.

[0049] Based on the above mechanism, the attribute-specific feature enhancement module generates yellow leaf defect attention maps respectively. Attention map of residual defects at the root And residual modulation is performed on the input features of the classification branch using learnable weights:

[0050]

[0051] in, and These represent the output responses logits of the yellow leaf defect detection branch and the root defect detection branch, respectively. For the Sigmoid function, and The modulation coefficients are learnable and initialized to 0.05. This indicates element-wise multiplication. This indicates the activation status of the root defect spatial response at different scales, when hour 1. When 5 o'clock Through this process, yellow leaf defect responses and root defect responses are introduced into classification features, enabling the classification branches to simultaneously utilize both overall material semantics and local attribute cues when distinguishing between the four types of defect states.

[0052] (4) The specific implementation method of the pose generation module is as follows: In the sorting task of this invention, the robot needs to stably grasp incoming materials with random posture distribution and redirect them to a designated workstation. This process requires not only the vision system to provide the two-dimensional position of the target in the image plane, but also to provide a semantic direction with clear head-to-tail orientation. The direction required for robot motion planning should be modeled as defined within a complete cycle. A unidirectional planar vector within the material to strictly distinguish between the bottom and top of the material.

[0053] However, existing methods for rotating target detection typically use directed bounding boxes (DWTs) to describe the target pose. While DWTs can fit the geometric envelope of a rotating target well, they essentially describe the principal axes of a rectangular bounding box, rather than unidirectional directions with head-to-tail semantics. Due to the central symmetry of the rectangular bounding box, the principal axes of the DWT satisfy... and Equivalence means that two poses with an angle difference of 180° can correspond to the same geometric envelope. Therefore, the effective representation space of the directed bounding box angle is the undirected axis direction of half a period, which can be expressed in common parameterization as: The semi-periodic interval. This representation inherently contains ambiguity regarding 180° flips, making it difficult to directly meet the semantic distinction between root and leaf tips in root cutting tasks.

[0054] See Figure 6 This invention uses the root key point as the visual anchor point for direction perception. The root region is not only a key area for judging the attributes of residual defects in the root, but also a physical reference end that needs to be aligned with the processing station when the robot is redirected. Therefore, the vision inspection module predicts the root end position through the root key point branches and constructs a unidirectional root end direction vector by combining it with the center point of the material.

[0055] To improve the physical consistency of the root visual anchor points, local correction is further performed using the output instance mask. Since keypoint regression may be affected by discrete prediction errors, the initial root keypoints predicted by the network... It may deviate from the actual visible area of ​​the material. After obtaining the instance mask, along... To the material center point unit direction Perform correction. .like If the point is outside the effective mask area, it will be projected to the nearest effective mask point; otherwise, it will be projected along... Move backward to the root side mask boundary, that is:

[0056] in, .

[0057] This correction process ensures the root visual anchor point. Located in the material mask area This improves the spatial consistency of subsequent pose construction.

[0058] After obtaining the corrected root reference point Then, combine the material center obtained from the centroid of the instance mask or the center of the bounding box. Establish a local coordinate system for the material within the image plane. Among them, local The axis is from the reference point Point to the center of the green vegetable Its attitude reference direction vector for:

[0059] Principal axis rotation angle of the target in the image plane Determined by the arctangent function:

[0060] in, Represents the local coordinate system of the material Axis relative to image coordinate system The directed angle of the axis. According to the right-hand rule, locally... The axis lies in the image plane and is parallel to it. The axes are orthogonal, used to describe the complete local pose of the material within the image plane. At this point, the root visual anchor point and the semantic orientation of the material's head and tail are determined, providing the geometric basis for subsequent grasping pose generation.

[0061] After completing the root-end visual anchor point correction and establishing the material's local coordinate system, the output of the proposed visual inspection module is converted into a grasping pose executable by the parallel robot. Traditional end-to-end grasping and inspection methods typically focus primarily on stable gripping points, while the grasping pose generation of this invention simultaneously serves three objectives: stable grasping, redirection, and process station diversion. Therefore, the gripping point not only needs to avoid the wide and vulnerable canopy area but also needs to ensure that the root end direction after grasping is aligned with the downstream root cutting station.

[0062] Based on the root reference points determined above Material Center and local attitude direction vector This invention defines the grab point locally. On the axis. Let This indicates that the crawl point is local. The projected distance on the axis is the grab point. It can be represented as:

[0063] In the implementation, the grab point is located at the root reference point. With Material Center Along the direction of the line connecting the stems, near the base of the stem, to improve the stability of the flexible gripper's grasp. Let... Indicates the crawl point is at arrive The proportional coefficients on the connecting lines are:

[0064] The present invention is based on actual grasping test settings. That is, the grab point is located from the root reference point. Pointing to the material center One-third of the way. Accordingly, the grab point is in the local area. The projected distance on the axis is:

[0065] This gripping point constraint positions the gripping area closer to the relatively rigid base of the petiole, rather than the easily deformable crown region, thus reducing compressive damage to the blade during gripping and high-speed transfer. Simultaneously, the orientation of the grippers within the image plane is related to the local... The axes are approximately orthogonal, allowing the flexible fingers to laterally wrap around the base of the petiole, improving posture stability during the grasping process.

[0066] Subsequently, based on the pixel ratio and the angle difference between the image coordinate system and the robot coordinate system, the gripping points in the image plane are converted into robot plane coordinates. The image coordinate system is represented as follows: ,in The axis extends to the right along the horizontal direction of the image. The axis points downwards along the vertical direction of the image. Let the center of the image be... The pixel ratio factor is The angle difference between the image coordinate system and the robot coordinate system is In this invention implementation, , For the crawl point Its physical displacement relative to the image center is:

[0067] After angle difference compensation, the planar position sent to the robot controller is:

[0068]

[0069] Meanwhile, the end effector rotation angle sent to the robot controller is:

[0070] in, and This represents the planar grasping position in the robot's coordinate system. In actual calculations, In coordinate rotation calculations, the coordinates are converted to radians for trigonometric function operations, while compensation is performed in angular form in the end-effector angle command. The robot end-effector coordinate system is represented as follows: ,in The axis is perpendicular to the end. and The axis describes the orientation of the gripper in the horizontal plane.

[0071] To ensure gripping stability under continuous transport conditions, the system only grips the target point. The final control command is generated and sent only when the target enters the preset trigger window, avoiding triggering the grasping before the target enters the robot's effective working area or is about to leave the grasping area, thereby improving the timing stability of the conveyor belt's dynamic grasping. Simultaneously, the system allocates tasks based on the defect attribute status output by the vision inspection module. If the target has defects such as root residue, yellowed leaf residue at the base, or both, it is determined that root cutting is required and assigned to the root cutting station; if the target does not have either of these two abnormal defect attributes, it is determined that root cutting is not required and assigned to the non-cutting station. Therefore, the visual perception results are not only used to generate the grasping pose but also directly participate in subsequent placement path and station selection.

[0072] The host computer sends the grasping pose and task information to the robot controller via the TCP / IP protocol. The communication data frame format is defined as follows: .in, , and The pose generation process described above calculates the planar grasping position and the end effector angle command in the robot coordinate system, respectively. This indicates the robot execution unit number, which is set to 1 in the system of this invention, indicating that the current task is executed by the parallel robot number 1. Indicates the instance number of the current target; This indicates the target placement point in the robot's Cartesian coordinate system, determined based on material properties. Coordinates are used to instruct the robot to perform the corresponding placement action.

[0073] (5) The specific implementation method of the robot execution module is as follows: The robot execution module corresponds to the four-degree-of-freedom parallel robot 1 and its end effector-mounted pneumatic flexible gripper 2. The four-degree-of-freedom parallel robot 1 used in this invention is a four-axis Delta parallel robot, model D3PB-600-P3. The pneumatic flexible gripper 2 is a RouTouch FM-A6V3, installed at the end effector of the four-degree-of-freedom parallel robot 1, used for low-damage material gripping. This gripper is driven by compressed air supplied by an external air compressor, and a stable working pressure is maintained during repeated gripping through a pneumatic circuit with pressure regulation function.

[0074] In the gripping strategy proposed in this scheme, the geometric center of the four-finger gripper is not used as the tool center point (TCP). Instead, the midpoint between two of the fingers is defined as the gripping TCP, and a corresponding offset is introduced during the end effector calibration process. The two selected fingers are used to provide the main gripping function for flexible materials, while the other two fingers serve as lateral support to prevent the material from being thrown out or slipping due to inertia during the robot's high-speed movement.

[0075] After receiving the structured data, the robot controller combines it with conveyor belt tracking information to perform grasping, redirection, and sorting / placement actions. Through the proposed grasping pose generation method, the multi-dimensional visual output of the proposed vision detection module is converted into grasping and placement instructions executable by the robot, achieving automated material sorting under continuous conveying conditions.

[0076] like Figure 7 As shown, the parallel robot employs a gantry-shaped motion trajectory to perform continuous actions such as conveyor belt tracking, gripping, lifting, transferring, redirecting, and placing. During the gripping phase, the robot compensates for the position of the moving target based on feedback from the feed conveyor belt encoder and performs conveyor belt tracking motion on the moving target, moving it from position P0 to position P1. Subsequently, the pneumatic flexible gripper closes at the target gripping position, achieving stable clamping of the material.

[0077] After grasping, the robot sequentially passes through positions P1, P2, P3, and P4, completing the processes of vertical lifting, transfer above the work area, repositioning, and placement at the target station. During the transfer, the end effector adjusts its rotation angle based on the root-end orientation predicted by the vision system, ensuring root-end alignment of the material before reaching the target station. This gate-shaped trajectory reduces lateral disturbance to flexible materials during high-speed transport and ensures that the material enters the processing or non-processing station with the required root-end orientation.

[0078] During the placement phase, a vertical release is employed. When the robot moves to position P4, the grippers open in advance above the target workstation, allowing the material to fall to the target position under the combined action of gravity and the residual downward velocity of the end effector. Compared to lateral unloading, this strategy reduces the adhesion between irregular materials and the flexible grippers, avoids horizontal friction and torsional disturbances during the release process, and thus improves the reliability of material separation from the grippers.

[0079] (6) The specific implementation method of the conveying device and encoder tracking is as follows: See Figure 2 The conveying device includes an incoming material conveyor belt 5 and a feeding conveyor belt 4. The incoming material conveyor belt 5 is used to transport the materials to be sorted from the loading area to the vision inspection area and the gripping area. The feeding conveyor belt 4 is used to transport the sorted materials to the downstream workstation.

[0080] This invention involves installing an encoder on the incoming material conveyor belt to acquire conveyor belt motion information during continuous feeding. The encoder outputs pulse signals, the frequency and number of which reflect the conveyor belt's speed and displacement. These encoder pulse signals are acquired by a PLC within the control cabinet and converted into conveyor belt displacement and linear velocity in the robot control program. The converted motion information is written into control variables for real-time reading by the parallel robot, enabling it to update the target position based on the conveyor belt's movement and thus complete conveyor belt tracking and grasping.

[0081] (7) Taking vegetable sorting as an example, the robot sorting method of the present invention will be further explained.

[0082] In the vegetable sorting scenario, vegetables containing yellow leaf defects and / or root residue defects are randomly distributed on the incoming conveyor belt 5. The industrial camera 3 acquires RGB images of the vegetables on the conveyor belt, and the vision inspection module takes the image as input and synchronously outputs the bounding box, instance mask, root key points, and four types of defect states (no defects, only yellow leaves, only root residue, and combined defects) for each vegetable instance.

[0083] The pose generation module (running in the host computer) corrects the root reference point based on the root keypoints and instance mask, and generates a unidirectional direction vector from the root to the leaf. Then, the gripping point is determined at one-third of the distance from the root reference point to the center point, and after coordinate transformation, it is sent to the robot controller.

[0084] The four-degree-of-freedom parallel robot 1 executes a gate-shaped trajectory motion according to the grasping posture, closes the pneumatic flexible gripper 2 at the grasping point to complete the grasping, adjusts the end posture according to the direction vector during the transfer process so that the root end of the vegetable is aligned with the target direction, and finally places the vegetable to the corresponding processing station (L2 station) or non-processing station (L1 station) according to the defect status.

[0085] During operation, the robot performs conveyor belt tracking and grasping based on the received target grasping pose and placement position information, and dynamically adjusts the placement point according to the material's state and shape. Materials determined to require processing are placed at station L2, while materials not requiring processing are placed at station L1.

[0086] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for guiding a robot to sort items, characterized in that, Includes the following steps: Step A: Obtain an image containing the materials to be sorted; Step B: Perform multi-task visual inspection on the image and simultaneously output the bounding box, instance mask, and root key points of the material to be sorted. Step C: Generate a unidirectional direction vector of the material to be sorted based on the root key points and instance mask; Step D: Generate a grasping pose based on the unidirectional direction vector and the preset grasping strategy.

2. The method for guiding a robot to sort items according to claim 1, characterized in that, The multi-task visual detection in step B includes: Multi-scale feature extraction and feature fusion are performed on the image to obtain a fused feature map at multiple scales; Based on the fused feature map, bounding box regression and mask prediction are performed, and the bounding box and instance mask are output. The root key points are predicted based on the fused feature map.

3. The method for guiding a robot to sort items according to claim 1, characterized in that, The multi-task visual detection in step B further includes an attribute-based feature enhancement method: The attribute prediction branch predicts whether the material to be sorted has a first attribute defect and whether it has a second attribute defect; the first defect enhancement branch aggregates local and contextual features in the fused feature map based on color defect priors to generate a first defect attention map and a first attribute response; the second defect enhancement branch extracts high-frequency structural information from the fused feature map and generates a second defect attention map and a second attribute response based on the high-frequency structural information; residual modulation is applied to the input features of the classification branch based on the first and second defect attention maps; and the classification branch determines the defect state category of the material to be sorted based on the modulated input features. In step C, the root key point is corrected according to the instance mask so that the corrected root reference point is located in the material area corresponding to the instance mask. A unidirectional direction vector is generated based on the root reference point and the center point of the material to be sorted. In step D, the gripping point is set in the direction from the root reference point to the center point, and the distance between the gripping point and the root reference point is a preset ratio of the distance between the root reference point and the center point; based on the pixel ratio coefficient and the angle difference between the image coordinate system and the robot coordinate system, the position and orientation angle of the gripping point in the image coordinate system are converted into the gripping pose in the robot coordinate system.

4. A system for guiding a robot to sort items, characterized in that, include: The visual inspection module is used to perform multi-task visual inspection on images containing materials to be sorted, and simultaneously output the bounding box, instance mask and root key points of the materials to be sorted. The pose generation module is used to generate a unidirectional direction vector of the material to be sorted based on the root key points and instance mask, and to generate a grasping pose based on the unidirectional direction vector and a preset grasping strategy.

5. The system for guiding a robot to sort items according to claim 4, characterized in that, The visual detection module includes: The feature extraction unit is used to perform multi-scale feature extraction and feature fusion on the image to obtain a fused feature map at multiple scales. The segmentation unit is used to perform bounding box regression and mask prediction based on the fused feature map, and output the bounding box and instance mask of the material to be sorted. A key point localization unit is used to predict the root key points of the material to be sorted based on the fused feature map.

6. The system for guiding a robot to sort items according to claim 4, characterized in that, The pose generation module includes: Anchor point correction unit is used to correct the root key point according to the instance mask, so that the corrected root reference point is located in the material area corresponding to the instance mask; A direction vector generation unit is used to generate a unidirectional direction vector based on the root reference point and the center point of the material to be sorted. The gripping point calculation unit is used to set the gripping point in the direction from the root reference point to the center point, and the distance between the gripping point and the root reference point is a preset ratio of the distance between the root reference point and the center point. The coordinate transformation unit is used to convert the position and orientation angle of the grasping point in the image coordinate system into the grasping pose in the robot coordinate system based on the pixel scaling factor and the angle difference between the image coordinate system and the robot coordinate system.

7. A robotic sorting system, characterized in that, include: Image acquisition device, used to acquire images containing materials to be sorted; A visual guidance device, comprising a visual detection module and a pose generation module; The visual inspection module is used to perform multi-task visual inspection on the image and simultaneously output the bounding box, instance mask, root key point and multi-attribute defect status of the material to be sorted; the pose generation module is used to generate a unidirectional direction vector of the material to be sorted based on the root key point and instance mask, and generate a grasping pose based on the unidirectional direction vector and the preset grasping strategy. The robot execution module is used to perform gripping and sorting operations on the materials to be sorted according to the gripping posture.

8. The robot sorting system according to claim 7, characterized in that, Also includes: A conveying device for conveying the materials to be sorted; An encoder, mounted on the conveying device, is used to output pulse signals; The control cabinet has a built-in PLC, which is used to collect the pulse signals and convert them into the displacement and linear velocity of the conveying device. The converted motion information is written into the control variables for the robot execution module to read in real time, so that the robot execution module can track and grab the materials to be sorted on the conveying device. The robot execution module includes a pneumatic flexible gripper, the center point of which is defined at the midpoint between two of its fingers, and the other two fingers are used for lateral support; the robot execution module controls the robot to move along a gate-shaped motion trajectory and place materials above the target workstation in a vertical release manner; The pose generation module generates a capture pose and sends control commands only when the capture point enters the preset trigger window.

9. A robot sorting method, characterized in that, Includes the following steps: Step S1: Acquire an image containing the materials to be sorted; Step S2: Perform multi-task visual inspection on the image and simultaneously output the bounding box, instance mask, root key point and multi-attribute defect status of the material to be sorted. Step S3: Generate a unidirectional direction vector of the material to be sorted based on the root key points and the instance mask, and generate a grasping pose based on the unidirectional direction vector and the preset grasping strategy. Step S4: Control the robot to grasp the materials to be sorted according to the grasping pose; Step S5: Based on the multi-attribute defect status, the task is diverted to determine the target workstation, and the robot is controlled to place the material to be sorted into the target workstation.

10. The robot sorting method according to claim 9, characterized in that, In step S1, the acquired image is sent to the frame queue, the frame queue size is set to 1, and when a new image arrives before the previous frame has been processed, the old frame is discarded and the latest frame is retained. In step S2, the multi-task visual detection further includes an attribute-based feature enhancement method: predicting whether the material to be sorted has a first attribute defect and whether it has a second attribute defect through an attribute prediction branch; performing local feature and context feature aggregation processing on the fused feature map based on color defect prior through a first defect enhancement branch to generate a first defect attention map and a first attribute response; extracting high-frequency structural information from the fused feature map through a second defect enhancement branch, and generating a second defect attention map and a second attribute response based on the high-frequency structural information; and performing residual modulation on the input features of the classification branch based on the first defect attention map and the second defect attention map. The classification branch determines the defect status category of the material to be sorted based on the modulated input features; In step S3, the root key point is corrected according to the instance mask so that the corrected root reference point is located in the material area corresponding to the instance mask. A unidirectional direction vector is generated based on the root reference point and the center point of the material to be sorted. The gripping point is set in the direction from the root reference point to the center point, and the distance between the gripping point and the root reference point is a preset ratio of the distance between the root reference point and the center point. Based on the pixel ratio coefficient and the angle difference between the image coordinate system and the robot coordinate system, the position and direction angle of the gripping point in the image coordinate system are converted into the gripping pose in the robot coordinate system.

Images