A kind of method, system and equipment for visual identification and pose determination of material package based on RGB image and depth image fusion
Patent Information
- Application Number
- CN202611156675.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-31
- Publication Date
- 2026-10-27
AI Technical Summary
现有方法的排序规则固定，缺少结合业务需求（如按高度优先、按行排列等）的灵活输出机制，难以适应不同产线的定制化需求
[0019]本发明的有益效果：识别稳定性更高。 同时融合RGB图像与深度图像信息，兼顾纹理特征与空间特征，即使单模态检测结果存在漏检或误检，也可通过另一模态进行补全，提高复杂场景下识别鲁棒性。
Smart Images
Figure CN122990400A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of machine vision, 3D point cloud processing and industrial automation recognition technology, and in particular to a method, system and device for material bag target recognition and pose determination based on the fusion of RGB images, depth images and ordered point clouds. Background Technology
[0002] In industrial scenarios such as logistics sorting, bagged goods handling, and stacking grasping, it is often necessary to quickly identify packaged targets and output their spatial position, orientation angle, and sorting results for robots to perform grasping operations. Packaged targets typically have regular shapes but similar surface textures and are stacked, which brings many challenges to visual recognition systems.
[0003] Existing technologies typically have the following problems: First, recognition based solely on RGB images suffers from poor stability. RGB images are significantly affected by changes in lighting, interference from backgrounds with similar colors, and mutual occlusion between targets, making it difficult to reliably obtain depth information of the target. When the surface texture of the material bag is weak or the lighting conditions change drastically, the performance of the RGB recognition model degrades significantly.
[0004] Secondly, recognition based solely on depth images is prone to target merging. While depth images provide spatial geometric information about targets, adjacent or stacked material bags are prone to "merging" on the depth map, meaning multiple targets appear as one in the depth domain. This can lead to multiple targets being misidentified as a single target, severely impacting the accuracy of subsequent grasping operations.
[0005] Third, existing recognition methods lack 3D pose information that can be directly used for robot grasping. Existing methods typically only output 2D detection boxes, lacking an effective mapping with 3D point clouds, making it difficult to directly obtain reliable grasping poses (including the 3D spatial position, attitude angles, and size information of the target center). Additional post-processing steps are required to convert 2D detection results into 3D grasping information, which is cumbersome and has limited accuracy.
[0006] Fourth, there is a lack of flexible sorting and output mechanisms in multi-objective scenarios. In industrial production line applications, robot grasping needs to be executed according to a certain priority order. Existing methods have fixed sorting rules and lack flexible output mechanisms that can be combined with business needs (such as height priority, row arrangement, etc.), making it difficult to adapt to the customized needs of different production lines.
[0007] Therefore, a visual recognition scheme is needed that can simultaneously utilize RGB images, depth images, and point cloud information, and can achieve target fusion recognition, 3D pose estimation, and result ranking output. Summary of the Invention
[0008] The technical problem to be solved by the present invention is that the prior art is prone to target adhesion when recognition is based solely on depth images. In order to solve the above problem, a method, system and device for visual recognition and pose determination of material bags based on the fusion of RGB images and depth images are provided.
[0009] The object of this invention is achieved in the following manner: A method for visual recognition and pose determination of material packages based on the fusion of RGB and depth images, comprising: Acquire the RGB image, depth image, and ordered point cloud corresponding to the pixels of the depth image of the target scene; Target detection is performed on the RGB image and the depth image respectively to obtain an RGB instance set and a depth instance set. The RGB instance set contains the mask of each RGB instance, and the depth instance set contains the mask of each depth instance. The masks of the RGB instances and the masks of the depth instances are mapped to a unified point cloud coordinate space, and the cross-union ratio between each RGB instance mask and each depth instance mask is calculated. Based on the comparison result of the cross-union ratio and a preset threshold, a cross-modal association relationship between the RGB instances and the depth instances is established. Based on the cross-modal association, the RGB instance and the depth instance are fused to generate at least one fused instance and a corresponding fused mask; Specifically, when it is determined that a depth instance corresponds to multiple RGB instances based on the cross-modal association relationship, the mask of the depth instance is segmented using the mask of the multiple RGB instances, thereby splitting the depth instance into multiple fused instances; Based on the effective pixel positions in the fusion mask, extract the target point cloud corresponding to the fusion mask from the ordered point cloud; The three-dimensional spatial information of the material bag is calculated based on the target point, and the three-dimensional spatial information includes the center position, size information and attitude angle; Output the identification results for each material package.
[0010] Also includes: When a depth instance corresponds to an RGB instance, a union operation is performed on the mask of the RGB instance and the mask of the depth instance to generate a fused mask; When a depth instance does not correspond to any RGB instance, the depth instance is directly retained as the fusion instance; When an RGB instance does not correspond to any depth instance, the RGB instance is directly retained as the fusion instance.
[0011] Mapping the mask of the RGB instance and the mask of the depth instance to a unified point cloud coordinate space includes: scaling the mask of the RGB instance and the mask of the depth instance to the same target size according to the ratio between their original image size and point cloud size.
[0012] The effective pixel position is the pixel position in the fusion mask that belongs to the target foreground region and has a corresponding non-zero depth value; the step of extracting the target point cloud from the ordered point cloud based on the effective pixel position in the fusion mask includes: traversing the fusion mask pixel by pixel, directly extracting the corresponding 3D point from the ordered point cloud for the effective pixel position in the fusion mask through pixel index, and discarding non-finite points.
[0013] The step of calculating the attitude angle of the material package based on the target point cloud includes: projecting the target point cloud onto a two-dimensional plane, calculating the minimum bounding rectangle of the two-dimensional projection point set of the target point cloud, and obtaining the attitude angle of the material package based on the minimum bounding rectangle.
[0014] Before performing RGB instance segmentation, a depth-first determination step is also included: the depth instance set is filtered according to a confidence threshold. When the number of filtered depth instances reaches a preset threshold, the target point cloud is extracted based on the depth instance and the corresponding ordered point cloud and the three-dimensional spatial information is calculated. If the number of valid targets obtained meets the preset requirements, the recognition result is directly output; otherwise, RGB instance segmentation continues.
[0015] The depth image is normalized and expanded into a multi-channel image before being input into the depth detection model; The identified multiple material packages are sorted according to a preset sorting rule, which includes at least one of the following: sorting by Z direction priority, sorting by X direction priority followed by Y direction priority, and sorting by Y direction priority followed by X direction priority. The target size is verified. When the size of the target point cloud along the length direction meets the preset double-packet criterion, the target is split into two independent sub-targets, and the center position, size and attitude angle of each sub-target are calculated respectively. The output of the identification results for each material package includes: recalculating the rotating display frame based on the contour of the fusion mask, and mapping the rotating display frame back to the original RGB image coordinate system for visualization output.
[0016] A material bag visual recognition and pose determination system based on RGB image and depth image fusion includes: The image acquisition module is used to acquire RGB images, depth images, and ordered point clouds corresponding to the pixels of the depth images of the target scene. A dual-modal detection module is used to perform target detection on the RGB image and the depth image respectively, and output an RGB instance set and a depth instance set; The cross-modal association module is used to map the masks of the RGB instances and the masks of the depth instances to a unified point cloud coordinate space, calculate the cross-union ratio between each RGB instance mask and each depth instance mask, and establish a cross-modal association relationship based on the comparison result of the cross-union ratio and a preset threshold. An instance fusion module, connected to the cross-modal association module, is used to fuse the RGB instance and the depth instance according to the cross-modal association relationship to generate a fused instance and a corresponding fusion mask. The instance fusion module includes an adhesion and segmentation unit, which is used to segment the mask of the depth instance with the mask of the multiple RGB instances when one depth instance corresponds to multiple RGB instances, thereby splitting the depth instance into multiple fused instances. A point cloud extraction module, connected to the instance fusion module, is used to extract the corresponding target point cloud from the ordered point cloud based on the effective pixel positions in the fusion mask. The pose calculation module is connected to the point cloud extraction module and is used to calculate the center position, size information and attitude angle of the material bag based on the target point cloud. The output module is used to output the identification results of each material package.
[0017] The instance fusion module also includes: The mask fusion unit is used to perform a union operation on the mask of the RGB instance and the mask of the depth instance when a depth instance corresponds to an RGB instance, to generate a fused mask; A single-modal retention unit is used to directly retain the depth instance as a fusion instance when the depth instance does not correspond to any RGB instance, and to directly retain the RGB instance as a fusion instance when the RGB instance does not correspond to any depth instance.
[0018] An electronic device includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the method.
[0019] The beneficial effects of this invention are: higher recognition stability. By simultaneously fusing RGB image and depth image information, it takes into account both texture and spatial features. Even if there are missed or false detections in the single-modal detection results, they can be supplemented by another modality, improving the robustness of recognition in complex scenes.
[0020] It can effectively solve the problem of depth overlap. By associating depth instances with multiple RGB instances and using RGB masks to segment depth targets, it can effectively separate adjacent or stacked targets that are stuck together on the depth map, solving the problem of difficulty in distinguishing multiple targets when relying solely on the depth image. Attached Figure Description
[0021] Figure 1 This is a flowchart illustrating the overall process of the method of the present invention.
[0022] Figure 2 This is a schematic diagram of ROI clipping and bounding box calculation for ordered point clouds in this invention.
[0023] Figure 3 This is a schematic diagram of the priority sorting of the bounding box X in this invention. Detailed Implementation
[0024] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.
[0025] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same technical meaning as commonly understood by one of ordinary skill in the art to which this application pertains.
[0026] A method for visual recognition and pose determination of material packages based on the fusion of RGB images and depth images includes the following steps: Step 1: Data Acquisition. Acquire the RGB image, depth image, and ordered point cloud of the target scene. The 3D points in the ordered point cloud correspond one-to-one with the pixel positions in the depth image. An ordered point cloud is one where the points in the point cloud correspond to the pixel positions in the image, allowing the corresponding 3D points to be extracted directly using pixel indexing.
[0027] Step 2: ROI Cropping. Based on preset cropping parameters (CropX, CropY, CropW, CropH), ROI cropping is performed on the RGB image, the depth image, and the ordered point cloud, respectively, to obtain the RGB region to be identified, the depth region to be identified, and the point cloud region to be identified. Specifically, the original RGB image is rectangularly cropped according to the cropping parameters obtained in the calibration stage; simultaneously, based on the resolution ratio of the RGB image to the depth image, the cropped region is mapped to the depth map and point cloud space to obtain the corresponding depth ROI and point cloud ROI. ROI cropping can reduce interference from irrelevant background areas, improving inference efficiency and recognition accuracy.
[0028] Step 3: Perform object detection on the RGB image and the depth image respectively. Input the depth region to be identified into the depth instance segmentation model to obtain a depth instance set; input the RGB region to be identified into the RGB instance segmentation model to obtain an RGB instance set.
[0029] In this embodiment, both the "deep instance segmentation model" and the "RGB instance segmentation model" use the existing YOLOv8 instance segmentation model (YOLOv8-seg) as the base network. They are not reused from the same model; instead, they are trained independently for depth map samples and RGB map samples, with weights derived separately and inference performed separately. YOLOv8-seg includes a backbone network for extracting multi-scale features, a neck network for fusing features at different scales, and object detection and instance mask branches. The object detection branch outputs the category, confidence score, and bounding box, while the instance mask branch outputs the segmentation mask corresponding to each package instance.
[0030] The RGB instance segmentation model takes a three-channel color image of the RGB region to be identified as input, and RGB images labeled with the outline and category of the package instance as training samples. The depth instance segmentation model takes a depth region to be identified as input, first converting the single-channel depth image into a three-channel grayscale image that the model can accept, and optionally normalizing the effective depth values. The training samples are depth images labeled with the outline and category of the package instance. Both training sets are subjected to size scaling, data augmentation, and instance annotation format conversion, respectively, to optimize the bounding box regression loss, classification loss, and mask segmentation loss. After training convergence, each set obtains its optimal weight.
[0031] During deployment, the two sets of optimal weights are converted to ONNX format and loaded by two independent YOLOv8-seg inferencers. In the software implementation corresponding to this embodiment, the RGB model file is best_rgb.onnx, and the depth model file is best.onnx. The model names are only used to describe one possible implementation and do not constitute a limitation on the model file name, network size, or training framework.
[0032] Each output instance of both models includes at least a class identifier, confidence score, 2D bounding box, and instance mask. The system first filters the two outputs based on the same confidence threshold, and then scales the retained masks from their respective image coordinate systems to an ordered point cloud coordinate system for subsequent cross-modal association and 3D point extraction.
[0033] Optionally, before performing depth instance segmentation, the single-channel depth image of the depth region to be identified is normalized and expanded into a multi-channel image before being input into the depth instance segmentation model.
[0034] Optionally, a depth-first recognition strategy can be adopted: first, the depth map is converted into a multi-channel image format suitable for model input and input into the depth instance segmentation model to obtain depth detection results; the depth detection results are filtered according to a confidence threshold to form a first candidate instance set; when the first candidate instance set reaches a set number threshold, 3D target extraction is first performed based on the depth results and point cloud; if a sufficient number of valid targets can be obtained directly, the recognition results are directly output; when the number of depth results is sufficient but the number of valid 3D geometric targets is insufficient, or when the depth results are insufficient, the RGB instance segmentation model is further called for detection.
[0035] Step 4: Establish cross-modal association. Map each RGB instance mask in the RGB instance set and each depth instance mask in the depth instance set to a unified point cloud coordinate space, and calculate the intersection-over-union (IoU) ratio between each RGB instance mask and each depth instance mask. Establish cross-modal association between RGB instances and depth instances based on the IoU ratio.
[0036] Optionally, the mapping includes: scaling the RGB instance mask and the depth instance mask to the same target size according to the ratio between their original image size and point cloud size, and then performing intersection-union calculation.
[0037] Preferably, the matching rule is as follows: if the IoU is greater than a preset threshold, the RGB instance is considered to match the depth instance; if the IoU is less than or equal to the preset threshold, they are considered not to match.
[0038] Step 5: Instance Fusion. Based on the cross-modal association, the RGB instance and the depth instance are fused to generate a fused instance and a corresponding fusion mask.
[0039] Specifically, the fusion strategy includes the following four scenarios: (1) When a depth instance is associated with multiple RGB instances (i.e., a depth instance matches multiple RGB instances), the depth instance is determined to be an adhered instance. The mask of the depth instance is segmented using the mask of the multiple RGB instances, and the depth instance is split into multiple fused instances, thereby achieving the separation of adhered targets. (2) When a depth instance is associated with an RGB instance (i.e., a depth instance matches an RGB instance), perform a union operation on the RGB instance mask and the depth instance mask to generate a fused mask; (3) When a depth instance is not associated with any RGB instance (i.e., the depth instance does not match any RGB instance), the depth instance is directly retained as the fusion instance; (4) When an RGB instance is not associated with any depth instance (i.e., the RGB instance is not matched with a depth instance), the RGB instance is directly retained as the fusion instance.
[0040] Step Six: Extract the target point cloud. Based on the fusion mask, extract the target point cloud corresponding to the fusion mask from the ordered point cloud.
[0041] Optionally, the extraction includes: traversing the fusion mask pixel by pixel, and for each valid pixel position in the fusion mask, directly extracting the corresponding 3D point from the ordered point cloud through the pixel index, and removing non-finite points (such as infinite or non-numerical points) and zero-depth points.
[0042] Step 7: Calculate 3D spatial information. Perform 3D geometric calculations on the target point cloud to obtain the center position, size information, and attitude angle of the material package.
[0043] Optionally, the calculation of the attitude angle includes: extracting the Z-value set of the target point cloud, calculating the median of the Z-values as the target height; projecting the target point cloud onto a two-dimensional plane, calculating the minimum bounding rotation rectangle for the two-dimensional point set that satisfies the height bandwidth constraint, obtaining the length, width, center point and angle information of the target based on the minimum bounding rotation rectangle, and performing normalization processing on the angle information to map it to a preset grasping angle range.
[0044] Let the target point cloud be P={p i =(x i ,y i ,z i First, take the set of Z coordinates of all valid points Z = {z | i=1,2,…,N}. i}, with its median z m =median(Z) serves as a robust estimate of the height of the material package surface; the height coordinate of the target center is taken as z. c =z m .
[0045] Select a point set P′={p} within the height bandwidth Δz. i ∈P | |z i -z m |<Δz}, and projected onto the XY plane to obtain Q={(x i ,y i ) | p i ∈P′}. In this embodiment, Δz can be 100 mm, or it can be set according to the thickness of the material bag and the depth noise.
[0046] Calculate the minimum circumscribed rectangle of a two-dimensional point set Q, R = minAreaRect(Q). Let the center of R be (c... x ,cy If the lengths of the two sides are s1 and s2, and the rotation angle is θ, then the center of the bag is C = (c x ,c y ,z m The width of the material bag is w=min(s1,s2), the length of the material bag is l=max(s1,s2), and the original attitude angle is θ.
[0047] Based on the conventions of the long side direction and the robot's grasping angle, θ is equivalently reversed and normalized within an interval to obtain the grasping posture angle r. z , make r z It falls within a preset range, such as [-90°, 90°]. When using coordinate calibration transformation, it can be further calculated as p′=R. c ·p+t c Transform the center point and boundary points in the camera coordinate system to the robot or workspace coordinate system, where R c To calibrate the rotation matrix, t c This is for calibrating the translation vector.
[0048] Optionally, it also includes verifying the target size: when the target length meets the preset double-package criterion, the target is split into two sub-targets.
[0049] Step 8: Sorting and Output. Sort the identified material packages according to a preset sorting rule and output the identification results for each package.
[0050] Optionally, the preset sorting rules include at least one of the following: sorting with priority in the Z direction; sorting with priority in the X direction and secondary priority in the Y direction; sorting with priority in the Y direction and secondary priority in the X direction; and each sorting direction supports both ascending and descending order configurations. This can meet the actual requirements of robot grasping path planning or stacking and unloading rules.
[0051] Optionally, the recognition results for each output package include: recalculating the rotated display frame based on the fused mask contour, and mapping the rotated display frame back to the original RGB image coordinate system for display output; the recognition results include target position, pose angle, size information, and display frame information. Therefore, the display frame is strictly consistent with the final output target, rather than simply using the original detection frame of a certain model.
[0052] A material bag visual recognition and pose determination system based on RGB image and depth image fusion includes: The image acquisition module is used to acquire RGB images, depth images, and ordered point clouds of the target scene, wherein the three-dimensional points in the ordered point cloud correspond one-to-one with the pixel positions of the depth image; The ROI cropping module is connected to the image acquisition module and is used to perform ROI cropping on the RGB image, the depth image and the ordered point cloud according to preset cropping parameters, and output the RGB region to be identified, the depth region to be identified and the point cloud region to be identified. A dual-modal detection module, connected to the ROI cropping module, includes a depth instance segmentation unit and an RGB instance segmentation unit. The depth instance segmentation unit is used to perform target detection on the depth region to be identified and output a depth instance set. The RGB instance segmentation unit is used to perform target detection on the RGB region to be identified and output an RGB instance set. The instance fusion module, connected to the dual-modal detection module, is used to map the RGB instance set and the depth instance set to a unified point cloud coordinate space, establish cross-modal association relationship through mask intersection-union ratio, and fuse RGB instances and depth instances according to the association relationship to generate fused instances and corresponding fused masks. A point cloud extraction module, connected to the instance fusion module, is used to extract the corresponding target point cloud from the ordered point cloud according to the fusion mask; The pose solving module is connected to the point cloud extraction module and is used to perform three-dimensional geometric calculations on the target point cloud to obtain the center position, size information and attitude angle of the material bag. The sorting output module, connected to the pose solving module, is used to sort multiple material packages according to a preset sorting rule and output the recognition results of each material package.
[0053] An electronic device includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, implements the method described above. Example
[0054] This embodiment provides a method for visual recognition and pose determination of material packages based on the fusion of RGB images and depth images.
[0055] I. Data Acquisition The system first acquires RGB images, depth images, and ordered point clouds corresponding to depth pixels using a depth camera. The ordered point cloud refers to a point cloud where the positions of points correspond to the pixel positions in the image, allowing direct extraction of the corresponding 3D points via pixel indexing.
[0056] II. ROI Cutting Based on the cropping parameters CropX, CropY, CropW, and CropH obtained during the calibration phase, the original RGB image is rectangularly cropped. Simultaneously, according to the resolution ratio between the RGB image and the depth image, the cropped region is mapped to the depth map and point cloud space to obtain the corresponding depth ROI and point cloud ROI. ROI cropping can reduce interference from irrelevant background areas, improving inference efficiency and recognition accuracy.
[0057] III. Depth Map Priority Recognition The depth map is converted into an image format suitable for model input. In this embodiment, the single-channel depth image is first normalized and then expanded into a three-channel image for input into the depth instance segmentation model to obtain depth detection results. The depth detection results are then filtered according to a confidence threshold to form a first candidate instance set.
[0058] This embodiment employs two separately trained YOLOv8-seg models: the depth model is trained using only depth map samples, and the RGB model is trained using only RGB image samples. Before inference, the depth map is expanded from a single channel to a three-channel grayscale input; the RGB image is directly input as a three-channel color ROI. The two models output the confidence score, bounding box, and mask for each package instance. During runtime, the depth model is executed first; the RGB model is only called if the depth result does not meet the requirement for the number of valid targets, and the results from both models are used for subsequent mask fusion.
[0059] When the first candidate instance set reaches the set threshold, 3D target extraction is first performed based on the depth results and point cloud; if a sufficient number of valid targets can be obtained directly, the recognition results are directly output to improve processing efficiency.
[0060] IV. RGB Fusion When there are sufficient depth results but insufficient effective 3D geometric targets, or when there are insufficient depth results, the RGB instance segmentation model is further invoked to detect RGB ROIs and obtain a second set of candidate instances.
[0061] Then, the RGB instances and depth instances are uniformly mapped to the point cloud coordinate space. Specifically, the RGB instance mask and the depth instance mask are scaled to the same target size according to the ratio between their original image size and the point cloud size, and the intersection-over-union ratio (IoU) between the corresponding masks is calculated to establish a cross-modal matching relationship.
[0062] The matching rules are as follows: If the IoU is greater than a preset threshold (e.g., 0.5), then the RGB instance is considered to match the depth instance; If a depth instance matches multiple RGB instances, the depth instance is determined to be a stuck instance. The depth instance is then segmented using multiple RGB instance masks, splitting the depth instance into multiple merged instances. If a depth instance matches an RGB instance, then perform a union fusion on the RGB mask and the depth mask; If no RGB instance is matched with the depth instance, the depth instance is retained; If no depth instance is matched for the RGB instance, the RGB instance is retained.
[0063] V. Point Cloud Extraction and Target Reconstruction The fused global mask is traversed pixel by pixel. For each valid pixel position in the mask, the corresponding 3D point is directly extracted from the ordered point cloud through the pixel index, and non-finite points and zero-depth points are removed to form the target point cloud.
[0064] VI. Position and Pose Determination For each target point cloud, extract its Z-value set and calculate the median as the target height; calculate the minimum bounding rectangle for the two-dimensional point set that satisfies the height bandwidth constraint to obtain the target's length, width, center point, and angle information.
[0065] Specifically, the Z-coordinates of all 3D points in the target point cloud are extracted, sorted, and the median is used as the target's height estimate. Then, the target point cloud is projected onto a 2D plane (XY plane), retaining only points whose Z-coordinates fall within the height bandwidth constraint. The minimum bounding rectangle of this retained 2D point set is calculated; the center of this rectangle represents the target's center in the XY plane. The rectangle's length and width correspond to the target's dimensions, and its rotation angle corresponds to the target's attitude angle. Finally, the angle information is normalized and mapped to a preset grasping angle range (e.g., -90° to 90°) to obtain the final attitude angle.
[0066] VII. Result Sorting Multiple valid targets are sorted according to preset rules. These rules include Z-priority sorting, X-priority sorting, and Y-priority sorting, with support for ascending / descending order in the X and Y directions. For example, in X-priority sorting, targets are first sorted by their X-coordinates from smallest to largest (or largest to smallest), and then sorted by their Y-coordinates if the X-coordinates are the same. This satisfies the practical requirements of robot grasping path planning or stacking and unloading rules.
[0067] 8. Display Frame Generation For the final valid target, the rotated display box is recalculated based on the fused mask contour and mapped back to the original RGB image coordinate system for display. Therefore, the display box is strictly consistent with the final output target, rather than simply using the original detection box of a certain model.
[0068] This embodiment provides a material bag visual recognition and pose determination system based on the fusion of RGB images and depth images, including: The image acquisition module is used to acquire RGB images, depth images, and ordered point clouds of the target scene, wherein the three-dimensional points in the ordered point cloud correspond one-to-one with the pixel positions of the depth image.
[0069] The ROI cropping module, connected to the image acquisition module, is used to perform ROI cropping on the RGB image, the depth image, and the ordered point cloud according to preset cropping parameters, and output the RGB region to be identified, the depth region to be identified, and the point cloud region to be identified.
[0070] A dual-modal detection module, connected to the ROI clipping module, includes a depth instance segmentation unit and an RGB instance segmentation unit. The depth instance segmentation unit is used to perform target detection on the depth region to be identified and output a depth instance set; the RGB instance segmentation unit is used to perform target detection on the RGB region to be identified and output an RGB instance set.
[0071] An instance fusion module, connected to the dual-modal detection module, is used to map the RGB instance set and the depth instance set to a unified point cloud coordinate space, establish cross-modal association relationships through mask intersection-union ratio (CIU), and fuse RGB instances and depth instances according to the association relationships to generate fused instances and corresponding fused masks. The instance fusion module includes: an association establishment unit for calculating the mask CIU to establish cross-modal association relationships; an adhesion segmentation unit for segmenting the depth instance mask using multiple RGB instance masks when a depth instance is associated with multiple RGB instances; a mask fusion unit for performing a union operation on the RGB mask and the depth mask when a depth instance is associated with one RGB instance; and a single-modal retention unit for directly retaining an instance when no match is found.
[0072] The point cloud extraction module is connected to the instance fusion module and is used to extract the corresponding target point cloud from the ordered point cloud according to the fusion mask.
[0073] The pose calculation module, connected to the point cloud extraction module, is used to perform three-dimensional geometric calculations on the target point cloud to obtain the center position, size information, and attitude angle of the material package. The pose calculation module includes: a height calculation unit, used to extract the median of the Z-value set of the target point cloud as the height; a rotation rectangle calculation unit, used to calculate the minimum bounding rectangle to obtain the length, width, center point, and angle; and an angle normalization unit, used to map the angle to a preset grasping angle range.
[0074] The sorting output module, connected to the pose solving module, is used to sort multiple material packages according to a preset sorting rule and output the recognition results of each material package. The sorting output module includes: a sorting unit for sorting according to Z-priority, X-priority, or Y-priority rules; a display frame generation unit for recalculating and rotating the display frame based on the fusion mask contour and mapping it back to the original RGB image coordinate system; and an output unit for outputting the target position, attitude angle, size information, and display frame information of each material package.
[0075] The present invention has the following advantages: 1. Higher recognition stability. By simultaneously fusing RGB image and depth image information, it takes into account both texture and spatial features. Even if there are missed or false detections in the single-modal detection results, they can be completed by another modality, thereby improving the robustness of recognition in complex scenes.
[0076] 2. Effectively solves the problem of depth overlap. By associating depth instances with multiple RGB instances and using RGB masks to segment depth targets, adjacent or stacked targets that are stuck together on the depth map can be effectively separated, solving the problem of difficulty in distinguishing multiple targets stuck together when relying solely on the depth image.
[0077] 3. It can directly output 3D grasping information. By fusing masks, it can directly extract the target point cloud from the ordered point cloud and calculate the 3D center, size and attitude angle, without the need for complicated coordinate transformation post-processing, which is convenient for robots to grasp directly.
[0078] 4. The displayed results are consistent with the fusion results. The display frame no longer uses a simple single-path detection frame, but generates a rotating frame based on the fused target mask, so that the interface display is consistent with the final capture result, avoiding the operational problems caused by inconsistencies between display and output.
[0079] 5. Flexible sorting. It can be sorted according to Z-priority, X-priority, or Y-priority, and supports ascending and descending order configuration, making it more suitable for customized applications in industrial production lines and meeting different grasping path planning needs.
[0080] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several changes and improvements without departing from the overall concept of the present invention, and these should also be considered within the scope of protection of the present invention.
Claims
1. A method for visual recognition and pose determination of material packages based on the fusion of RGB images and depth images, characterized in that, include: Acquire the RGB image, depth image, and ordered point cloud corresponding to the pixels of the depth image of the target scene; Target detection is performed on the RGB image and the depth image respectively to obtain an RGB instance set and a depth instance set. The RGB instance set contains the mask of each RGB instance, and the depth instance set contains the mask of each depth instance. The masks of the RGB instances and the masks of the depth instances are mapped to a unified point cloud coordinate space, and the cross-union ratio between each RGB instance mask and each depth instance mask is calculated. Based on the comparison result of the cross-union ratio and a preset threshold, a cross-modal association relationship between the RGB instances and the depth instances is established. Based on the cross-modal association, the RGB instance and the depth instance are fused to generate at least one fused instance and a corresponding fused mask; Specifically, when it is determined that a depth instance corresponds to multiple RGB instances based on the cross-modal association relationship, the mask of the depth instance is segmented using the mask of the multiple RGB instances, thereby splitting the depth instance into multiple fused instances; Based on the effective pixel positions in the fusion mask, extract the target point cloud corresponding to the fusion mask from the ordered point cloud; The three-dimensional spatial information of the material bag is calculated based on the target point, and the three-dimensional spatial information includes the center position, size information and attitude angle; Output the identification results for each material package.
2. The method for visual recognition and pose determination of material packages based on RGB image and depth image fusion according to claim 1, characterized in that, Also includes: When a depth instance corresponds to an RGB instance, a union operation is performed on the mask of the RGB instance and the mask of the depth instance to generate a fused mask; When a depth instance does not correspond to any RGB instance, the depth instance is directly retained as the fusion instance; When an RGB instance does not correspond to any depth instance, the RGB instance is directly retained as the fusion instance.
3. The method for visual recognition and pose determination of material packages based on RGB image and depth image fusion according to claim 1, characterized in that, The step of mapping the mask of the RGB instance and the mask of the depth instance to a unified point cloud coordinate space includes: scaling the mask of the RGB instance and the mask of the depth instance to the same target size according to the ratio between their original image size and point cloud size.
4. The method for visual recognition and pose determination of material packages based on RGB image and depth image fusion according to claim 1, characterized in that, The effective pixel position is the pixel position in the fusion mask that belongs to the target foreground region and has a corresponding non-zero depth value; The step of extracting the target point cloud from the ordered point cloud based on the effective pixel positions in the fusion mask includes: traversing the fusion mask pixel by pixel, directly extracting the corresponding 3D points from the ordered point cloud for the effective pixel positions in the fusion mask through pixel indexing, and discarding non-finite points.
5. The method for visual recognition and pose determination of material packages based on RGB image and depth image fusion according to claim 1, characterized in that, The step of calculating the attitude angle of the material package based on the target point cloud includes: projecting the target point cloud onto a two-dimensional plane, calculating the minimum bounding rectangle of the two-dimensional projection point set of the target point cloud, and obtaining the attitude angle of the material package based on the minimum bounding rectangle.
6. The method for visual recognition and pose determination of material packages based on RGB image and depth image fusion according to claim 1, characterized in that, Before performing RGB instance segmentation, a depth-first determination step is also included: the depth instance set is filtered according to a confidence threshold. When the number of filtered depth instances reaches a preset threshold, the target point cloud is extracted based on the depth instance and the corresponding ordered point cloud and the three-dimensional spatial information is calculated. If the number of valid targets obtained meets the preset requirements, the recognition result is directly output; otherwise, RGB instance segmentation continues.
7. The method for visual recognition and pose determination of material packages based on RGB image and depth image fusion according to claim 1, characterized in that, The depth image is normalized and expanded into a multi-channel image before being input into the depth detection model; The identified multiple material packages are sorted according to a preset sorting rule, which includes at least one of the following: sorting by Z direction priority, sorting by X direction priority followed by Y direction priority, and sorting by Y direction priority followed by X direction priority. The target size is verified. When the size of the target point cloud along the length direction meets the preset double-packet criterion, the target is split into two independent sub-targets, and the center position, size and attitude angle of each sub-target are calculated respectively. The output of the identification results for each material package includes: recalculating the rotating display frame based on the contour of the fusion mask, and mapping the rotating display frame back to the original RGB image coordinate system for visualization output.
8. A material bag visual recognition and pose determination system based on RGB image and depth image fusion, characterized in that, include: The image acquisition module is used to acquire RGB images, depth images, and ordered point clouds corresponding to the pixels of the depth images of the target scene. A dual-modal detection module is used to perform target detection on the RGB image and the depth image respectively, and output an RGB instance set and a depth instance set; The cross-modal association module is used to map the masks of the RGB instances and the masks of the depth instances to a unified point cloud coordinate space, calculate the cross-union ratio between each RGB instance mask and each depth instance mask, and establish a cross-modal association relationship based on the comparison result of the cross-union ratio and a preset threshold. An instance fusion module, connected to the cross-modal association module, is used to fuse the RGB instance and the depth instance according to the cross-modal association relationship to generate a fused instance and a corresponding fusion mask; The instance fusion module includes an adhesion and segmentation unit, which is used to segment the mask of the depth instance using the mask of the multiple RGB instances when one depth instance corresponds to multiple RGB instances, thereby splitting the depth instance into multiple fusion instances. A point cloud extraction module, connected to the instance fusion module, is used to extract the corresponding target point cloud from the ordered point cloud based on the effective pixel positions in the fusion mask. The pose calculation module is connected to the point cloud extraction module and is used to calculate the center position, size information and attitude angle of the material bag based on the target point cloud. The output module is used to output the identification results of each material package.
9. The material bag visual recognition and pose determination system based on RGB image and depth image fusion according to claim 1, characterized in that, The instance fusion module also includes: The mask fusion unit is used to perform a union operation on the mask of the RGB instance and the mask of the depth instance when a depth instance corresponds to an RGB instance, to generate a fused mask; A single-modal retention unit is used to directly retain the depth instance as a fusion instance when the depth instance does not correspond to any RGB instance, and to directly retain the RGB instance as a fusion instance when the RGB instance does not correspond to any depth instance.
10. An electronic device comprising a processor and a memory, the memory storing a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 7.