Intelligent identification method and system for grabbing scattered parts

Abstract

The invention discloses an intelligent recognition method and system for grabbing scattered parts, and relates to the technical field of disordered grabbing. In order to solve the problems of high data labeling cost, low occlusion scene pose precision, difficulty in cross-domain generalization and the like in the prior art, a full-process closed loop of'simulation data synthesis-multi-modal segmentation-pose estimation-semi-supervised training 'is constructed: a field adaptive simulation data set is generated through PyBullet + PyRender, and a MaskNet algorithm is obtained based on ViT improved YOLO11 to realize multi-modal instance segmentation; a MaskPosenet two-stage pose estimation algorithm is designed, and cross-domain generalization is realized in combination with pseudo-label semi-supervised learning of a cosine scheduling threshold. According to the method, the data cost is greatly reduced, the complex scene capturing precision and generalization ability are improved, and the method is adaptive to industrial scenes such as logistics sorting and flexible manufacturing and has extremely high application value.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot disordered grabbing, and particularly relates to an intelligent identification method and system for grabbing scattered parts. BACKGROUND

[0002] With the deep promotion of industrial automation and intelligent manufacturing, robot disordered grabbing technology has become the key to improving production efficiency and reducing labor costs. However, the existing disordered grabbing technology still faces many bottlenecks: On the one hand, deep learning algorithms rely on large-scale high-quality labeled data, but part labeling in real scenarios is time-consuming, labor-intensive and costly. In addition, the shapes of parts in complex industrial environments are diverse and randomly stacked, and some parts have high reflectivity and weak texture characteristics, further increasing the difficulty of data acquisition. On the other hand, existing instance segmentation networks mostly use traditional CNN architectures, which have poor adaptability to stacked and occluded scenes. Moreover, they are disconnected from pose estimation networks, losing deep features and resulting in insufficient pose estimation accuracy. At the same time, the fusion of RGB and depth modal information is inefficient, and pure vision or single-modal algorithms have weak robustness in complex scenes and lack effective cross-domain generalization solutions. The differences between simulation data and real data are large, and the practicality of the model is limited.

[0003] Therefore, there is an urgent need for an intelligent identification algorithm with low cost, high precision and strong generalization to support robots to efficiently complete disordered grabbing of scattered parts. SUMMARY

[0004] To solve the problems of long time consumption and high cost in obtaining large-scale, high-quality and diversified labeled data in the prior art disordered grabbing algorithm, pose estimation difficulty in the presence of overlapping and mutual occlusion in disordered stacking scenes, and inability to efficiently fuse color and depth modal information, the application provides a disordered grabbing intelligent identification algorithm and system.

[0005] In a first aspect, the application provides an intelligent identification method for grabbing scattered parts, which comprises the following steps: S1, domain adaptive simulation data synthesis: a physical simulation environment is constructed by PyBullet to simulate part stacking to a static pose, combined with PyRender to render light and texture, object geometry, texture and light randomization are introduced for enhancement, and a dataset containing depth images, instance segmentation masks and 6DoF pose labels is generated; S2, multi-modal instance segmentation: a MaskNet algorithm based on improved YOLO11 of ViT is constructed, depth data is encoded into XYZ feature maps and fused with RGB features, feature expression is optimized through URD, AFF and DAN modules, and part instance segmentation masks are output; S3 and 6DoF Pose Estimation: The MaskPoseNet algorithm is constructed, repeatedly using the segmented semantic information from step S2. A two-stage strategy of coarse and fine matching is employed, combined with learnable background labeling and weighted SVD, to output the 6DoF pose of the part; and S4. Pseudo-label semi-supervised training: The model is pre-trained using the dataset from step S2. Pseudo-labels are generated for real unlabeled data. Confidence scores are calculated through local point cloud matching. The confidence threshold is dynamically adjusted using a cosine scheduling strategy. The two types of data are fused to train the model to achieve cross-domain generalization.

[0006] Preferably, in step S1, PyBullet simulates the free-fall process of the parts, and after all the parts are stable, PyRender generates color image, depth image and mask image data.

[0007] Preferably, when generating data annotations in step S1, if the area of ​​pixels with inconsistent depths within the mask outline is greater than one-tenth of the total mask area, then polygonal mask annotations are generated.

[0008] Preferably, the depth of the ViT layer in MaskNet is set to n = 20. After discarding the 0th dimension clstoken of ViT, the URD module reshapes the feature vector into a feature map through linear transformation, slicing and downsampling.

[0009] Preferably, the AFF module learns modal local dependencies through local attention and captures scene semantics through global attention to generate a fusion weight map, which weights and fuses RGB features and depth features.

[0010] Further preferred options include: In the feature concatenation stage, the RGB features and depth features are first concatenated along the channel dimension to obtain a feature containing bimodal information. Then, the features flow into two attention modules simultaneously. In the local attention module, the local differences of the concatenated features are learned. In the global attention module, global average pooling is performed on the features to obtain global statistical information. Several 1x1 convolutions are used to interact with the BatchNorm and ReLU modules to capture more macroscopic scene-level semantics. Finally, the local attention results are added to the global attention results to obtain a comprehensive attention map; and then, through the Sigmoid function, This is mapped to the [0, 1] interval to generate weights, which are used for weighted fusion of RGB features and depth features.

[0011] Preferably, the DAN module computes spatial attention and channel attention in parallel, and then adds them to the original features after weighting by learnable coefficients to enhance the feature representation capability.

[0012] Preferably, the coarse matching stage introduces learnable background labels, inputs sparse color features and geometric features into the Geometric Transformer, and solves the initial pose through singular value decomposition; The fine matching stage transforms the target point cloud according to the coarse matching posture, superimposes sinusoidal position coding and dense color features, and after fusing features through Sparse to Dense Transformer, optimizes the posture accuracy through weighted SVD.

[0013] Preferably, the confidence score is obtained by multiplying the correct matching ratio by the proportion of foreground points, and the cosine scheduling threshold is calculated according to the formula. Dynamic adjustment, where c thresh Here, c is the confidence threshold, and c is the confidence score. min and c max These are the initial and final confidence score thresholds, respectively; t represents the current training iteration round, and T is the total number of training iterations.

[0014] In a second aspect, embodiments of the present invention provide an intelligent identification system for grasping scattered parts. This system is used to implement the intelligent identification method as described in any one of the first aspects, including: The simulation data generation module is used to perform the domain-adaptive simulation data synthesis step and generate a labeled simulation dataset; The multimodal segmentation module is used to deploy the MaskNet algorithm, implement multimodal instance segmentation, and output masks. The pose estimation module is used to deploy the MaskPosenet algorithm and reuse mask semantic information to output 6DoF pose. The semi-supervised training module is used to perform pseudo-label semi-supervised training steps to optimize the model's generalization performance.

[0015] Compared with the prior art, the beneficial results of the present invention are as follows: (1) Accurate depth images, instance segmentation and 6DoF pose annotation data were obtained, reducing the time and cost of data acquisition: PyBullet's physical simulation capability was used to generate realistic final static poses of objects, and then PyRender's rendering advantages were combined to simulate multi-dimensional disturbances such as lighting and materials, so as to obtain accurate depth images, instance segmentation and 6DoF pose annotations, reducing the time and cost of data acquisition.

[0016] (2) Domain-based adaptive technology reduces the difference between simulation data and real data, ensuring the credibility of the data: By introducing domain-based adaptive technology, the distribution difference between simulation data and real data is greatly reduced by randomizing and diversifying factors such as object geometry, texture, and lighting, so that simulation data is no different from real data and can be used for real grasping needs.

[0017] (3) Based on Vision Transformer, the algorithm of YOLO11 is improved, which enhances the model’s performance in recognition and segmentation: The algorithm introduces the ViT backbone network to enhance feature extraction capabilities by encoding and fusing deep modal and RGB modal, and combines multimodal attention fusion and spatial and channel dual attention mechanisms, which significantly improves the model’s performance in recognition and segmentation.

[0018] (4) The MaskPosenet pose estimation algorithm improves the accuracy of pose estimation: by combining the RGB-D data of the target object with a high-precision mask, the semantic information generated in the segmentation stage is directly reused in the pose estimation stage. With the help of learnable background labels and multiple pose assumptions, the initial pose is established in the coarse matching stage. Then, in the fine matching stage, the interaction and fusion of sparse and dense features are achieved by combining modules such as position encoding, Geometric Transformer and linear cross attention. The weighted singular value decomposition algorithm is used to obtain a more accurate pose estimation.

[0019] (5) The pseudo-label semi-supervised learning method improves the accuracy and stability of the model: By combining high-quality labeled information in simulation data with pseudo-supervised information in real data, the model not only achieves a smooth transfer from simulation to real scene, but also achieves significant improvement in accuracy and stability.

[0020] In summary, this invention significantly improves upon existing technologies in terms of data generation efficiency and economy, model stability and accuracy, and algorithm precision and effectiveness, thus meeting the requirements of "high reliability and high stability" for disordered data crawling. Attached Figure Description

[0021] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0022] Figure 1This is a flowchart illustrating an intelligent identification method for picking up scattered parts according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the simulated physical environment according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating the simulation data generation process according to an embodiment of the present invention; Figure 4 This is a flowchart illustrating the annotation information generation process according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the Masknet network structure in an embodiment of the present invention; Figure 6 This is a schematic diagram of the URD module structure in an embodiment of the present invention; Figure 7 This is a schematic diagram of the AFF module structure in an embodiment of the present invention; Figure 8 This is a schematic diagram of the DAN spatial attention module in an embodiment of the present invention; Figure 9 This is a schematic diagram of the DAN channel attention module in an embodiment of the present invention; Figure 10 This is a schematic diagram of the MaskPosenet network structure in an embodiment of the present invention; Figure 11 This is a schematic diagram illustrating the flow of input data in a network according to an embodiment of the present invention. Figure 12 This is a schematic diagram of the coarse matching module structure in an embodiment of the present invention; Figure 13 This is a schematic diagram of the fine matching module structure in an embodiment of the present invention; Figure 14 This is a schematic diagram of the entire semi-supervised learning process in an embodiment of the present invention; Figure 15 This is a schematic diagram of the architecture of an intelligent recognition system for picking up scattered parts according to an embodiment of the present invention. Detailed Implementation

[0023] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] Currently, typical unordered crawling algorithms in similar technologies mainly include the following forms: 1. A Robotic Unordered Grasping Method Based on Point Cloud Reconstruction Working principle: Based on the structured light images captured by the structured light camera, stereo matching is performed to obtain the pixels of the matched structured light image. The depth information is restored based on the matched pixels. The PCL library is used to reconstruct a 3D point cloud model from the matched pixels. Then, the obtained 3D point cloud model is used to create a planar grasping network to realize the robot's unordered grasping.

[0026] Advantages: Effectively improves the accuracy of the robot's disordered grasping of parts and reduces the interference of ambient light on the camera's acquisition of part information.

[0027] Disadvantages: It cannot effectively solve the problem of overlapping and mutual occlusion of workpieces in disordered stacking scenarios; 2. An unordered grasping system based on the Mask R-CNN model and ICP algorithm Working principle: The system acquires RGB and depth images of stacked workpieces using a depth camera and generates point cloud data. The Mask R-CNN model is used to segment the RGB and depth images into instances. The workpiece pose is estimated by combining the ICP algorithm with template point cloud registration, thus completing the unordered grasping process.

[0028] Advantages: Enables instance segmentation and pose estimation of stacked workpieces, thereby effectively completing unordered grasping tasks.

[0029] Disadvantages: Data acquisition is difficult, costly, time-consuming, and pose estimation accuracy is hard to guarantee.

[0030] Based on this, the present invention addresses the shortcomings of existing technologies in disordered grasping algorithms, such as the time-consuming and costly acquisition of large-scale, high-quality, and diverse labeled data, the difficulty in pose estimation under overlapping and mutual occlusion of workpieces in disordered stacked scenes, and the inability to efficiently fuse color and depth modal information. It provides an intelligent recognition algorithm and system for disordered grasping, the core of which is: 1. Utilize PyBule's physical simulation capabilities to generate realistic final static poses of objects, and then combine PyRender's rendering advantages to simulate multi-dimensional perturbations such as lighting and materials to obtain accurate depth images, instance segmentation, and 6DoF pose annotations; on this basis, introduce domain adaptation, and significantly reduce the distribution difference between simulation data and real data by enhancing the randomization and diversification of object geometry, texture, lighting, and other factors.

[0031] 2. MaskNet, an improved YOLO11 multimodal instance segmentation algorithm based on VIT, is an improved version of the algorithm. This algorithm integrates the encoding of depth mode and RGB mode, introduces the ViT backbone network to enhance feature extraction capabilities, and combines multimodal attention fusion and spatial and channel dual attention mechanisms to achieve more accurate and stable recognition and segmentation results in complex scenes, laying a good foundation for subsequent pose estimation and precise grasping.

[0032] 3. The 6DoF pose estimation algorithm—MaskPosenet—combines the RGB-D data of the target object with a high-precision mask and directly reuses the semantic information generated in the Masknet segmentation stage for pose estimation. This allows the algorithm to maintain high accuracy even in scenes with high occlusion or unclear textures, while saving network computational overhead. Furthermore, leveraging learnable background markers and multiple pose assumptions, it first establishes the initial pose in the coarse matching stage, and then uses a fine matching stage combining positional encoding, GeometricTransformer, and linear cross-attention modules to achieve interactive fusion of sparse and dense features. A weighted singular value decomposition algorithm is then used to obtain a more accurate pose estimate.

[0033] 4. Utilizing the fundamental idea of ​​combining supervised loss and pseudo-label loss. For pose estimation tasks, a scheme combining pseudo-labeling and self-training strategies is proposed. This involves using a pre-trained model to predict unlabeled real data and constructing confidence scores through local point cloud matching to effectively filter the prediction results. A cosine scheduling strategy is employed to dynamically adjust the pseudo-label confidence threshold, ensuring the introduction of high-quality pseudo-supervision information throughout the training process.

[0034] Through the above-mentioned technical means, the embodiments of the present invention can provide solid data support for 6D pose estimation and disordered grasping tasks of robots in real environments, obtain more accurate and stable recognition and segmentation effects in complex scenes, and enable pose estimation to have strong accuracy and generalization, effectively supporting disordered grasping tasks.

[0035] To achieve the above objectives, the unordered grasping intelligent recognition algorithm and grasping system provided in this embodiment of the invention mainly include the following functional modules and technical steps, such as... Figures 1-15 As shown below, a detailed explanation will follow.

[0036] In a first aspect, embodiments of the present invention disclose an intelligent recognition method for grasping scattered parts, such as... Figure 1 As shown, the method includes the following steps: S1. Domain Adaptive Simulation Data Synthesis: A physical simulation environment is built using PyBullet to simulate the stacking of parts to a static pose. PyRender is used to render lighting and textures, and randomization enhancements are introduced for object geometry, texture, and lighting to generate a dataset containing depth images, instance segmentation masks, and 6DoF pose annotations. Specifically, in this step, PyBullet simulates the free-fall process of the parts. After all the parts stabilize, PyRender generates color images, depth images, and mask image data. When generating data annotations, if the area of ​​pixels with inconsistent depths within the mask outline is greater than one-tenth of the total mask area, polygonal mask annotations are generated.

[0037] Furthermore, simulated physical environments such as Figure 2 As shown, PyBullet and PyRender are used to perform physical simulation and real-time 3D visualization to create a material pose estimation dataset. By combining PyBullet's accurate physical simulation with PyRender's real-time rendering capabilities, a seamless transition from complex physical simulation to intuitive visualization is achieved on the same platform, greatly simplifying the process of creating pose estimation datasets.

[0038] Simulation data generation includes: based on domain adaptation technology, by introducing random perturbations that conform to the real environment into the simulation environment, training samples that "seem real and are rich in variation" are generated, effectively narrowing the distribution gap between simulation data and real data.

[0039] The method for generating datasets based on domain-adaptive techniques samples training data through two distributions: a task-specific state distribution p(x), which is fully randomized in terms of stacking states, number of objects, object poses, and camera parameters; and an observation distribution p(y|x), used to simulate the working characteristics of the sensors. When sampling a single data point, a 3D model and dynamic simulation are used to sample the state x from p(x). k Furthermore, through the rendering process from p(y) k |x k Sampling generates a synthetic depth image y k .

[0040] The simulation dataset synthesis process based on domain adaptive technology is as follows: Figure 3As shown, the process involves several steps. First, a simulation environment is created using PyBullet, modeling the plane and boxes. Second, several different objects are randomly sampled, and a stacking center is determined through random sampling. Third, an initial pose is randomly set, and the objects are dropped from a selected height. PyBullet is used to dynamically simulate the falling process until all objects stabilize and come to a stop, thus forming the desired disordered stacking scene. Next, camera parameters are randomly sampled to obtain the scene's state information under this camera setting. Finally, all scene state information is imported into PyRender, where lighting and object textures and colors are rendered, generating color images, depth images, and mask image data. This multi-stage simulation and rendering approach effectively compensates for the shortcomings of a single platform in physical simulation or image rendering.

[0041] Data annotation generation includes: visualizing each material individually in a material stacking scenario and generating a corresponding depth map y. i (i=1,2,3,...m). Separate y values ​​one by one. i With y o The comparison process is as follows: If there are pixels of varying depths within the corresponding mask contour, and the area of ​​these pixels is less than one-tenth of the total mask area, then the object is considered occluded, and the generation of its polygon mask annotation is skipped; otherwise, the object is considered unoccluded, and the corresponding polygon mask annotation is generated. The data annotation generation technology roadmap is as follows: Figure 4 As shown. Finally, information is digitally acquired: each image and its corresponding mask are read and written into the scene instance mask annotation file; the rotation and translation matrices of the mask on each image are read and written into the scene pose annotation file; and the camera parameters captured in each image are read and written into the camera parameter annotation file.

[0042] By combining the above simulation data rendering process and data annotation generation process, a dataset of synthetic depth images and color images can be constructed.

[0043] S2. Multimodal instance segmentation: Construct a MaskNet algorithm based on ViT and improved YOLO11, encode depth data into XYZ feature maps and fuse them with RGB features, optimize feature representation through URD, AFF and DAN modules, and output part instance segmentation mask; Specifically, in this step, the depth of the ViT layer in MaskNet is set to n = 20. After discarding the 0th dimension clstoken of ViT, the URD module reshapes the feature vector into a feature map through linear transformation, slicing and downsampling.

[0044] The DAN module performs parallel computation of spatial attention and channel attention, and then adds the weighted values ​​to the original features using learnable coefficients to enhance the feature representation capability.

[0045] The AFF module learns modal local dependencies through local attention and captures scene semantics through global attention to generate a fusion weight map, which weights and fuses RGB features with deep features. Specifically, it includes: In the feature concatenation stage, the RGB features and depth features are first concatenated along the channel dimension to obtain a feature containing bimodal information. Then, the features flow into two attention modules simultaneously. In the local attention module, the local differences of the concatenated features are learned. In the global attention module, global average pooling is performed on the features to obtain global statistical information. Several 1x1 convolutions are used to interact with the BatchNorm and ReLU modules to capture more macroscopic scene-level semantics. Finally, the local attention results are added to the global attention results to obtain a comprehensive attention map; and then, through the Sigmoid function, This is mapped to the [0, 1] interval to generate weights, which are used for weighted fusion of RGB features and depth features.

[0046] Specifically, the MaskNet algorithm structure is as follows: Addressing the challenge of identifying densely packed, mutually occluded objects with weak texture in unordered grasping and recognition tasks, which makes object recognition difficult, an improved YOLO11 algorithm based on ViT is designed to enhance the accuracy of object recognition and segmentation in unordered grasping scenarios. This algorithm is named MaskNet, and its overall structure is as follows: Figure 5 As shown, it mainly includes input encoding, backbone network, neck network and task head.

[0047] MaskNet comprehensively improves upon YOLO11 in terms of backbone network feature extraction, multimodal and multi-scale feature fusion, and the dual spatial and channel attention mechanism. These optimizations work together in the algorithm, enabling MaskNet to better perform recognition and segmentation tasks in unordered grasping scenarios and to work in conjunction with subsequent pose estimation algorithms.

[0048] The network input encoding module includes: introducing deep modal features, converting them into XYZ feature maps through a special encoding method, and then using them as supplementary modal feature inputs to provide additional spatial information support.

[0049] The RGB branch follows traditional encoding methods and maintains a standard three-channel input. Meanwhile, since the pose labels used in the simulation dataset are based on the camera coordinate system, to maintain consistency between features and labels, a 3D point cloud is calculated using Equation 2-1, combining depth data and the camera intrinsic matrix. This elevates the (u, v, d) values ​​of each pixel in the depth image to (x, y, z) based on the camera coordinate system, forming the final X, Y, Z feature maps.

[0050] (2-1); In this encoding method, each pixel in the X and Y feature map channels corresponds to the horizontal coordinate of each pixel in the depth data within the camera reference frame, while each pixel in the Z feature map channel corresponds to the vertical coordinate of each pixel in the depth data within the camera reference frame. These feature maps are then concatenated along the feature dimensions, thus preserving the original height and width information to form a unified depth representation XYZ. The XYZ feature maps have shape consistency with their corresponding RGB feature maps. This encoding method not only effectively preserves the geometric details of the object but also lays a foundation for subsequent feature fusion.

[0051] The URD module includes: replacing the CBS and C3K2 modules in YOLO11 with a VIT module of depth n=20 in the backbone network, which reduces the complexity of the feature extraction network while achieving efficient feature extraction. In addition, the Up-sampleReshapeDown-sample (URD) ​​module is added to flatten the feature vectors output by ViT into feature maps, laying the foundation for subsequent multimodal feature fusion.

[0052] The URD module restores feature sequences from different ViT layers into feature maps of different resolutions, such as... Figure 6 As shown, the URD module first discards the 0th dimension clstoken, then expands the channel dimension to 1024 through a linear transformation layer. Subsequently, it uses slicing operations to reshape the feature sequence into a feature map with shape (H / 2, W / 2, 16). Finally, through a pixel-wise convolution and N 3 x 3 convolutions (the size of N is related to the nth layer of the ViT to which the feature sequence belongs), the spatial resolution is gradually reduced and the number of channels is increased, thereby constructing a complete feature pyramid.

[0053] The AFF module includes: an attention-based feature fusion module AFF added to the end of the neck network. This module works in conjunction with the original Y0L011 neck network to construct a feature pyramid that integrates multimodal and multi-scale information.

[0054] AFF is a multimodal feature fusion module based on an attention mechanism that merges feature pyramids from RGB and deep modalities to form a feature pyramid that integrates texture color features and geometric size features.Figure 7 As shown, AFF uses a "local attention + global attention" approach to generate a fusion weight map, and then uses this weight map to perform weighted fusion of RGB features and depth features.

[0055] The specific steps are as follows: In the feature concatenation stage, the RGB features and depth features are first concatenated along the channel dimension to obtain a feature containing bimodal information.

[0056] Then, the features flow into two attention modules simultaneously. In the local attention module, the local differences of the concatenated features are learned, allowing the network to learn which positions rely more on depth information and which positions rely more on RGB information. In the global attention module, global average pooling is performed on the features to obtain global statistical information. Then, several 1x1 convolutions are used to interact with the BatchNorm and ReLU modules to capture more macroscopic scene-level semantics.

[0057] Finally, the local attention results are added to the global attention results to obtain a comprehensive attention map. Then, using the Sigmoid function (Equation 2-2), this map is mapped to the [0, 1] interval to generate weights for weighted fusion of RGB and depth features. In this way, the weights obtained through training can adaptively adjust the fusion weights of these two modalities at the pixel level, thereby improving the overall feature representation capability.

[0058] (2-2); The DAN module includes three lightweight Dual Atention Network (DAN) modules added to the head of the neck network to capture feature dependencies in both spatial and channel dimensions, thereby enhancing feature representation capabilities. Feature maps at different scales are not only fed into the YOLO11 detector head for bounding box and class prediction, but also generate a series of parameters describing instance masks. The final segmentation mask is generated by superimposing the predictions from the feature maps.

[0059] DAN computes spatial and channel attention in parallel and independently, then fuses the results to finely allocate weights to the output feature map along both spatial and channel dimensions. The DAN spatial attention module is as follows: Figure 8 As shown.

[0060] First, the original feature map A is processed using BatchNorm normalization and ReLU activation to obtain three feature maps B, C, and D with the same shape. Next, feature maps B and C are unfolded spatially, with feature map B being transposed to obtain shapes R. (HxW)xB And one for R Bx(HxW)The feature matrix is ​​obtained by multiplying the two matrices. This yields a matrix describing the similarity between spatial locations in the feature map. Subsequently, a spatial attention map S∈R is generated through Softmax normalization. (HxW)x(HxW) .

[0061] Next, the attention map is multiplied by the feature map D, and the result is restored to the original spatial shape. Finally, it is weighted by a learnable coefficient α (initialized to 0, gradually learning more weights as training progresses), and accumulated pixel by pixel with the original feature map, as shown in Equation 2-3.

[0062] (2-3); DAN channel attention such as Figure 9 As shown, feature map B is first transposed to obtain shape R. Bx(HxW) The feature map is then multiplied with the untransposed feature map C, and after Softmax normalization, the channel attention map S∈R is obtained. CxC , representing the relationship between one channel and another. Finally, the output feature map is calculated using Equation 2-4, also through the learnable coefficient β.

[0063] (2-4); S3 and 6DoF Pose Estimation: The MaskPoseNet algorithm is constructed, repeatedly using the segmented semantic information from step S2. A two-stage strategy of coarse and fine matching is employed, combined with learnable background labeling and weighted SVD, to output the 6DoF pose of the part; and Specifically, in the coarse matching stage, learnable background labels are introduced, and sparse color features and geometric features are input into the Geometric Transformer. The initial pose is solved by singular value decomposition. In the fine matching stage, the target point cloud is transformed according to the coarse matching pose, and sinusoidal position codes and dense color features are superimposed. After the features are fused by the Sparse to Dense Transformer, the pose accuracy is optimized by weighted SVD.

[0064] The confidence score is obtained by multiplying the correct matching ratio by the proportion of foreground points, and the cosine scheduling threshold is calculated using the formula. Dynamic adjustment, where c thresh Here, c is the confidence threshold, and c is the confidence score. min and c max These are the initial and final confidence score thresholds, respectively; t represents the current training iteration round, and T is the total number of training iterations.

[0065] The MaskPoseNet network structure for pose estimation is as follows: Figure 9 As shown.

[0066] To address the challenges of 6DoF pose estimation in unordered scenes, such as occlusion, segmentation errors, and sensor noise, this invention treats it as a partial-to-partial matching problem between the target model point cloud and the object's sampled point cloud. To this end, an end-to-end pose estimation algorithm—MaskPoseNet—is proposed, which can predict the 6D pose of the target object by requiring only an RGB image, a depth image, and the target object's model point cloud as input.

[0067] The MaskPosenet algorithm is a combination of the Masknet and Posenet algorithms. The Posenet algorithm uses the VIT network to extract feature correspondences, introduces a learnable background labeling mechanism, and adopts a two-stage strategy of "coarse matching - fine matching" to effectively improve the stability of pose estimation and obtain more refined pose estimates.

[0068] The core inputs to PoseNet include: the RGB image of the target object, its point cloud and segmentation mask, and two model RGB images, model point clouds, and model masks rendered from different viewpoints. Figure 11 As shown, this illustrates how this data flows and is processed within the network during the training process.

[0069] Building upon this, the training process employs a random sampling point cloud generation strategy to simulate various occlusion scenarios and observation biases, thereby improving the network's stability and accuracy in complex environments. The inference process abandons random sampling, instead processing each of the model images, point clouds, and masks generated from 42 different viewpoints through a pre-trained feature extraction module and performing uniform sampling, retaining 2048 point cloud points. This strategy ensures the density and uniformity of the point cloud, thus improving the final pose estimation accuracy.

[0070] The overall scheme for feature extraction of target and model objects in PoseNet includes two parts: dense color feature extraction and geometric structure feature extraction.

[0071] (1) Dense color feature extraction To maintain consistency with the image feature extraction used in MaskNet, PoseNet also utilizes the ViT network to extract dense color features of objects. Considering that MaskNet has already generated a multi-scale RGB feature pyramid containing P1 to P5, and that feature maps at different resolutions have their own advantages in detecting objects at different scales, to reduce computational complexity, PoseNet selects the best-matching feature map from the P1 to P5 feature pyramids already generated by the MaskNet network as the basis for subsequent feature extraction, based on the bounding box size of the proposed object mask, thus avoiding redundant feature extraction. The specific steps are as follows: 1. Select the feature map that matches the bounding box size of each object mask. The mathematical expression is as follows: (3-1); (3-2); 2. The original RGB image is cropped using a proposed object mask, and the cropped mask RGB image is adjusted to a preset fixed size using bilinear interpolation.

[0072] 3. Perform feature location mapping, extract the RGB features corresponding to the mask from the selected feature map, and then adjust the features of the mask region to a preset specific size through bilinear interpolation.

[0073] 4. Multiply the mask image and the mask RGB feature image pixel by pixel to form the desired dense color features.

[0074] After image feature acquisition, point cloud data also needs to be extracted simultaneously. All point cloud data are first normalized, i.e., scaled according to the maximum radius. The target object point cloud has been downsampled to 2048 points during the input encoding stage. To ensure consistency, the model object's point cloud is also reduced to 2048 points by sampling from the farthest point.

[0075] Next, based on the encoded positions of the point cloud sampling points, the corresponding point set is extracted from the dense color features. This process ensures that, during the RGB image and point cloud feature extraction stages, the algorithm can successfully obtain the dense color features F∈R of the proposed object. NX3 and dense point cloud P∈R NX3 and the dense color features F of the model objects m ∈R NXC and dense point cloud P m ∈R NXC N is the number of dense points, set to 2048, and C is the number of feature channels.

[0076] Finally, for the dense point clouds P and P m Sampling was performed on the farthest point, and then downsampled to 196 points to obtain sparse point clouds SP and SP2. m And by using the encoding location information of the sampling points, sparse color features SF and SF' are extracted from the corresponding dense color features. m .

[0077] (2) Geometric structure feature extraction To capture geometrically embedded features that are invariant to point cloud transformations, and thus more completely represent the 3D shape of objects, a geometric feature extraction module was designed. The main process is as follows: 1. Calculate the distance between all point pairs. This allows each point to describe the overall structural distribution of the point cloud globally through its geometric distances to other points.

[0078] 2. Within a local area of ​​the point cloud, select k nearest neighbor points as references and construct corresponding vector representations for each. Then, analyze the geometric relationships (angles) between these vectors to characterize the configuration features of the local region in the point cloud. Ultimately, each point can not only perceive its distance from other points globally, but also capture fine spatial configuration and orientation information at a local scale.

[0079] 3. To further obtain stable and generalizable geometric features, a sinusoidal function-based encoding method is used to project the aforementioned distance and angle information. This process is similar to positional encoding, enabling geometric relationships at different scales to be mapped into a learnable representation space: (3-3); (3-4); In the sinusoidal function-based positional encoding, d represents the total feature dimension, pos represents the feature position, and i represents the dimension in which the feature is located. Subsequently, by averaging multiple neighborhood relationships of the same point, the diverse local angular information is simplified into a unified angular feature representation.

[0080] 4. The encoding results from global distance and local angle are merged to obtain the complete geometric feature embedding.

[0081] Combine the sparse point clouds SP and SP of the target object and the model object. m The geometric structure embedding features G and G are respectively input into the above algorithm flow to extract the geometric structure embedding features G and G. m .

[0082] In summary, after processing by the feature extraction network, the dense point cloud P of the target object is identified by the dense color feature F, while the sparse point cloud SP is identified by the sparse color feature SF and the geometric embedding feature G. Similarly, the dense point cloud P of the model object is identified by... m The corresponding feature is dense color feature F. m Its sparse point cloud SP m The corresponding feature is the sparse color feature SF. m and geometric embedding features G m .

[0083] Coarse pose matching includes: matching SF and SF m The input is fed into a shared fully connected layer, and after linear transformation, it is combined with the learnable background labels f1 and f2, respectively. mThen stitch them together. Next, stitch the SF and SF with the background markers stitched together. m And G and G m The optimal rigid body transformation is obtained by inputting into a multi-layered Geometric Transformer network and then solving for it using singular value decomposition. Combined with a reprojection error filtering strategy, the optimal coarse-matched pose is obtained, such as... Figure 12 As shown. The specific steps are as follows: 1. Introduce background labeling information. Since the assignment matrix can only represent P and P', m Part-to-part correspondence. To this end, learnable background labels are introduced, denoted as f. bg ∈R C and f m bg ∈R C Where C is the number of feature channels, and a background point P∈R is introduced into the point cloud. 1X3 This simple design solves the problem of assigning non-overlapping points between two point clouds, thus effectively establishing part-to-part correspondences based on feature similarity.

[0084] 2. Extract high-level point features (fusion features H and H' that combine color and geometric structure features). m This approach aims to maintain the internal geometry of the point cloud and the geometric consistency between point clouds. To this end, a Geometric Transformer combining a geometric attention module and a cross-attention module was designed.

[0085] 3. Calculate the point set correspondence between P and Pm. The point set correspondence between P and Pm is represented by a cosine similarity matrix, as shown in Equation 3-5.

[0086] (3-5); Here, temp is a scaling factor used to control numerical stability. The resulting similarity matrix reflects the matching probability between points in the two point clouds and serves as the basis for establishing soft correspondences.

[0087] 4. Obtain the soft allocation matrix. The soft allocation matrix A is obtained using Equation 3-6: (3-6); in, These represent the Softmax operations performed along the rows and columns of the matrix, respectively, where τ is a constant.

[0088] 5. Perform singular value decomposition. The matching pairs {P, P} obtained from the soft-assignment matrix are then processed using the singular value decomposition algorithm. m} Perform calculations to obtain the optimal rotation matrix R and translation vector t. The final rigid body transformation (R, t) can align the source point set to the target point set.

[0089] 6. Select the optimal coarse-matching pose. Multiply the reprojection error of each point pair in the point cloud under the candidate pose by its matching probability and sum them to obtain the weighted error: (3-7); Finally, the candidate poses with the smallest weighted error are selected as the optimal coarse-matched poses (R0). init , t init ).

[0090] In summary, it is possible to obtain a stable and accurate initial rigid body transformation to achieve a relatively accurate initial attitude even in the presence of noise and uncertainty.

[0091] The pose fine matching module aims to further refine the initial pose (R) obtained in the coarse matching stage. init , t init By constructing dense correspondences, more accurate rigid body transformations (Ri) can be solved. finit , t finit ).

[0092] In the fine matching stage, a positional encoder is introduced to transform the target object point cloud according to the coarse matching pose. This transformation is then added to the dense color feature F (Fm for the model point cloud) via the positional encoder to form a fused feature, which is then compared with G and G... m The inputs are fed into a Sparse to Dense Transformer network, and finally, the optimal pose solution is obtained through a weighted singular value decomposition algorithm (R0). finit , t finit ),like Figure 13 As shown. The specific steps are as follows: 1. Transform the target object point cloud according to the coarse-matched pose to obtain the pre-aligned point cloud P', with the following relationship: (3-8); 2. The pre-aligned point cloud P´ obtained through the position encoder is added to the dense color feature F (Fm for the model point cloud) to form a fused feature, as follows: (3-9); Among them, f bg It is a learnable background marker, and PE(·) represents the location encoding module. This module extracts local geometric features through a multi-scale abstraction layer, groups neighborhood points with different radii and sampling numbers, and after passing through a shared multilayer perceptron and max pooling operation, the features at different scales are concatenated and mapped to a specified dimension to finally generate location information encoding.

[0093] 3. Establish correspondences between points while continuously fusing global geometric information from the sparse layer. To address the limited modeling effectiveness of point-to-point exchanges, a sparse-to-dense feature transformer is designed by combining a Geometric Transformer and a linear cross-attention module. The sparse features enhanced by the Geometric Transformer are used as keys and values, while the dense features are used as queries, enabling the diffusion of global geometric information to dense features. Ultimately, after multiple layers of propagation, the dense features are integrated into the contextual information enhanced by the Geometric Transformer. This design provides a more stable and accurate feature foundation for the subsequent fine-matching stage.

[0094] 4. Obtain predicted point cloud pairs. Similar to the coarse matching stage, the fine matching stage still calculates the cosine similarity matrix based on feature similarity to obtain the correspondence between point sets, and thus obtains the soft assignment matrix. However, in order to make full use of the dense soft assignment matrix, all foreground points are directly weighted and fused. Since this weighted summation takes into account the information and matching probability of all target points, it can obtain a more accurate predicted point cloud with better matching relationships.

[0095] 5. Weighted Singular Value Decomposition. To further improve the robustness and accuracy of alignment, fine matching employs weighted singular value decomposition to solve the rigid body transformation. In the weighted singular value decomposition algorithm, the weight w i Its main function is reflected in the calculation of the centroid and covariance matrix: (3-10); (3-11); In summary, the weights focus more on reliable correspondences when calculating rigid transformations, thus improving the accuracy and stability of the alignment results. Finally, the algorithm obtains the optimal pose solution (R0) through a weighted singular value decomposition algorithm. finit , t finit ).

[0096] S4. Pseudo-label semi-supervised training: The model is pre-trained using the dataset from step S2. Pseudo-labels are generated for real unlabeled data. Confidence scores are calculated through local point cloud matching. The confidence threshold is dynamically adjusted using a cosine scheduling strategy. The two types of data are fused to train the model to achieve cross-domain generalization.

[0097] To reduce data labeling costs and maintain performance stability, a scheme combining pseudo-label semi-supervised learning and self-training strategies was designed based on the specific requirements of pose estimation algorithms. By combining labeled and unlabeled data, the model performance is improved in the case of scarce data labeling, with the aim of further enhancing the model's generalization ability.

[0098] The basic idea of ​​semi-supervised learning algorithms that combine pseudo-labels is to use existing models to predict unlabeled data, use high-confidence prediction results as pseudo-labels, and then train these pseudo-labels together with a limited number of real labeled data.

[0099] In pose estimation tasks, this method has significant advantages: First, simulation data provides the model with rich and accurate instance masks and pose annotations, enabling the network to efficiently learn the geometric structure and semantic features of target objects during the pre-training stage, laying a relatively solid knowledge foundation. Second, by directly applying the pre-trained model to real-world scenes, a large number of reliable pseudo-labels can be generated, thereby supplementing supervision information in the real data domain and further enhancing the model's adaptability to complex real-world scenes.

[0100] For the proposed pose estimation algorithm, a method is designed to select high-confidence predicted labels as pseudo-labels. This method evaluates the local distance between the transformed point cloud and the model point cloud and combines it with the foreground mask to construct a quantitative evaluation index that can reflect the alignment quality of the target, thus providing a reliable confidence basis for pseudo-label selection.

[0101] The calculation process for pose estimation confidence is as follows: 1. Transform the target object's point cloud using the predicted pose, calculate the Euclidean distance between each predicted point and all points in the model point cloud, and then select the closest point for each predicted point to obtain its minimum distance d. i .

[0102] 2. Construct a foreground mask m based on the matching labels. i (When point i belongs to the foreground, m) i =1; otherwise m i =0), so that subsequent scoring only considers the foreground points, thus effectively eliminating the interference of background noise.

[0103] 3. Set the minimum distance threshold d thresh = 0.2, this value is a hyperparameter obtained after multiple experiments and parameter tuning. For each predicted point, if its minimum distance is less than the preset threshold d thresh If the match is correct, the point is considered to be a correct match and assigned a value of 1; otherwise, it is assigned a value of 0, resulting in a binary matching score δ. i As shown in Equation 4-1: (4-1); 4. Multiply the above binary matching result with the foreground mask to obtain the matching score considering only the foreground points, and calculate the proportion r of correctly matched foreground points, as shown in Equation 4-2: (4-2); Where N represents the total number of points in the point cloud.

[0104] 5. To further reflect the sufficiency of foreground points in the sample, the correct matching ratio r is multiplied by the proportion of foreground points in the total number of points to obtain the pose matching confidence score c, as shown in Equation 4-3: (4-3); The purpose of this is that when a sample has fewer foreground points, its overall score will decrease even if the matching ratio is high; conversely, when there are more foreground points, the score will increase accordingly. This product operation effectively balances the impact of the foreground point ratio on the final predicted score, thus making the score more realistically reflect the actual alignment quality.

[0105] 6. Compare the confidence score c obtained from the pre-trained model inference with the confidence threshold c. thresh In comparison, it exceeds the confidence threshold c. thresh The data was used as pseudo-label data. To further improve the stability of training, c thresh It increases cosine-order with each training round, and its mathematical expression is: (4-4); Where: t represents the current training iteration round, T is the total number of training iterations, and c min and c max These are the initial and final confidence score thresholds, respectively. This cosine scheduling strategy allows the threshold to be lower in the early stages of training to fully utilize pseudo-labels for learning. As training progresses, the threshold is gradually increased, forcing the model to gradually adopt only pseudo-labels with higher confidence, thereby improving training stability and final alignment accuracy.

[0106] The entire semi-supervised learning process is as follows Figure 14 As shown. The specific training process based on pseudo-label semi-supervised learning is as follows: 1. In the pre-training phase, the model is first trained under full supervision using a labeled simulation dataset to achieve high accuracy in object instance segmentation and 6DoF pose estimation. Next, the pre-trained model is directly applied to an unlabeled real dataset to generate instance masks of the target objects and corresponding pose predictions (i.e., rotation matrices and translation vectors). Finally, confidence scores are calculated using the same method as for pose estimation confidence. Only samples with prediction confidence scores exceeding the confidence threshold of the current training epoch are retained as pseudo-labels. This filters out erroneous predictions to some extent, ensuring the quality of the pseudo-supervision information.

[0107] 2. After generating pseudo-labels, convert unlabeled real data samples into pseudo-labeled samples.

[0108] 3. During the joint training phase, labeled simulation data is mixed with unlabeled real data (with pseudo-labels). This allows the model to fully utilize the rich information from the precisely labeled simulation data, and also learn the latent distribution in the real data through the pseudo-labels, gradually reducing the domain difference between simulation and real data, thereby improving cross-domain generalization performance.

[0109] 4. A self-training strategy is employed. In each training phase, the pose estimation neural network is continuously fine-tuned using mixed labeled data, gradually improving the pre-trained model's performance on real data. This generates higher-quality pseudo-labels and updates the pseudo-label loss. Through continuous self-training, a positive feedback mechanism is formed, further promoting the model's performance in real-world scenarios.

[0110] The beneficial effects of the embodiments of the present invention include: At the data level: By using PyBullet + PyRender + domain adaptation technology, high-precision labeled datasets can be generated at low cost, improving labeling efficiency by more than 60% and significantly reducing data acquisition costs; At the segmentation level: The MaskNet algorithm integrates ViT and a multi-attention module, which significantly improves the utilization of multimodal features and achieves an instance segmentation accuracy of over 95% in stacked occluded scenes; At the pose estimation level: a two-stage strategy + mask reuse + weighted SVD reduces the 6DoF pose estimation error by 15%-20%, while maintaining high robustness in weak texture and high occlusion scenes. At the generalization level: pseudo-label semi-supervised learning enables a smooth transfer from the simulation domain to the real domain, improving the success rate of capturing real-world scenes to over 97%, and adapting to various types of industrial parts; Application level: It forms a closed-loop solution for the entire process, requiring no manual intervention, and can be directly adapted to 6DoF robotic arms, greatly improving the efficiency and intelligence level of industrial grasping operations.

[0111] Further reference Figure 15 As an implementation of the methods shown in the above figures, this application provides an embodiment of an intelligent recognition system for grasping scattered parts. This system embodiment is similar to... Figure 1 Corresponding to the method embodiments shown, the system can be specifically applied to various electronic devices.

[0112] Secondly, embodiments of the present invention also disclose an intelligent recognition system for grasping scattered parts, which is used to implement the intelligent recognition method as described in any one of the first aspects, such as... Figure 15 As shown, it includes: a simulation data generation module 151, a multimodal segmentation module 152, a pose estimation module 153, and a semi-supervised training module 154.

[0113] In one specific embodiment, the simulation data generation module 151 is used to perform a domain-adaptive simulation data synthesis step to generate a labeled simulation dataset; the multimodal segmentation module 152 is used to deploy the MaskNet algorithm to achieve multimodal instance segmentation and output a mask; The pose estimation module 153 is used to deploy the MaskPosenet algorithm and reuse the mask semantic information to output the 6DoF pose; the semi-supervised training module 154 is used to perform pseudo-label semi-supervised training steps to optimize the model's generalization performance.

[0114] The functions and methods of the above modules correspond to each other, and will not be repeated here.

[0115] The invention will be further described in detail below with reference to specific embodiments, taking the scenario of randomly scattered automotive bolts as an example: (I) Implementation of Simulation Data Synthesis 3D model preparation: Obtain the CAD model of the automotive bolts, import it into the PyBullet simulation environment, model the plane and box, and set the gravitational acceleration to 9.8 m / s² and the friction coefficient to 0.3; Scene generation: Randomly sample 5-10 bolts, determine the stack center, drop them from a height of 50cm, simulate free fall to a stable state, and form a disordered stacking scene; Rendering parameter settings: In PyRender, set the light intensity to 500-1000 lux, randomly adjust the texture roughness to 0.2-0.8, and generate a color map, depth map, and mask image; Data augmentation: Randomize bolt geometry (error ±5%) and illumination angle (0-360°) to generate 10,000 sets of simulation data, and label the mask, pose, and camera parameters.

[0116] (II) MaskNet Algorithm Training and Implementation Input parameters: The RGB image and depth map are both 640×480 pixels in size, and the XYZ feature map is generated by calculation through the camera intrinsic matrix; Network parameters: ViT depth n=20, URD module linear transformation channel number 1024, the feature map shape after slicing is H / 2×W / 2×16; AFF module local attention convolution kernel 3×3, global attention uses 2 layers of 1×1 convolution; DAN module learnable coefficients α, β initialized to 0; Training settings: batch size=16, learning rate=1e-4, training epochs=100, using stochastic gradient descent optimizer, loss function is cross-entropy loss + Dice loss.

[0117] (III) Implementation of the MaskPosenet Algorithm Feature extraction: The target point cloud was downsampled to 2048 points, and the model point cloud was sampled to 196 points from the farthest point. The geometric feature encoding dimension was 256. Coarse matching parameters: background marker dimension is the same as the number of feature channels (256 dimensions), number of Geometric Transformer network layers = 6, scaling factor temp = 0.07; Fine-match parameters: Position encoding uses sinusoidal encoding, Sparse to Dense Transformer layer number = 4, weighted SVD weights are calculated according to the formula. .

[0118] (iv) Implementation of pseudo-label semi-supervised training Confidence score calculation: distance threshold d thresh =0.2mm, foreground image area ≥30%; Threshold scheduling: Initial confidence threshold c min =0.5, final threshold c max =0.8, total training iterations T=200; joint training: the ratio of simulated labeled data to real pseudo-labeled data = 1:3, the learning rate is dynamically adjusted, decreasing to 0.8 times the previous round every 50 iterations.

[0119] (v) Test Results In scenarios involving 10-15 randomly stacked automotive bolts, the algorithm of this invention achieves an instance segmentation accuracy of 96.8%, a 6DoF pose estimation error of ≤0.3mm / 0.2°, a real-world scene capture success rate of 97.5%, reduces data annotation costs by 70%, and improves capture efficiency by more than 40% compared to existing technologies, fully meeting the needs of industrial production.

[0120] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A method for intelligent identification for scattered parts grasping, characterized in that, The method comprises the following steps: S1, field adaptive simulation data synthesis: a physical simulation environment is constructed by PyBullet to simulate the stacking of parts to a stationary pose, combined with PyRender to render light and texture, and object geometry, texture and light randomization is introduced to enhance the generation of a dataset containing depth images, instance segmentation masks and 6DoF pose labels; S2, multi-modal instance segmentation: a MaskNet algorithm based on an improved YOLO11 ViT is constructed, the depth data is encoded into XYZ feature maps and fused with RGB features, and the feature expression is optimized through URD, AFF and DAN modules, and the part instance segmentation mask is output; S3, 6DoF pose estimation: a MaskPosenet algorithm is constructed, the segmentation semantic information of step S2 is repeatedly used, a two-stage strategy of coarse matching and fine matching is adopted, a learnable background marker is combined with weighted SVD, and the part 6DoF pose is output; and S4, pseudo-label semi-supervised training: a model is pre-trained using the dataset of step S2, pseudo-labels are generated for real unlabeled data, confidence scores are calculated through local point cloud matching, a cosine scheduling strategy is used to dynamically adjust the confidence threshold, and the model is trained by fusing two types of data to realize cross-domain generalization.

2. The intelligent identification method of claim 1, wherein, In step S1, PyBullet simulates the free falling process of the parts, and after all the parts are stable, PyRender generates color image, depth image and mask image data.

3. The intelligent identification method of claim 1, wherein, When generating data labels in step S1, if the area of pixels with inconsistent depth within the mask contour is greater than one tenth of the total area of the mask, a polygon mask label is generated.

4. The intelligent identification method of claim 1, wherein, In the MaskNet, the depth of the ViT layer is set to n = 20, after the clstoken in the 0th dimension of ViT is discarded, the feature vector is reshaped into a feature map through linear transformation, slice shaping and downsampling.

5. The intelligent identification method of claim 1, wherein, The AFF module generates a fusion weight map by learning the local dependence of the modal through local attention and capturing scene semantics through global attention, and fuses the RGB features and depth features by weighting.

6. The intelligent identification method of claim 5, wherein, Specifically, it comprises: In the feature concatenation stage, first, the RGB features and the depth features are concatenated in the channel dimension to obtain a feature containing dual-modal information; Then, the feature flows into two attention modules at the same time, the local differences of the concatenated feature are learned in the local attention module; in the global attention module, the global statistical information is obtained by global average pooling, and a number of 1x1 convolution, BatchNorm and ReLU modules are used for channel interaction to capture more macro scene-level semantics; Finally, the local attention result and the global attention result are added to obtain a comprehensive attention map; And through the Sigmoid function, which is mapped to the interval [0, 1] to generate weights for the weighted fusion of RGB features and depth features.

7. The intelligent identification method of claim 1, wherein, The DAN module calculates spatial attention and channel attention in parallel, and adds the weighted result to the original feature after learning the coefficient to enhance the feature expression ability.

8. The intelligent identification method of claim 1, wherein, In the coarse matching stage, a learnable background marker is introduced, sparse color features and geometric features are input into the Geometric Transformer, and the initial pose is solved by singular value decomposition; The fine matching stage transforms the target point cloud according to the coarse matching attitude, superimposes the sinusoidal position coding and the dense color feature, fuses the features through the Sparse to Dense Transformer, and optimizes the attitude precision through the weighted SVD.

9. The intelligent identification method of claim 1, wherein, The confidence score is obtained by multiplying the correct proportion matched with the foreground point proportion, and the cosine scheduling threshold is obtained according to the formula Dynamic adjustment, wherein c thresh is a confidence threshold, c is a confidence score, c min and c max are initial and final confidence score thresholds respectively; t represents a current training iteration round, and T is a total training iteration number.

10. An intelligent identification system for scattered parts picking, characterized in that, The system is used for implementing the intelligent identification method as claimed in any one of claims 1-9, comprising: A simulation data generation module is configured to perform a field adaptive simulation data synthesis step to generate a simulation data set containing annotations; A multi-modal segmentation module is configured to deploy a MaskNet algorithm to implement multi-modal instance segmentation and output a mask; A pose estimation module is configured to deploy a MaskPosenet algorithm to reuse semantic information of the mask to output a 6DoF pose; A semi-supervised training module is configured to perform a pseudo-label semi-supervised training step to optimize the generalization performance of the model.

Images