Adaptive stacked workpiece point cloud data synthesis and instance segmentation method and device

Abstract

The invention relates to an adaptive stacked workpiece point cloud data synthesis and instance segmentation method and device, and the method comprises the steps: collecting the point cloud data of a single workpiece, and carrying out the automatic reverse modeling through a feature preserving reconstruction algorithm, so as to obtain a CAD model of the single workpiece; establishing and training a stacked workpiece instance segmentation model comprising a backbone network module, a semantic prediction module, a central point offset prediction module, a binary clustering module, a centripetal offset module and a voting module; and completing the point cloud data and refining edge details through a point cloud data completion and detail feature preserving algorithm, and inputting the processed point cloud data into the trained instance segmentation model to realize accurate instance segmentation of the stacked workpiece point cloud data. According to the invention, through technologies of adaptive point cloud data synthesis, deep learning instance segmentation, point cloud data completion and the like, the quality and segmentation precision of the point cloud data are significantly improved, and the method is suitable for workpiece identification and operation tasks in a complex industrial environment.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a method and device for adaptively synthesizing and instance segmenting point cloud data of stacked workpieces. Background Art

[0002] With the rapid development of modern manufacturing industry, the continuous improvement of automation and intelligence level has put forward higher requirements for the accurate identification and processing of workpieces. Especially in industrial production, stacking of workpieces is a common storage method, but it also brings challenges to automated sorting and processing. Existing workpiece instance segmentation methods are difficult to extract key effective features of workpieces when dealing with workpiece stacking, weak texture and other situations, resulting in low segmentation accuracy.

[0003] Point cloud data, as a form of three-dimensional data that describes the surface shape of an object, provides more information about the position and depth of the target object for instance segmentation of stacked workpieces, and has become one of the mainstream data in machine vision. However, when collecting and processing point cloud data, existing technologies are limited by the accuracy and shooting angle of three-dimensional scanners, resulting in missing point cloud data and blurred and deformed edge details. On the other hand, the point cloud instance segmentation method based on deep learning requires a large amount of training data, and the acquisition and annotation of point cloud data requires a lot of manpower, which makes it difficult for the workpiece instance segmentation method based on deep learning to adapt to different workpiece stacking scenarios. Summary of the invention

[0004] In order to solve the above problems, the present invention proposes a method and device for adaptively synthesizing point cloud data and instance segmentation of stacked workpieces, which reduces the dependence on high-cost and time-consuming manually labeled training data, while improving the accuracy of instance segmentation and the processing efficiency of automated sorting in the case of stacked workpieces.

[0005] The specific plan is as follows:

[0006] On the one hand, an adaptive stacked workpiece point cloud data synthesis and instance segmentation method includes:

[0007] S1, collects point cloud data of a single workpiece;

[0008] S2, establish a feature-preserving reconstruction algorithm to automatically reverse model the collected point cloud data of a single workpiece to obtain a CAD model of the single workpiece;

[0009] S3, generating stacked workpiece point cloud data based on the obtained CAD model of the single workpiece, and automatically annotating the point cloud data to obtain point cloud data with instance labels;

[0010] S4, builds and trains a stacked artifact instance segmentation model based on point cloud data with instance labels;

[0011] S5, collecting point cloud data and RGB images of different viewing angles of the stacked workpieces to be segmented;

[0012] S6, establishing a completion and detail feature preservation algorithm, based on RGB images of different viewing angles, using the established completion and detail feature preservation algorithm to complete the collected point cloud data of the stacked workpiece to be instance segmented and refine the edge details;

[0013] S7, inputting the point cloud data after completing and refining the edge details into the trained stacked workpiece instance segmentation model to complete the instance segmentation of the stacked workpiece point cloud data to be instance segmented.

[0014] Furthermore, the S2 specifically includes:

[0015] Point cloud denoising and normal vector initialization steps: Set any point on the point cloud Along its normal vector Move, define The obtained initialized normal vector n i ; Among them, p i Represents any point on the point cloud, p i ′ =p i +∈ i n i Indicates p i The new position of p j Indicates p i Points within the neighborhood r, p j ′ indicates p j For a point moved along its normal vector, the value of r is twice the average of the gaps between all points. Indicates that at point p i The covariance matrix at ;

[0016] Regularization processing steps for normal vectors in the edge area of ​​the point cloud: Let the set of initialized normal vectors μs obtained in the point cloud denoising and normal vector initialization steps be the original distribution, let μt be the target distribution, and build an optimization model to minimize the conversion cost from the original distribution to the target distribution. The model expression is When the conversion cost reaches the minimum, and k represents p i The number of points in the neighborhood is calculated according to λ j The value of can convert all vectors in the original distribution into vectors in the target distribution; where n j Represents the normal vector in the original distribution; the normal vectors in the edge area of ​​the point cloud are divided into two equal clusters, respectively recorded as and represents the first representative normal vector in the original distribution and serves as the first target distribution, Represents the second representative normal vector in the original distribution and serves as the second target distribution; j Used to control the normal vector n j Transformed into the target distribution or When j =1, it means that the normal vector n j Convert to normal vector When j = 0, indicating the normal vector n j Convert to normal vector

[0017] Identify the edge area of ​​the point cloud and adjust the position of the edge points based on the regularized normal: Determine the representative normal vector Is the angle greater than 30° and the conversion cost less than or equal to 0.25? If so, then determine point p i For the edge area points, repeat the point cloud denoising and normal vector initialization steps to complete the position of the edge point based on the regularized normal adjustment;

[0018] Steps for obtaining a CAD model of a single workpiece: A CAD model of a single workpiece is obtained by using Poisson reconstruction.

[0019] Furthermore, the covariance matrix is ​​used to characterize the point p i The direction of the local plane, if n i For p i The correct normal vector is Close to zero vector, the optimization model is constructed based on this When the optimization model formula reaches the minimum value, the initialized normal vector n can be obtained. i and decide i The distance parameter ∈ of the moving distance i .

[0020] Furthermore, the S3 specifically includes:

[0021] Virtual scene preparation step: simulating the industrial scene, and randomly importing a CAD model of a single workpiece, wherein the number, material, initial coordinates, and rotation angle of the CAD model are randomly generated;

[0022] The simulation steps of the surface structured light 3D scanner are as follows: an infrared projector and left and right IR stereo cameras are configured in a virtual scene, coded stripes are projected onto the surface of stacked workpieces by the infrared projector, stripe images reflected at different angles on the virtual object are collected by the left and right IR stereo cameras, the same-name points of the images collected by the left and right IR stereo cameras are determined by phase decoding, the 3D coordinate points of the object are reconstructed by a stereo matching algorithm, and the synthetic point cloud data of the stacked workpieces are obtained;

[0023] Steps for automatic annotation of point cloud data: disable the visibility of all artifacts in the virtual scene, enable the visibility of one of the artifacts separately, generate synthetic point cloud data of the artifact through the surface structured light 3D scanner simulation step, calculate the center of mass of the instance and assign labels representing the artifact to all points in the point cloud. After traversing all artifacts, the labels of all points in the space can be obtained to complete the automatic annotation of the point cloud data of stacked artifacts.

[0024] Further, in S4, the established stacked artifact instance segmentation model includes a backbone network, a semantic prediction module, a center point offset prediction module, a binary clustering module, a centripetal offset module and a voting module; the backbone network includes 3DUnet and two fully connected layers;

[0025] Train a stacked artifact instance segmentation model, including:

[0026] S41, inputting the three-dimensional space coordinate information xyz, color information rgb and normal vector information of the point cloud into the backbone network to obtain point-by-point high-dimensional features of the point cloud;

[0027] S42, feeds the high-dimensional features of each point to the semantic prediction module composed of 3 layers of MLP to predict the semantic score of each class. The semantic score is recorded as S∈[0,1] N×M , where N and M are the number of points and classes respectively, and the class with the highest score will be the semantic label of the point;

[0028] S43, the center point offset prediction module and the semantic prediction module use 3 layers of MLP to predict the offset of each point Where i∈{1,…,N}, let is the centroid of the instance to which point i belongs, is the 3D coordinate of point i, and L1 regression loss is used to constrain points with the same instance label to learn the offset. The calculation formula of L1 regression loss is as follows: in, and Respectively represent the predicted offset of point i in the x-axis, y-axis and z-axis directions in three-dimensional space, and Respectively represent the coordinate values ​​of the center of mass of the instance to which point i belongs on the x-axis, y-axis, and z-axis, and Represent the coordinate values ​​of point i on the x-axis, y-axis and z-axis respectively;

[0029] S44, calculate the center score of each point, the center score calculation formula is: Among them, d max is the distance between the centroid of each instance and the farthest point, where Scetnetr(i) is the center score of point i;

[0030] S45, by setting the threshold θ in the binary clustering module d , classify the points with center scores greater than the threshold as high center value points HCPs, and classify the remaining points as low center value points LCPs, and cluster the HCPs with the same semantic label by combining the DBSCAN clustering algorithm with the semantic prediction result of step S42;

[0031] S46, using the centripetal shift module, subtracts the offset o of each point predicted in step S43 from the LCPs coordinates. i Make LCPs gather towards the center of the instance;

[0032] S47, through the voting module, selects HCPs that share the same semantic label with LCPs, counts which instance these HCPs belong to within the mean size range of each category, and uses this as the attribution of LCPs.

[0033] Furthermore, the S6 specifically includes:

[0034] S61, after obtaining the multi-view image data, firstly voxelize the three-dimensional space where the workpieces are stacked, and sample the voxel center point as a candidate point;

[0035] S62, projecting the candidate three-dimensional space point onto images of different perspectives using the camera's intrinsic and extrinsic parameters, and then selecting a square neighborhood on the images of different perspectives with the projection point of the candidate point as the center;

[0036] S63, calculating the photometric consistency within the square neighborhood on the images of different viewpoints, and restoring the three-dimensional information from the two-dimensional images of different viewpoints using a stereo matching algorithm for the candidate points with consistent photometric consistency, thereby completing the completion of the missing points of the point cloud;

[0037] S64, refine the edge details of the point cloud data of the stacked workpiece to be instance segmented through the point cloud denoising and normal vector initialization steps, the normal vector regularization processing step of the point cloud edge area, and the step of identifying the point cloud edge area and adjusting the position of the edge point based on the regularized normal, so as to obtain the completed point cloud data.

[0038] On the other hand, an adaptive stacked workpiece point cloud data synthesis and instance segmentation device includes:

[0039] A point cloud data acquisition module for a single workpiece, used to acquire point cloud data for a single workpiece;

[0040] The CAD model acquisition module of a single workpiece is used to establish a feature-preserving reconstruction algorithm, automatically reverse model the collected point cloud data of a single workpiece, and obtain the CAD model of the single workpiece;

[0041] The data automatic annotation module is used to generate the point cloud data of the stacked workpiece based on the obtained CAD model of the single workpiece, and automatically annotate the point cloud data to obtain the point cloud data with instance labels;

[0042] An instance segmentation model building and training module is used to build and train a stacked artifact instance segmentation model based on point cloud data with instance labels;

[0043] A data collection module for stacked workpieces to be segmented by an instance, used to collect point cloud data and RGB images of stacked workpieces to be segmented by an instance from different perspectives;

[0044] The point cloud data completion module for stacked workpieces to be segmented is used to establish a point cloud data completion and detail feature preservation algorithm for stacked workpieces, and complete the collected point cloud data of stacked workpieces to be segmented based on RGB images from different perspectives and refine edge details;

[0045] The instance segmentation module is used to input the point cloud data with completed and refined edge details into the trained instance segmentation model to complete the instance segmentation of the stacked workpiece point cloud data to be instance segmented.

[0046] The present invention adopts the above technical solution and has the following beneficial effects:

[0047] (1) The present invention generates a CAD model of a single workpiece by establishing a feature-preserving reconstruction algorithm, and generates virtual stacked workpiece point cloud data based on the principles of physical rendering and surface structured light 3D scanner, and automatically annotates these data, thereby reducing the reliance on high-cost and time-consuming manual annotation training data, significantly reducing the time and cost of data preparation, and improving the efficiency and quality of model training;

[0048] (2) The stacked artifact instance segmentation model proposed in the present invention includes a backbone network, a semantic prediction module, a center point offset prediction module, a binary clustering module, a centripetal offset module and a voting module. Through multi-stage feature extraction and optimization, the model can accurately identify and segment stacked artifacts, and can maintain high accuracy even in the case of weak textures and complex stacking, significantly improving the accuracy and robustness of artifact recognition.

[0049] (3) The present invention completes the collected point cloud data of real stacked workpieces and optimizes edge details through a point cloud data completion and detail feature preservation algorithm. The algorithm uses multi-view image information and stereo matching technology to restore missing parts in the point cloud data, and refines edge details through a feature preservation reconstruction algorithm, thereby improving the overall quality and integrity of the point cloud data, thereby providing a more reliable data foundation for subsequent instance segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1The method for adaptively stacking workpiece point cloud data synthesis and instance segmentation according to an embodiment of the present invention;

[0051] Figure 2 A flow chart of a feature-preserving reconstruction algorithm according to an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of synthetic data according to an embodiment of the present invention;

[0053] Figure 4 It is a framework diagram of an instance segmentation model of an example of the present invention;

[0054] Figure 5 This is a flowchart of the point cloud completion and detail feature preservation algorithm of an example of the present invention;

[0055] Figure 6 It is a structural block diagram of an adaptive stacked workpiece point cloud data synthesis and instance segmentation device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention is further described in detail below in conjunction with embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0057] like Figure 1 As shown, the present invention provides an adaptive stacked workpiece point cloud data synthesis and instance segmentation method, comprising:

[0058] S1, collect point cloud data of a single workpiece.

[0059] S2, establish a feature-preserving reconstruction algorithm, automatically reverse model the collected point cloud data of a single workpiece, and obtain a CAD model of the single workpiece.

[0060] Specifically, Figure 2 As shown, the establishment of a feature preserving reconstruction algorithm, automatic reverse modeling of the collected point cloud data of a single workpiece, and obtaining a CAD model of the single workpiece specifically includes:

[0061] Point cloud denoising and normal vector initialization steps: Set any point on the point cloud Along its normal vector Move, define The obtained initialized normal vector n i ; Among them, p i Represents any point on the point cloud, p′ i =p i +∈ i n i Indicates p i The new position of p j Indicates p i Points within the neighborhood r, p j ′ indicates pj For a point moved along its normal vector, the value of r is twice the average of the gaps between all points. represents the covariance matrix at point pi;

[0062] Regularization processing steps for normal vectors in the edge area of ​​the point cloud: Let the set of initialized normal vectors μs obtained in the point cloud denoising and normal vector initialization steps be the original distribution, let μt be the target distribution, and build an optimization model to minimize the conversion cost from the original distribution to the target distribution. The model expression is When the conversion cost reaches the minimum, and k represents p i The number of points in the neighborhood is calculated according to λ j The value of can convert all vectors in the original distribution into vectors in the target distribution; where n j Represents the normal vector in the original distribution; the normal vectors in the edge area of ​​the point cloud are divided into two equal clusters, respectively recorded as and represents the first representative normal vector in the original distribution and serves as the first target distribution, Represents the second representative normal vector in the original distribution and serves as the second target distribution; j Used to control the normal vector n j Transformed into the target distribution or When j =1, it means that the normal vector n j Convert to normal vector When j = 0, indicating the normal vector n j Convert to normal vector

[0063] Identify the edge area of ​​the point cloud and adjust the position of the edge points based on the regularized normal: Determine the representative normal vector Is the angle greater than 30° and the conversion cost less than or equal to 0.25? If so, then determine point p i For the edge area points, repeat the point cloud denoising and normal vector initialization steps to complete the position of the edge point based on the regularized normal adjustment;

[0064] Steps for obtaining a CAD model of a single workpiece: A CAD model of a single workpiece is obtained by using Poisson reconstruction.

[0065] Specifically, the covariance matrix is ​​used to characterize the point p i The direction of the local plane at n, that is, the covariance matrix encodes the direction of the local surface. i For p i The correct normal vector is ni Can reflect the true normal vector information, then is close to the zero vector, so by minimizing This method can achieve the effect of denoising and initializing the normal vector at the same time, so this is used as the basis for building the optimization model When the model formula reaches the minimum value, the initialized normal vector n can be obtained. i and decide i The distance parameter ∈ of the moving distance i .

[0066] S3, generating stacked workpiece point cloud data based on the obtained CAD model of the single workpiece, and automatically annotating the point cloud data to obtain point cloud data with instance labels.

[0067] Specifically, Figure 3 As shown, the method of generating stacked workpiece point cloud data based on the obtained CAD model of a single workpiece and automatically annotating the point cloud data to obtain point cloud data with instance labels specifically includes:

[0068] Virtual scene preparation step: simulating the industrial scene, and randomly importing a CAD model of a single workpiece, wherein the number, material, initial coordinates, and rotation angle of the CAD model are randomly generated;

[0069] The simulation steps of the surface structured light 3D scanner are as follows: an infrared projector and left and right IR stereo cameras are configured in a virtual scene, coded stripes are projected onto the surface of stacked workpieces by the infrared projector, stripe images reflected at different angles on the virtual object are collected by the left and right IR stereo cameras, the same-name points of the images collected by the left and right IR stereo cameras are determined by phase decoding, the 3D coordinate points of the object are reconstructed by a stereo matching algorithm, and the synthetic point cloud data of the stacked workpieces are obtained;

[0070] Steps for automatic annotation of point cloud data: disable the visibility of all artifacts in the virtual scene, enable the visibility of one of the artifacts separately, generate synthetic point cloud data of the artifact through the surface structured light 3D scanner simulation step, calculate the center of mass of the instance and assign labels representing the artifact to all points in the point cloud. After traversing all artifacts, the labels of all points in the space can be obtained to complete the automatic annotation of the point cloud data of stacked artifacts.

[0071] S4, establish and train a stacked artifact instance segmentation model based on point cloud data with instance labels.

[0072] Specifically, Figure 4 As shown, the establishment and training of the stacked artifact instance segmentation model specifically includes:

[0073] The established stacked artifact instance segmentation model includes a backbone network, a semantic prediction module, a center point offset prediction module, a binary clustering module, a centripetal offset module and a voting module; the backbone network includes 3DUnet and two fully connected layers;

[0074] Train a stacked artifact instance segmentation model, including:

[0075] S41, inputting the three-dimensional space coordinate information xyz, color information rgb and normal vector information of the point cloud into the backbone network to obtain point-by-point high-dimensional features of the point cloud;

[0076] S42, feeds the high-dimensional features of each point to the semantic prediction module composed of 3 layers of MLP to predict the semantic score of each class. The semantic score is recorded as S∈[0,1] N×M , where N and M are the number of points and classes respectively, and the class with the highest score will be the semantic label of the point;

[0077] S43, the center point offset prediction module and the semantic prediction module use 3 layers of MLP to predict the offset of each point Where i∈{1,…,N}, let is the centroid of the instance to which point i belongs, is the 3D coordinate of point i, and L1 regression loss is used to constrain points with the same instance label to learn the offset. The calculation formula of L1 regression loss is as follows: in, and Respectively represent the predicted offset of point i in the x-axis, y-axis and z-axis directions in three-dimensional space, and Respectively represent the coordinate values ​​of the center of mass of the instance to which point i belongs on the x-axis, y-axis, and z-axis, and Represent the coordinate values ​​of point i on the x-axis, y-axis and z-axis respectively;

[0078] S44, calculate the center score of each point, the center score calculation formula is: Among them, d max is the distance between the centroid of each instance and the farthest point, where Scetnetr(i) is the center score of point i;

[0079] S45, by setting the threshold θ in the binary clustering module d , classify the points with center scores greater than the threshold as high center value points HCPs, and classify the remaining points as low center value points LCPs, and cluster the HCPs with the same semantic label by combining the DBSCAN clustering algorithm with the semantic prediction result of step S42;

[0080] S46, using the centripetal shift module, subtracts the offset o of each point predicted in step S43 from the LCPs coordinates. i Make LCPs gather towards the center of the instance;

[0081] S47, through the voting module, selects HCPs that share the same semantic label with LCPs, counts which instance these HCPs belong to within the mean size range of each category, and uses this as the attribution of LCPs.

[0082] S5, collecting point cloud data of the stacked workpiece to be instance segmented and RGB images of different viewing angles.

[0083] S6, establishing a point cloud data completion and detail feature preservation algorithm for stacked workpieces, completing the collected point cloud data of the stacked workpieces to be instance segmented and refining the edge details based on RGB images from different perspectives.

[0084] Specifically, Figure 5 As shown, the point cloud data completion and detail feature preservation algorithm of the stacked workpieces is established to complete the collected point cloud data of the stacked workpieces to be instance segmented and refine the edge details, specifically including:

[0085] S61, after obtaining the multi-view image data, firstly voxelize the three-dimensional space where the workpieces are stacked, and sample the voxel center point as a candidate point;

[0086] S62, projecting the candidate three-dimensional space point onto images of different perspectives using the camera's intrinsic and extrinsic parameters, and then selecting a square neighborhood on the images of different perspectives with the projection point of the candidate point as the center;

[0087] S63, calculate the photometric consistency in the square area on the images of different viewpoints, and judge the probability of the candidate point on the surface of the object according to the photometric consistency. For the candidate point with high probability, we use the stereo matching algorithm to restore the three-dimensional information from the two-dimensional images of different viewpoints, so as to complete the missing points of the point cloud;

[0088] S64, refines the edge details of the real point cloud data through the point cloud denoising and normal vector initialization steps, the normal vector regularization step of the point cloud edge area, and the step of identifying the point cloud edge area and adjusting the position of the edge point based on the regularized normal, so as to obtain the completed high-quality point cloud data; wherein, the point cloud denoising and normal vector initialization steps, the normal vector regularization step of the point cloud edge area, and the step of identifying the point cloud edge area are specifically implemented the same as S2.

[0089] S7, inputting the point cloud data after completing and refining the edge details into the trained instance segmentation model to complete the instance segmentation of the stacked workpiece point cloud data to be instance segmented.

[0090] like Figure 6 As shown, this embodiment also discloses an adaptive stacked workpiece point cloud data synthesis and instance segmentation device, including:

[0091] A single workpiece point cloud data acquisition module 601 is used to acquire point cloud data of a single workpiece;

[0092] A single workpiece CAD model acquisition module 602 is used to establish a feature-preserving reconstruction algorithm, automatically reverse model the collected point cloud data of the single workpiece, and obtain a CAD model of the single workpiece;

[0093] The data automatic labeling module 603 is used to generate stacked workpiece point cloud data based on the obtained CAD model of a single workpiece, and automatically label the point cloud data to obtain point cloud data with instance labels;

[0094] An instance segmentation model building and training module 604 is used to build and train a stacked workpiece instance segmentation model based on point cloud data with instance labels;

[0095] The data collection module 605 for stacked workpieces to be segmented by an instance is used to collect point cloud data and RGB images of different viewing angles of the stacked workpieces to be segmented by an instance;

[0096] The point cloud data completion module 606 for stacked workpieces to be segmented by an instance is used to establish a point cloud data completion and detail feature preservation algorithm for stacked workpieces, and to complete the collected point cloud data of the stacked workpieces to be segmented by an instance based on RGB images of different viewing angles and refine edge details;

[0097] The instance segmentation module 607 is used to input the point cloud data after completing and refining the edge details into the trained instance segmentation model to complete the instance segmentation of the stacked workpiece point cloud data to be instance segmented.

[0098] The specific implementation of an adaptive stacked workpiece point cloud data synthesis and instance segmentation device is the same as the adaptive stacked workpiece point cloud data synthesis and instance segmentation method, and this embodiment will not be repeated.

[0099] Although the present invention has been specifically shown and described in conjunction with the preferred embodiments, it should be understood by those skilled in the art that various changes may be made to the present invention in form and details without departing from the spirit and scope of the present invention as defined by the appended claims, all of which are within the scope of protection of the present invention.

Claims

1. An adaptive stacked workpiece point cloud data synthesis and instance segmentation method, characterized in that: include: S1, collects point cloud data of a single workpiece; S2, establish a feature-preserving reconstruction algorithm to automatically reverse model the collected point cloud data of a single workpiece to obtain a CAD model of the single workpiece; S3, generating stacked workpiece point cloud data based on the obtained CAD model of the single workpiece, and automatically annotating the point cloud data to obtain point cloud data with instance labels; S4, builds and trains a stacked artifact instance segmentation model based on point cloud data with instance labels; S5, collecting point cloud data and RGB images of different viewing angles of the stacked workpieces to be segmented; S6, establishing a completion and detail feature preservation algorithm, based on RGB images of different viewing angles, using the established completion and detail feature preservation algorithm to complete the collected point cloud data of the stacked workpiece to be instance segmented and refine the edge details; S7, inputting the point cloud data after completing and refining the edge details into the trained stacked workpiece instance segmentation model to complete the instance segmentation of the stacked workpiece point cloud data to be instance segmented.

2. The method for adaptively stacking workpiece point cloud data synthesis and instance segmentation according to claim 1, characterized in that: The S2 specifically includes: Point cloud denoising and normal vector initialization steps: Set any point on the point cloud Along its normal vector Move, define The obtained initialized normal vector n i ; Among them, p i Represents any point on the point cloud, p i ′ =p i +∈ i n i Indicates p i The new position of p j Indicates p i Points within the neighborhood r, p j ′ indicates p j The value of r for a point moved along its normal vector is twice the average value of the gaps between all points. Indicates that at point p i The covariance matrix at ; Regularization processing steps for normal vectors in the edge area of ​​the point cloud: Let the set of initialized normal vectors μs obtained in the point cloud denoising and normal vector initialization steps be the original distribution, let μt be the target distribution, and build an optimization model to minimize the conversion cost from the original distribution to the target distribution. The model expression is When the conversion cost reaches the minimum, and k represents p i The number of points in the neighborhood is calculated according to λ j The value of can convert all vectors in the original distribution into vectors in the target distribution; where n j Represents the normal vector in the original distribution; the normal vectors in the edge area of ​​the point cloud are divided into two equal clusters, respectively recorded as and represents the first representative normal vector in the original distribution and serves as the first target distribution, Represents the second representative normal vector in the original distribution and serves as the second target distribution; j Used to control the normal vector n j Transformed into the target distribution or When j =1, it means that the normal vector n j Convert to normal vector When j = 0, indicating the normal vector n j Convert to normal vector Identify the edge area of ​​the point cloud and adjust the position of the edge points based on the regularized normal: Determine the representative normal vector Is the angle greater than 30° and the conversion cost less than or equal to 0.25? If so, then determine point p i For the edge area points, repeat the point cloud denoising and normal vector initialization steps to complete the position adjustment of the edge points based on the regularized normal; Steps for obtaining a CAD model of a single workpiece: A CAD model of a single workpiece is obtained by using Poisson reconstruction.

3. The method for adaptively stacking workpiece point cloud data synthesis and instance segmentation according to claim 2, characterized in that: The covariance matrix is ​​used to characterize the point p i The direction of the local plane, if n i For p i The correct normal vector is Close to zero vector, the optimization model is constructed based on this When the optimization model formula reaches the minimum value, the initialized normal vector n can be obtained. i and decide i The distance parameter ∈ of the moving distance i .

4. The method for adaptively stacking workpiece point cloud data synthesis and instance segmentation according to claim 1, characterized in that: The S3 specifically includes: Virtual scene preparation step: simulating the industrial scene, and randomly importing a CAD model of a single workpiece, wherein the number, material, initial coordinates, and rotation angle of the CAD model are randomly generated; The simulation steps of the surface structured light 3D scanner are as follows: an infrared projector and left and right IR stereo cameras are configured in a virtual scene, coded stripes are projected onto the surface of stacked workpieces by the infrared projector, stripe images reflected at different angles on the virtual object are collected by the left and right IR stereo cameras, the same-name points of the images collected by the left and right IR stereo cameras are determined by phase decoding, the 3D coordinate points of the object are reconstructed by a stereo matching algorithm, and the synthetic point cloud data of the stacked workpieces are obtained; Steps for automatic annotation of point cloud data: disable the visibility of all artifacts in the virtual scene, enable the visibility of one of the artifacts separately, generate synthetic point cloud data of the artifact through the surface structured light 3D scanner simulation step, calculate the center of mass of the instance and assign labels representing the artifact to all points in the point cloud. After traversing all artifacts, the labels of all points in the space can be obtained to complete the automatic annotation of the point cloud data of stacked artifacts.

5. The method for adaptively stacking workpiece point cloud data synthesis and instance segmentation according to claim 1, characterized in that: In S4, the established stacked artifact instance segmentation model includes a backbone network, a semantic prediction module, a center point offset prediction module, a binary clustering module, a centripetal offset module and a voting module; the backbone network includes 3DUnet and two fully connected layers; Train a stacked artifact instance segmentation model, including: S41, inputting the three-dimensional space coordinate information xyz, color information rgb and normal vector information of the point cloud into the backbone network to obtain point-by-point high-dimensional features of the point cloud; S42, feeds the high-dimensional features of each point to the semantic prediction module composed of 3 layers of MLP to predict the semantic score of each class. The semantic score is recorded as S∈[0,1] N×M , where N and M are the number of points and classes respectively, and the class with the highest score will be the semantic label of the point; S43, the center point offset prediction module and the semantic prediction module use 3 layers of MLP to predict the offset of each point Where i∈{1,…,N}, let is the centroid of the instance to which point i belongs, is the 3D coordinate of point i, and L1 regression loss is used to constrain points with the same instance label to learn the offset. The calculation formula of L1 regression loss is as follows: in, and Respectively represent the predicted offset of point i in the x-axis, y-axis and z-axis directions in three-dimensional space, and Respectively represent the coordinate values ​​of the center of mass of the instance to which point i belongs on the x-axis, y-axis, and z-axis, and Represent the coordinate values ​​of point i on the x-axis, y-axis and z-axis respectively; S44, calculate the center score of each point, the center score calculation formula is: Among them, d max is the distance between the centroid of each instance and the farthest point, where Scetnetr(i) is the center score of point i; S45, by setting the threshold θ in the binary clustering module d , classify the points with center scores greater than the threshold as high center value points HCPs, and classify the remaining points as low center value points LCPs, and cluster the HCPs with the same semantic label by combining the DBSCAN clustering algorithm with the semantic prediction result of step S42; S46, using the centripetal shift module, subtracts the offset o of each point predicted in step S43 from the LCPs coordinates. i Make LCPs gather towards the center of the instance; S47, through the voting module, selects HCPs that share the same semantic label with LCPs, counts which instance these HCPs belong to within the mean size range of each category, and uses this as the attribution of LCPs.

6. The method for adaptively stacking workpiece point cloud data synthesis and instance segmentation according to claim 1, characterized in that: The S6 specifically includes: S61, after obtaining the multi-view image data, firstly voxelize the three-dimensional space where the workpieces are stacked, and sample the voxel center point as a candidate point; S62, projecting the candidate three-dimensional space point onto images of different perspectives using the camera's intrinsic and extrinsic parameters, and then selecting a square neighborhood on the images of different perspectives with the projection point of the candidate point as the center; S63, calculating the photometric consistency within the square neighborhood on the images of different viewpoints, and restoring the three-dimensional information from the two-dimensional images of different viewpoints using a stereo matching algorithm for the candidate points with consistent photometric consistency, thereby completing the completion of the missing points of the point cloud; S64, refine the edge details of the point cloud data of the stacked workpiece to be instance segmented through the point cloud denoising and normal vector initialization steps, the normal vector regularization processing step of the point cloud edge area, and the step of identifying the point cloud edge area and adjusting the position of the edge point based on the regularized normal, so as to obtain the completed point cloud data.

7. An adaptive stacked workpiece point cloud data synthesis and instance segmentation device, characterized in that: include: A point cloud data acquisition module for a single workpiece, used to acquire point cloud data for a single workpiece; The CAD model acquisition module of a single workpiece is used to establish a feature-preserving reconstruction algorithm, automatically reverse model the collected point cloud data of a single workpiece, and obtain the CAD model of the single workpiece; The data automatic annotation module is used to generate the point cloud data of the stacked workpiece based on the obtained CAD model of the single workpiece, and automatically annotate the point cloud data to obtain the point cloud data with instance labels; An instance segmentation model building and training module is used to build and train a stacked artifact instance segmentation model based on point cloud data with instance labels; A data collection module for stacked workpieces to be segmented by an instance, used to collect point cloud data and RGB images of stacked workpieces to be segmented by an instance from different perspectives; The point cloud data completion module for stacked workpieces to be segmented is used to establish a point cloud data completion and detail feature preservation algorithm for stacked workpieces, and complete the collected point cloud data of stacked workpieces to be segmented based on RGB images from different perspectives and refine edge details; The instance segmentation module is used to input the point cloud data with completed and refined edge details into the trained instance segmentation model to complete the instance segmentation of the stacked workpiece point cloud data to be instance segmented.

Images