Disordered material identification and positioning method and system based on 3D visual technology

Abstract

The invention provides a disordered material identification and positioning method and system based on a 3D visual technology. The method comprises the steps that S1, a 3D model of a material object is converted into point cloud data, downsampling processing is conducted on the point cloud data, template point cloud of the material object is generated, and PPF features of the template point cloud of the material object are calculated; s2, in a disorderly stacked material scene, based on the obtained RGB image and depth image of the material scene, performing material segmentation by using an SAM model to obtain a material point cloud of each instance, and calculating a PPF feature of the material point cloud in the material scene; and S3, matching the PPF features of the material point clouds in the material scene with the PPF features of the template point clouds to obtain a 6D pose of each material, and realizing recognition and positioning of disordered materials through single matching. According to the method, the deep learning model and the 3D vision technology are combined, the accuracy and efficiency of disorderly stacked material recognition are improved, and the method is widely applied to automatic grabbing and assembling tasks in storage, production line and other complex environments.

Description

Technical Field

[0001] The present invention relates to the field of robot automation technology, and in particular to a method and system for identifying and locating disordered materials based on 3D vision technology. Background Art

[0002] In the field of automated production and warehousing, robots are often used for material grabbing and assembly tasks. However, most existing automated systems rely on the orderly placement of materials, or require complex teaching processes to specify the exact position and posture of materials, which not only increases the complexity of operation, but also reduces the flexibility and efficiency of the system.

[0003] Although traditional 2D vision systems can identify surface features of objects, their accuracy and robustness are greatly limited when dealing with scenes with complex stacking, partial occlusion, and disordered placement of materials. Although traditional 3D vision systems can generate three-dimensional models of materials by acquiring depth information of materials, thereby more accurately identifying the position and posture of materials, they also have many limitations, including: difficulty in segmenting materials in complex scenes, disorderly stacked materials often occlude each other, and traditional rule-based methods are difficult to effectively segment; inaccurate recognition of material posture: existing systems often rely on manual marking and teaching to determine the posture of objects, resulting in low processing efficiency, especially when the material shape is complex or irregular; unable to cope with dynamic scenes: in production or warehousing environments, the state of materials may change continuously, and existing systems find it difficult to adjust grasping and assembly strategies in real time.

[0004] Therefore, how to effectively use 3D vision technology combined with deep learning models to solve the problem of identifying and positioning disordered stacked materials has become an important challenge in the current automation field.

[0005] Patent document CN117001404A (application number: 202311051378.X) discloses an automatic loading and unloading device for machine tools, which belongs to the field of mechanical processing technology; it includes a double-station processing module, a material storage module, a vision module and a quick-change clamping claw module, the material storage module includes a loading material tray and a unloading material tray, the vision module is a 3D camera, and the 3D camera uses a target recognition and positioning algorithm to match and locate the CAD model of the material with the 3D image, and quickly and accurately obtain the 3D posture of the material. A six-axis robot is arranged between the double-station processing modules. Summary of the invention

[0006] In view of the defects in the prior art, the purpose of the present invention is to provide a method and system for identifying and locating disordered materials based on 3D vision technology.

[0007] According to the present invention, a method for identifying and locating disordered materials based on 3D vision technology includes:

[0008] Step S1: converting the 3D model of the material object into point cloud data, downsampling the point cloud data, generating a template point cloud of the material object, and calculating the PPF feature of the template point cloud of the material object;

[0009] Step S2: In the disorderly stacked material scene, based on the acquired RGB image and depth image of the material scene, the SAM model is used to segment the materials, obtain the material point cloud of each instance, and calculate the PPF feature of the material point cloud in the material scene;

[0010] Step S3: Match the PPF features of the material point cloud in the material scene with the PPF features of the template point cloud to achieve 6D pose estimation for each material.

[0011] Preferably, the step S1 comprises:

[0012] Step S1.1: converting the 3D model of the material object into point cloud data;

[0013] Step S1.2: downsampling the point cloud data using a voxel grid method to obtain downsampled point cloud data;

[0014] Step S1.3: downsampling the downsampled point cloud data through a downsampling technique based on geometric features to obtain processed point cloud data and generate a template point cloud of the material object;

[0015] Step S1.4: Calculate the PPF features of the material object template point cloud based on the processed point cloud data.

[0016] Preferably, the step S1.2 comprises:

[0017] Step S1.2.1: dividing the point cloud data of the material object into a plurality of voxels according to a preset voxel size;

[0018] Step S1.2.2: In each voxel, the geometric center point in the voxel is selected as the representative point to implement downsampling processing;

[0019]

[0020] Among them, S j represents the set of points within each voxel, (x i ,y i ,z i ) represents the spatial coordinates of each point within the voxel.

[0021] Preferably, the step S1.3 comprises:

[0022] Step S1.3.1: Calculate the normal vector of each point based on all points selected by the voxel grid method;

[0023]

[0024] Among them, N i Indicates point P i The neighborhood point set, n j is the neighborhood point P j The normal vector of |N i | is the number of neighborhood points;

[0025] Step S1.3.2: Based on all points selected by the voxel grid method, calculate the curvature k of each point i ;

[0026]

[0027] Among them, r i It's point P i The radius of curvature at , which indicates the degree of curvature around the point;

[0028] Step S1.3.3: According to the changes of the normal vector and the curvature, determine whether the point is a redundant point, delete the redundant points, retain the points that meet the preset requirements, and implement downsampling processing;

[0029] The change of the normal vector is obtained by calculating the angle between the normal vectors of adjacent points:

[0030] cosθ=n1·n2 / (||n1|||n2||))

[0031] The composite metric is calculated based on the change of the normal vector and the curvature, and compared with the set threshold. When the composite metric is greater than the set threshold, it is judged as a redundant point.

[0032] S=α·|Δn|+β·κ

[0033] Among them, α represents the weight of the normal vector change; β represents the curvature weight; Δn represents the change of the normal vector; and k represents the curvature.

[0034] Preferably, the step S1.4 comprises:

[0035] Step S1.4.1: For the downsampled point cloud, calculate each point p i The normal vector N(p i );

[0036] N(p i )=normal(p i )

[0037] Among them, normal means Pi Point normal vector, calculated by P i Neighborhood points n i The normal vector of is obtained;

[0038]

[0039] Step S1.4.2: For the downsampled point cloud, calculate any two points (p i , p j )’s point pair feature PPF;

[0040] F(m1, m2)=(||d||, ∠(n1, d), ∠(n2,d), ∠(n1, n2))

[0041] T PPF ={F(m i , m j )|(m i , m j )∈M, i≠j}

[0042] Among them, F(m1, m2) represents any two points (p i , p j )’s point pair feature PPF; d represents the Euclidean distance between two points; ∠(n1, n2) represents the angle between vectors; ∠(n1, d)∠(n2, d) represents the angle between the normal vector and the point pair vector; M represents a point on the model.

[0043] Preferably, step S2 comprises:

[0044] Step S2.1: Based on the RGB-D camera, obtain the RGB image and depth information of the material scene;

[0045] Step S2.2: preprocessing the acquired RGB image and depth information of the material scene to obtain the preprocessed RGB image and depth information of the material scene;

[0046] Step S2.3: The preprocessed RGB image of the material scene uses the SAM model to identify each material object and generate a binary mask;

[0047] Step S2.4: Calculate the depth information of only the material area based on the binary mask of each material object and the depth information of the preprocessed material scene;

[0048] Step S2.5: Based on the depth information of only the material area and the camera internal parameters, the point cloud information is calculated;

[0049]

[0050] Z = Depth(u,v)

[0051] Among them, f x and f y is the focal length of the camera in the x and y directions; c x and c y is the coordinate of the optical center of the image; (X, Y, Z) represents the point cloud coordinate point, Depth (u, v) represents the depth value at (u, v), which represents the distance from the camera to the surface of the object;

[0052] Step S2.6: Calculate the PPF feature of the current material point cloud.

[0053] Preferably, the step S2.2 comprises:

[0054] Step S2.2.1: adjusting the brightness distribution of the RGB image by histogram equalization so that the contrast of the adjusted RGB image meets the preset requirements;

[0055] Step S2.2.2: normalizing the depth information of each pixel to obtain normalized depth information;

[0056]

[0057] Among them, Z represents min Indicates the minimum depth value, Z max Indicates the maximum depth value;

[0058] Step S2.2.3: Mapping the G channel in the adjusted RGB pixel to the normalized depth information;

[0059] Step S2.2.4: Calculate the normal vector information of each pixel through the normalized depth information;

[0060]

[0061] Among them, u′ represents the image horizontal coordinate, and v′ represents the image vertical coordinate;

[0062] Step S2.2.5: Normalize the normal vector information and map it to the B channel of the adjusted RGB pixel.

[0063] Preferably, step S3 comprises:

[0064] Step S3.1: Select a reference point s from the surface of the object in the material scene r , and find the corresponding point m in the material model r ;

[0065] Step S3.2: Define the material model relative to the reference point s rThe local coordinate system of the material scene allows any reference point s to be selected r and find the corresponding point m in the material model r The position information and normal vector are aligned;

[0066] Step S3.3: Select the point pair (s r ,s i )∈S has similar feature vector F material model point pairs (m r , m i )∈M;

[0067] Step S3.4: Rotate the point m in the material model based on the rotation matrix i and point s in the material scene i Alignment, ensuring that the points of the material model and the points in the material scene are completely consistent in terms of geometric relationship;

[0068]

[0069] in, Represents the inverse transformation matrix that transforms the points in the global coordinate system back to the scene local coordinate system, which reverses the transformation from the scene to the global coordinate system and remaps the globally aligned points to the scene local coordinate system; R x (a) represents the rotation matrix of angle α around the x-axis, T m→g Represents the transformation matrix that transforms the material model points from the material model local coordinate system to the global coordinate system;

[0070] Step S3.5: Obtain the optimal local coordinate system through the generalized Hough voting method so that the material in the material scene matches the maximum number of points of the material model, thereby determining the posture of the material.

[0071] Preferably, the step S3.5 comprises:

[0072] Step S3.2.1: Define a two-dimensional accumulator array (N m , N angle ), where N m Represents the number of sampling points in the model, N angle represents the sampling step size of the rotation angle α;

[0073] Step S3.2.2: For the reference point s in the material scene r , select all the points in the scene except the reference point s r Other points outside the point to form a point pair (s r ,s i ), and calculate the feature F for each point pair s (s r ,s i ); the feature F s(s r ,s i ) includes the distance and relative direction between pairs of points;

[0074] Step S3.2.3: The calculated feature F s As the key value, search in the global material model and find the corresponding material model point pair (m r , m i ), so that they have similar characteristics;

[0075] Step S3.2.4: For each matched material model point pair (m r , m i ), calculate the rotation angle α required to align the material model with the material scene; for the calculated angle α, vote +1 in the corresponding cell in the accumulator array; when the whole process is completed, determine the optimal local coordinate system based on the position with the most votes in the accumulator array, and determine the best material model point and rotation angle combination (m r ,a), obtain the posture of the material in the scene.

[0076] According to the present invention, a disordered material recognition and positioning system based on 3D vision technology is provided, comprising:

[0077] Module M1: used to convert the 3D model of the material object into point cloud data, downsample the point cloud data, generate a template point cloud of the material object, and calculate the PPF feature of the template point cloud of the material object;

[0078] Module M2: used to segment materials in disorderly stacked material scenes based on the acquired RGB image and depth image of the material scene using the SAM model, obtain the material point cloud of each instance, and calculate the PPF features of the material point cloud in the material scene;

[0079] Module M3: used to match the PPF features of the material point cloud in the material scene with the PPF features of the template point cloud to achieve 6D pose estimation for each material.

[0080] Compared with the prior art, the present invention has the following beneficial effects:

[0081] 1. The present invention combines deep learning models with 3D vision technology to effectively improve the accuracy and efficiency of identifying disordered stacked materials, and is widely used in automated grasping and assembly tasks in complex environments such as warehousing and production lines;

[0082] 2. The present invention uses the SAM model to segment materials based on RGB data and depth data, significantly improving the segmentation effect of the SAM model;

[0083] 3. The present invention uses the SAM model to perform deep learning segmentation on the input RGB-D image and generate a binary mask for each material instance. By combining the depth data to obtain the depth information containing only the material area, the SAM model can be used to perform accurate instance segmentation on the materials in the complex disordered stacking scene and extract the independent point cloud information of each material.

[0084] 4. In order to improve the accuracy and efficiency of segmentation, the present invention adds image preprocessing before SAM model segmentation. By fusing the RGB image with the Depth image, the RGB information is retained while the depth information and geometric information are added. The RGB image is denoised and the color is enhanced, which effectively improves the segmentation effect of the SAM model.

[0085] 5. The present invention can adapt to a variety of materials and diversified application scenarios. The PPF feature only depends on geometric information and is applicable to objects of various shapes and sizes, without being restricted by specific material types. By dynamically constructing the PPF template, the present invention can be promoted and applied in various scenarios such as industry and logistics.

[0086] 6. The present invention can obtain accurate 6D pose information of the material through a single recognition, without relying on secondary positioning, and can directly carry out subsequent assembly tasks after being grasped by the robot;

[0087] 7. The present invention combines the voxel grid method with the downsampling technology based on geometric features. The voxel grid method is used to reduce redundant data in a large range, while the geometric feature downsampling ensures that the key geometric features of the material are retained by retaining points in important geometric feature areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0089] Figure 1 This is a flow chart of the disordered material recognition and positioning method based on 3D vision technology.

[0090] Figure 2 Comparison chart of the model point cloud before and after sampling.

[0091] Figure 3 Computes a schematic for a point pair feature of two oriented points.

[0092] Figure 4 This is the material segmentation effect diagram.

[0093] Figure 5 This is the point cloud segmentation effect diagram.

[0094] Figure 6Computes a schematic for transforming between model and scene coordinates.

[0095] Figure 7 This is the effect after matching the model point cloud with the scene point cloud. DETAILED DESCRIPTION

[0096] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0097] Example 1

[0098] According to the present invention, a method for identifying and locating disordered materials based on 3D vision technology is provided, such as Figure 1 As shown, including:

[0099] Step 1: Convert the 3D model of the material object into point cloud data, perform downsampling on the point cloud data, generate a template point cloud of the material object, and calculate the PPF feature of the template point cloud of the material object; in this embodiment, the 3D model includes a CAD and a Mesh model;

[0100] Specifically, the step 1 includes:

[0101] In this embodiment, the voxel grid method and the downsampling technology based on geometric features are combined. The voxel grid method is used to reduce redundant data in a large area, while the geometric feature downsampling ensures that the key geometric features of the material are retained by retaining points in important geometric feature areas;

[0102] More specifically, first, the point cloud space is divided into multiple voxels according to the preset voxel size, and the voxel size V is set. size , which determines the spatial range of each voxel. Larger voxels reduce the number of representative points but may lose some geometric details; smaller voxels retain more details but increase the amount of calculation. Then, within each voxel, the geometric center point within the voxel is selected as the representative point; in the voxel grid method, each point P of the point cloud data i =(x i ,y i , z i ) is mapped to the center of the voxel grid closest to it; for each point set S within a voxel j , where j represents the voxel number; the voxel center point C can be calculated by the following formula j :

[0103]

[0104] For all points selected by voxel sampling, calculate their normal vector n i , the normal vector reflects the geometric characteristics of the surface at that point. The calculation of the normal vector can be completed by the weighted average of the local neighborhood points; suppose the normal vector of each point is calculated by the following formula:

[0105]

[0106] Among them, N i Indicates point P i The neighborhood point set, n j is the neighborhood point P j The normal vector of |N i | is the number of neighborhood points.

[0107] Then calculate the curvature k of each point i , the curvature reflects the degree of curvature of the surface at the point; generally speaking, the curvature of the edge or corner of the material is larger, while the curvature of the flat area is smaller; through the curvature formula:

[0108]

[0109] Among them, r i It's point P i The radius of curvature at a point indicates the degree of curvature around that point.

[0110] According to the changes of normal vector and curvature, determine whether the point is a redundant point; specifically:

[0111] The change of the normal vector is obtained by calculating the angle between the normal vectors of adjacent points through cosθ=n1·n2 / (||n1||||n2||)); the importance of the point is judged by introducing a composite metric, S=α·|Δn|+β·κ, where α is the weight of the normal vector change, β is the curvature weight, and Δn is the change of the normal vector; in this embodiment, the above θ is the Δn here; by configuring different α and β, the filtering of redundant points of materials with different geometric structures can be handled.

[0112] By combining the voxel grid method with the downsampling technology based on geometric features, the points with rich geometric information in the 3D model can be retained as much as possible, and the downsampled material point cloud P = {p1, p2, ..., p n}, where p i Represents each point in the material point cloud. After downsampling, the point cloud data can reduce redundant data and improve calculation efficiency, such as Figure 2 The following is a comparison chart of the sample material before and after downsampling.

[0113] After sampling, the point cloud P is calculated for each point p i The normal vector N(pi ); The calculation of the normal vector is based on the local geometric structure of the point cloud, usually by finding the neighborhood points of the point and calculating the normal vector of its surface;

[0114] N(p i )=no rmal(p i )

[0115] Among them, normal means P i Point normal vector, calculated by P i Neighborhood points n i The normal vector of is obtained;

[0116]

[0117] Any two points in the material point cloud (p i , P j )’s point pair feature PPF (Point Pair Feature) is as follows Figure 3 As shown, specifically including:

[0118] The Euclidean distance d between two points 12 =|m1-m2|

[0119] P1P2 represents the Euclidean distance between points P1 and P2, and (X1, Y1, Z1) and (X2, Y2, Z2) are the coordinates of the two points respectively.

[0120] The angle between the vectors ∠(n1, n2)

[0121] The angle between the normal vector and the point pair vector is ∠(n1, d)∠(n2, d)

[0122] Therefore, the point pair information of the material point cloud can be expressed as:

[0123] F(m1, m2)=(||d||, ∠(n1, d), ∠(n2, d), ∠(n1, n2))

[0124] All calculated PPF features F(m1, m2) will be stored in the template database to form a complete material template T PPF .

[0125] T PPF ={F(m i , m j )|(m i , m j )∈M, i≠j}

[0126] At this point, the material template registration is complete.

[0127] Step 2: In the disorderly stacked material scene, based on the acquired RGB image and depth image of the material scene, the SAM model is used to segment the materials, obtain the material point cloud of each instance, and calculate the PPF feature of the material point cloud in the material scene;

[0128] Specifically, the step 2 includes:

[0129] Use the SAM model to segment the RGB-D image of the input scene, generate a binary mask for each material instance, and extract the corresponding material point cloud P according to the binary mask. input .

[0130] Specifically, through the RGB-D camera, the RGB image and depth image of the material scene can be obtained at the same time; the RGB image provides the color information of the scene, while the depth image provides the distance information from the object to the camera for each pixel, which provides the necessary data for the generation of point clouds.

[0131] The RGB image is preprocessed and fused with Depth information before being input into the SAM model. Specifically, the image brightness distribution is adjusted by histogram equalization to improve the image contrast so that the SAM model can better distinguish the background and the material. Then, RGB and Depth are fused in two ways.

[0132] Specifically, RGB and Depth are fused in two ways, including:

[0133] Pixel-level fusion: First, normalize the depth value of each pixel to the interval [0, 1], and use the following formula Then the G channel in the RGB pixel is mapped to the normalized Depth information, which can increase the depth information displayed in the RGB information.

[0134] Feature-level fusion: Calculate the normal vector information of each pixel through the depth image. When calculating the normal vector, the depth difference between adjacent pixels in the depth image can be used to approximate the normal vector of each pixel using the following formula: The normal vector information is normalized and mapped to the B channel in the RGB pixel, so that the geometric information in the image can be better displayed.

[0135] Through the above two preprocessing methods, RGB data and Depth data can be combined, retaining RGB information while adding depth information and geometric information. This can significantly improve the segmentation effect of the subsequent SAM model.

[0136] The SAM model is a segmentation model based on deep learning, which can classify images at the pixel level and identify different material instances in the image. In this embodiment, the SAM model processes the fused RGB image, identifies each material object and generates a binary mask for it. The mask is used to distinguish which parts are the areas of interest. In this embodiment, the areas of interest are the material objects, and which parts are the background or irrelevant areas. Figure 4 As shown, the root material is then segmented out.

[0137] According to the material mask information output by the SAM model and the depth data of the sample material collected, the depth information containing only the material area can be calculated. The point cloud information can be obtained through the camera internal parameters and the point cloud computing formula.

[0138] Camera internal parameters: where f x and f y is the focal length of the camera in the x and y directions, usually in pixels, c x and c y is the coordinate of the optical center of the image, usually the center of the image.

[0139] Calculate point cloud:

[0140]

[0141] Z = Depth(u,v)

[0142] Where (u, v) is the position of the pixel in the image (usually a two-dimensional pixel coordinate). Z = Depth (u, v) is the depth value of the pixel, indicating the distance from the camera to the surface of the object. The material point cloud data is finally obtained as follows: Figure 5 shown.

[0143] The SAM model can accurately segment materials in complex and disordered stacking scenes and extract independent point clouds for each material.

[0144] Calculate PPF features for material point cloud;

[0145] For each segmented material point cloud P input , repeat the normal calculation and point pair feature PPF calculation to obtain the PPF feature set T of the material point cloud input .

[0146] This process is consistent with building a material template to generate the PPF features of the material point cloud.

[0147] Step 3: Match the PPF features of the material point cloud in the material scene with the PPF features of the template point cloud to achieve 6D pose estimation for each material;

[0148] Step S3.1: Define the material model relative to any reference point s on the surface of the object in the scene r The local coordinate system is used to align the material model and the scene in local features; the material model and the material in the scene are completely consistent in geometric relationship based on the rotation matrix;

[0149] Specifically, the step 3.1 includes:

[0150] Local coordinate system definition: Select a reference point sr from the scene and assume it is on the surface of the object. Then find a corresponding point m in the model r The purpose of this process is to match the points in the scene with the points in the model. Next, we need to ensure that the position information and normal vectors of these two points are aligned, that is, to align the surface normal direction of the model with the normal direction in the scene. Then the rigid body transformation from model space to scene space can be achieved by combining the reference point in the model with the rotation angle (m r , α) to explain, defined as the model relative to the reference point s r Local Coordinates: This part deals with the position and direction issues through the definition of local coordinates, so that the model and scene can be aligned on specific local features.

[0151] Model-to-scene transformation: Given a reference point s r , select the point pair (s r ,s i )∈S has a similar eigenvector F. Here, the eigenvector F describes the “relative distance” and “relative direction” between two points, that is, relative to the model point pair (m r , m i )∈M. This feature is used to find point pairs in the scene that are similar to the corresponding relationship in the model for matching.

[0152] Specifically, through the transformation matrix T m→g , point m in the model r Move to the origin of the local coordinate system. At the same time, rotate the model so that its normal vector Align to the x-axis direction of the local coordinate system. The purpose of this step is to normalize the reference point in the model so that it can be used as a reference for subsequent alignment.

[0153] Through the corresponding transformation matrix T s→g, perform similar processing on the points in the scene, align the reference point in the scene to the origin of the local coordinate system, and ensure that the direction is consistent with the normal vector of the model. This step is to unify the model and the scene under the same benchmark for comparison.

[0154] Finally, another point m in the model i Rotate along the x-axis so that it is aligned with point s in the scene i Align. This rotation is given by the rotation matrix R x (α) is completed, and its purpose is to ensure that the points in the model and the points in the scene are completely consistent in geometric relationship.

[0155] Through the above steps, the transformation from model to scene can be finally defined as like Figure 6 As shown, where T m→g Represents the transformation matrix that transforms the model points from the model local coordinate system to the global coordinate system. This transformation includes translation and rotation so that the model points are aligned to the global coordinate system. x (α) represents the rotation matrix of angle α around the x-axis. Represents the inverse transformation matrix that transforms points in the global coordinate system back to the scene's local coordinate system. It reverses the transformation from the scene to the global coordinate system, remapping the globally aligned points back to the scene's local coordinate system.

[0156] Step 3.2: Calculate the object pose. Use the generalized Hough voting method to find the optimal local coordinate system so that the model part in the scene matches the maximum number of points of the model itself, thereby obtaining the pose of the object; this is to determine the optimal position and pose of the model in the scene. After defining the local coordinate system, a two-dimensional accumulator array is used, where the number of rows N m Indicates the number of sampling points in the model; the number of columns N angle Represents the sampling step size of the rotation angle α.

[0157] This accumulator array is similar to a two-dimensional table, where each row represents a point in the model and each column represents a different rotation angle. r , select other points in the scene to form point pairs (s r ,s i ), and calculate the feature F for each point pair s (s r ,s i ). This feature includes the distance and relative direction between point pairs, and the purpose is to find geometric features in the scene that can match the model.

[0158] The calculated feature Fs As the key value (key), search in the global model description to find the corresponding model point pair (m r , m i ) so that they have similar features (including distance and normal vector).

[0159] For each matched model point pair (m r , m i ), through the previous formula Calculate the rotation angle α required to align the model with the scene. This rotation angle represents the rotation that the model needs to make along a certain axis in order to match the scene point pair. For the calculated angle a, the corresponding cell in the accumulator array "votes" (+1). This voting process is similar to accumulating the number of matches in a two-dimensional table to count which position and rotation angle combination best matches all points. When the entire process is completed, the position with the most votes in the accumulator array represents the best model point and rotation angle combination (m r , α), which is also the optimal local coordinate system. This combination can be used to represent the posture of the object in the scene.

[0160] In this way, the best 6D pose of the material in the current scene is determined, that is, the rotation matrix R and the translation vector t, where the translation vector t is the difference between the XYZ coordinates of the model midpoint and the scene midpoint. This enables accurate recognition of the material pose. Figure 7 As shown in the figure, the point cloud visualization effect diagram after the input material A is matched with the template point cloud after downsampling of the material 3D model.

[0161] Example 2

[0162] Embodiment 2 is a preferred embodiment of Embodiment 1

[0163] According to the present invention, a disordered material recognition and positioning system based on 3D vision technology is provided, comprising:

[0164] Module 1: converting the 3D model of the material object into point cloud data, downsampling the point cloud data, generating a template point cloud of the material object, and calculating the PPF feature of the template point cloud of the material object; in this embodiment, the 3D model includes a CAD and a Mesh model;

[0165] Specifically, the module 1 includes:

[0166] In this embodiment, the voxel grid method and the downsampling technology based on geometric features are combined. The voxel grid method is used to reduce redundant data in a large area, while the geometric feature downsampling ensures that the key geometric features of the material are retained by retaining points in important geometric feature areas;

[0167] More specifically, first, the point cloud space is divided into multiple voxels according to the preset voxel size, and the voxel size V is set. size , which determines the spatial range of each voxel. Larger voxels reduce the number of representative points but may lose some geometric details; smaller voxels retain more details but increase the amount of calculation. Then, within each voxel, the geometric center point within the voxel is selected as the representative point; in the voxel grid method, each point P of the point cloud data i =(x i ,y i , z i ) is mapped to the center of the voxel grid closest to it; for each point set S within a voxel j , where j represents the voxel number; the voxel center point C can be calculated by the following formula j :

[0168]

[0169] For all points selected by voxel sampling, calculate their normal vector n i , the normal vector reflects the geometric characteristics of the surface at that point. The calculation of the normal vector can be completed by the weighted average of the local neighborhood points; suppose the normal vector of each point is calculated by the following formula:

[0170]

[0171] Among them, N i Indicates point P i The neighborhood point set, n j is the neighborhood point P j The normal vector of |N i | is the number of neighborhood points.

[0172] Then calculate the curvature k of each point i , the curvature reflects the degree of curvature of the surface at the point; generally speaking, the curvature of the edge or corner of the material is larger, while the curvature of the flat area is smaller; through the curvature formula:

[0173]

[0174] Among them, r i It's point P i The radius of curvature at a point indicates the degree of curvature around that point.

[0175] According to the changes of normal vector and curvature, determine whether the point is a redundant point; specifically:

[0176] The change of the normal vector is obtained by calculating the angle between the normal vectors of adjacent points through cosθ=n1·n2 / (||n1||||n2||)); the importance of the point is judged by introducing a composite metric, S=α·|Δn|+β·κ, where α is the weight of the normal vector change, β is the curvature weight, and Δn is the change of the normal vector; in this embodiment, the above θ is the Δn here; by configuring different α and β, the filtering of redundant points of materials with different geometric structures can be handled.

[0177] By combining the voxel grid method with the downsampling technology based on geometric features, the points with rich geometric information in the 3D model can be retained as much as possible, and the downsampled material point cloud P = {p1, p2, ..., pn} is obtained, where pi represents each point in the material point cloud. After downsampling, the point cloud data can reduce redundant data and improve computational efficiency, such as Figure 2 The following is a comparison chart of the sample material before and after downsampling.

[0178] After sampling the point cloud P, calculate the normal vector N(pi) of each point pi; the calculation of the normal vector is based on the local geometric structure of the point cloud, usually by finding the neighborhood points of the point and calculating the normal vector of its surface;

[0179] N(p i )=no rmal(p i )

[0180] Among them, normal means P i Point normal vector, calculated by P i Neighborhood points n i The normal vector of is obtained;

[0181]

[0182] Any two points (pi, p j )’s point pair feature PPF (Point Pair Feature) is as follows Figure 3 As shown, specifically including:

[0183] The Euclidean distance d between two points 12 =|m1-m2|

[0184] P1P2 represents the Euclidean distance between points P1 and P2, and (X1, Y1, Z1) and (X2, Y2, Z2) are the coordinates of the two points respectively.

[0185] The angle between the vectors ∠(n1, n2)

[0186] The angle between the normal vector and the point pair vector is ∠(n1, d)∠(n2, d)

[0187] Therefore, the point pair information of the material point cloud can be expressed as:

[0188] F(m1, m2)=(||d||, ∠(n1, d), ∠(n2, d), ∠(n1, n2))

[0189] All calculated PPF features F(m1,m2) will be stored in the template database to form a complete material template T PPF .

[0190] T PPF ={F(m i , m j )|(m i , m j )∈M, i≠j}

[0191] At this point, the material template registration is complete.

[0192] Module 2: In the disorderly stacked material scene, based on the acquired RGB image and depth image of the material scene, the SAM model is used to segment the materials, obtain the material point cloud of each instance, and calculate the PPF features of the material point cloud in the material scene;

[0193] Specifically, the module 2 includes:

[0194] Use the SAM model to segment the RGB-D image of the input scene, generate a binary mask for each material instance, and extract the corresponding material point cloud P according to the binary mask. input .

[0195] Specifically, through the RGB-D camera, the RGB image and depth image of the material scene can be obtained at the same time; the RGB image provides the color information of the scene, while the depth image provides the distance information from the object to the camera for each pixel, which provides the necessary data for the generation of point clouds.

[0196] The RGB image is preprocessed and fused with Depth information before being input into the SAM model. Specifically, the image brightness distribution is adjusted by histogram equalization to improve the image contrast so that the SAM model can better distinguish the background and the material. Then, RGB and Depth are fused in two ways.

[0197] Specifically, RGB and Depth are fused in two ways, including:

[0198] Pixel-level fusion: First, normalize the depth value of each pixel to the interval [0, 1], and use the following formula Then the G channel in the RGB pixel is mapped to the normalized Depth information, which can increase the depth information displayed in the RGB information.

[0199] Feature-level fusion: Calculate the normal vector information of each pixel through the depth image. When calculating the normal vector, the depth difference between adjacent pixels in the depth image can be used to approximate the normal vector of each pixel using the following formula: The normal vector information is normalized and mapped to the B channel in the RGB pixel, so that the geometric information in the image can be better displayed.

[0200] Through the above two preprocessing methods, RGB data and Depth data can be combined, retaining RGB information while adding depth information and geometric information. This can significantly improve the segmentation effect of the subsequent SAM model.

[0201] The SAM model is a segmentation model based on deep learning, which can classify images at the pixel level and identify different material instances in the image. In this embodiment, the SAM model processes the fused RGB image, identifies each material object and generates a binary mask for it. The mask is used to distinguish which parts are the areas of interest. In this embodiment, the areas of interest are the material objects, and which parts are the background or irrelevant areas. Figure 4 As shown, the root material is then segmented out.

[0202] According to the material mask information output by the SAM model and the depth data of the sample material collected, the depth information containing only the material area can be calculated. The point cloud information can be obtained through the camera internal parameters and the point cloud computing formula.

[0203] Camera internal parameters: where f x and f y is the focal length of the camera in the x and y directions, usually in pixels, c x and c y is the coordinate of the optical center of the image, usually the center of the image.

[0204] Calculate point cloud:

[0205]

[0206] Z = Depth(u,v)

[0207] Where (u, v) is the position of the pixel in the image (usually a two-dimensional pixel coordinate). Z = Depth (u, v) is the depth value of the pixel, indicating the distance from the camera to the surface of the object. The material point cloud data is finally obtained as follows: Figure 5 shown.

[0208] The SAM model can accurately segment materials in complex and disordered stacking scenes and extract independent point clouds for each material.

[0209] Calculate PPF features for material point cloud;

[0210] For each segmented material point cloud P input , repeat the normal calculation and point pair feature PPF calculation to obtain the PPF feature set T of the material point cloud input .

[0211] This process is consistent with building a material template to generate the PPF features of the material point cloud.

[0212] Module 3: Match the PPF features of the material point cloud in the material scene with the PPF features of the template point cloud to achieve 6D pose estimation for each material;

[0213] Module S3.1: Define the material model relative to any reference point s on the surface of an object in the scene r The local coordinate system is used to align the material model and the scene in local features; the material model and the material in the scene are completely consistent in geometric relationship based on the rotation matrix;

[0214] Specifically, the module 3.1 includes:

[0215] Local coordinate system definition: select a reference point s from the scene r , and assume it is on the surface of the object. Then find a corresponding point m in the model r The purpose of this process is to match the points in the scene with the points in the model. Next, we need to ensure that the position information and normal vectors of these two points are aligned, that is, to align the surface normal direction of the model with the normal direction in the scene. Then the rigid body transformation from model space to scene space can be achieved by combining the reference point in the model with the rotation angle (m r , α) to explain, defined as the model relative to the reference point s r Local Coordinates: This part deals with the position and direction issues through the definition of local coordinates, so that the model and scene can be aligned on specific local features.

[0216] Model-to-scene transformation: Given a reference point s r , select the point pair (s r ,s i)∈S has a similar eigenvector F. Here, the eigenvector F describes the “relative distance” and “relative direction” between two points, that is, relative to the model point pair (m r , m i )∈M. This feature is used to find point pairs in the scene that are similar to the corresponding relationship in the model for matching.

[0217] Specifically, through the transformation matrix T m→g , point m in the model r Move to the origin of the local coordinate system. At the same time, rotate the model so that its normal vector Align to the x-axis direction of the local coordinate system. The purpose of this step is to normalize the reference point in the model so that it can be used as a reference for subsequent alignment.

[0218] Through the corresponding transformation matrix T s→g , perform similar processing on the points in the scene, align the reference point in the scene to the origin of the local coordinate system, and ensure that the direction is consistent with the normal vector of the model. This step is to unify the model and the scene under the same benchmark for comparison.

[0219] Finally, another point m in the model i Rotate along the x-axis so that it is aligned with point s in the scene i Align. This rotation is given by the rotation matrix R x (α) is completed, and its purpose is to ensure that the points in the model and the points in the scene are completely consistent in geometric relationship.

[0220] Through the above modules, the transformation from model to scene can be finally defined as like Figure 6 As shown, where T m→g Represents the transformation matrix that transforms the model points from the model local coordinate system to the global coordinate system. This transformation includes translation and rotation so that the model points are aligned to the global coordinate system. x (α) represents the rotation matrix of angle α around the x-axis. Represents the inverse transformation matrix that transforms points in the global coordinate system back to the scene's local coordinate system. It reverses the transformation from the scene to the global coordinate system, remapping the globally aligned points back to the scene's local coordinate system.

[0221] Module 3.2: Calculate the pose of the object. The generalized Hough voting method is used to find the optimal local coordinate system so that the model part in the scene matches the maximum number of points of the model itself, thereby obtaining the pose of the object; this is to determine the optimal position and pose of the model in the scene. After defining the local coordinate system, a two-dimensional accumulator array is used, where the number of rows Nm Indicates the number of sampling points in the model; the number of columns N angle Represents the sampling step size of the rotation angle α.

[0222] This accumulator array is similar to a two-dimensional table, where each row represents a point in the model and each column represents a different rotation angle. r , select other points in the scene to form point pairs (s r ,s i ), and calculate the feature F for each point pair s (s r ,s i ). This feature includes the distance and relative direction between point pairs, and the purpose is to find geometric features in the scene that can match the model.

[0223] The calculated feature F s As the key value (key), search in the global model description to find the corresponding model point pair (m r , m i ) so that they have similar features (including distance and normal vector).

[0224] For each matched model point pair (m r , m i ), through the previous formula Calculate the rotation angle α required to align the model with the scene. This rotation angle represents the rotation that the model needs to make along a certain axis in order to match the scene point pair. For the calculated angle α, the corresponding cell in the accumulator array "votes" (+1). This voting process is similar to accumulating the number of matches in a two-dimensional table to count which position and rotation angle combination best matches all points. When the entire process is completed, the position with the most votes in the accumulator array represents the best model point and rotation angle combination (m r , α), which is also the optimal local coordinate system. This combination can be used to represent the posture of the object in the scene.

[0225] In this way, the best 6D pose of the material in the current scene is determined, that is, the rotation matrix R and the translation vector t, where the translation vector t is the difference between the XYZ coordinates of the model midpoint and the scene midpoint. This enables accurate recognition of the material pose. Figure 7 As shown in the figure, the point cloud visualization effect diagram after the input material A is matched with the template point cloud after downsampling of the material 3D model.

[0226] This embodiment can quickly and accurately identify the 6D pose of each material in a disorderly stacked material scene, thereby realizing automated material grasping and assembly.

[0227] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0228] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for identifying and locating disordered materials based on 3D vision technology, characterized in that: include: Step S1: converting the 3D model of the material object into point cloud data, downsampling the point cloud data, generating a template point cloud of the material object, and calculating the PPF feature of the template point cloud of the material object; Step S2: In the disorderly stacked material scene, based on the acquired RGB image and depth image of the material scene, the SAM model is used to segment the materials, obtain the material point cloud of each instance, and calculate the PPF feature of the material point cloud in the material scene; Step S3: Match the PPF features of the material point cloud in the material scene with the PPF features of the template point cloud to obtain the 6D pose of each material, and realize the recognition and positioning of disordered materials through a single matching.

2. The method for identifying and locating disordered materials based on 3D vision technology according to claim 1, characterized in that: The step S1 comprises: Step S1.1: converting the 3D model of the material object into point cloud data; Step S1.2: downsampling the point cloud data using a voxel grid method to obtain downsampled point cloud data; Step S1.3: downsampling the downsampled point cloud data through a downsampling technique based on geometric features to obtain processed point cloud data and generate a template point cloud of the material object; Step S1.4: Calculate the PPF features of the material object template point cloud based on the processed point cloud data.

3. The method for identifying and locating disordered materials based on 3D vision technology according to claim 2 is characterized in that: The step S1.2 comprises: Step S1.2.1: dividing the point cloud data of the material object into a plurality of voxels according to a preset voxel size; Step S1.2.2: In each voxel, the geometric center point in the voxel is selected as the representative point to implement downsampling processing; Among them, S j represents the set of points within each voxel, (x i ,y i ,z i ) represents the spatial coordinates of each point within the voxel.

4. The method for identifying and locating disordered materials based on 3D vision technology according to claim 2 is characterized in that: The step S1.3 comprises: Step S1.3.1: Calculate the normal vector of each point based on all points selected by the voxel grid method; Among them, N i Indicates point P i The neighborhood point set, n j is the neighborhood point P j The normal vector of |N i | is the number of neighborhood points; Step S1.3.2: Based on all points selected by the voxel grid method, calculate the curvature k of each point i ; Among them, r i It's point P i The radius of curvature at , which indicates the degree of curvature around the point; Step S1.3.3: According to the changes of the normal vector and the curvature, determine whether the point is a redundant point, delete the redundant points, retain the points that meet the preset requirements, and implement downsampling processing; The change of the normal vector is obtained by calculating the angle between the normal vectors of adjacent points: cosθ=n1·n2(||n1||n2||)) The composite metric is calculated based on the change of the normal vector and the curvature, and compared with the set threshold. When the composite metric is greater than the set threshold, it is judged as a redundant point. S=α·|Δn|+β·κ Among them, α represents the weight of the normal vector change; β represents the curvature weight; Δn represents the change of the normal vector; and k represents the curvature.

5. The method for identifying and locating disordered materials based on 3D vision technology according to claim 2, characterized in that: The step S1.4 comprises: Step S1.4.1: For the downsampled point cloud, calculate each point p i The normal vector N(p i ); N(p i )=normal(p i ) Among them, normal means P i Point normal vector, calculated by P i Neighborhood points n i The normal vector of is obtained; Step S1.4.2: For the downsampled point cloud, calculate any two points (p i , p j )’s point pair feature PPF; F(m1, m2)=(||d||, ∠(n1, d), ∠(n2, d), ∠(n1, n2)) T PPF ={F(m i ,m j )|(m i ,m j )∈M,i≠j} Among them, F(m1, m2) represents any two points (p i , p j )’s point pair feature PPF; d represents the Euclidean distance between two points; ∠(n1, n2) represents the angle between vectors; ∠(n1, d)∠(n2, d) represents the angle between the normal vector and the point pair vector; M represents a point on the model.

6. The method for identifying and locating disordered materials based on 3D vision technology according to claim 1, characterized in that: The step S2 comprises: Step S2.1: Based on the RGB-D camera, obtain the RGB image and depth information of the material scene; Step S2.2: preprocessing the acquired RGB image and depth information of the material scene to obtain the preprocessed RGB image and depth information of the material scene; Step S2.3: The preprocessed RGB image of the material scene uses the SAM model to identify each material object and generate a binary mask; Step S2.4: Calculate the depth information of only the material area based on the binary mask of each material object and the depth information of the preprocessed material scene; Step S2.5: Based on the depth information of only the material area and the camera internal parameters, the point cloud information is calculated; Z=Depth(u,v) Among them, f x and f y is the focal length of the camera in the x and y directions; c x and c y is the coordinate of the optical center of the image; (X, Y, Z) represents the point cloud coordinate point, Depth (u, v) represents the depth value at (u, v), which represents the distance from the camera to the surface of the object; Step S2.6: Calculate the PPF feature of the current material point cloud.

7. The method for identifying and locating disordered materials based on 3D vision technology according to claim 6, characterized in that: The step S2.2 comprises: Step S2.2.1: adjusting the brightness distribution of the RGB image by histogram equalization so that the contrast of the adjusted RGB image meets the preset requirements; Step S2.2.2: normalizing the depth information of each pixel to obtain normalized depth information; Among them, Z represents min Indicates the minimum depth value, Z max Indicates the maximum depth value; Step S2.2.3: Mapping the G channel in the adjusted RGB pixel to the normalized depth information; Step S2.2.4: Calculate the normal vector information of each pixel through the normalized depth information; Among them, u′ represents the image horizontal coordinate, and v′ represents the image vertical coordinate; Step S2.2.5: Normalize the normal vector information and map it to the B channel of the adjusted RGB pixel.

8. The method for identifying and locating disordered materials based on 3D vision technology according to claim 1, characterized in that: The step S3 comprises: Step S3.1: Select a reference point s from the surface of the object in the material scene r , and find the corresponding point m in the material model r ; Step S3.2: Define the material model relative to the reference point s r The local coordinate system of the material scene allows any reference point s to be selected r and find the corresponding point m in the material model r The position information and normal vector are aligned; Step S3.3: Select the point that matches the material scene r ,s i )∈S has similar feature vector F material model point pairs (m r , m i )∈M; Step S3.4: Rotate the point m in the material model based on the rotation matrix i and point s in the material scene i Alignment, ensuring that the points of the material model and the points in the material scene are completely consistent in terms of geometric relationship; in, Represents the inverse transformation matrix that transforms the points in the global coordinate system back to the scene local coordinate system, which reverses the transformation from the scene to the global coordinate system and remaps the globally aligned points to the scene local coordinate system; R x (a) represents the rotation matrix of angle α around the x-axis, T m→g Represents the transformation matrix that transforms the material model points from the material model local coordinate system to the global coordinate system; Step S3.5: Obtain the optimal local coordinate system through the generalized Hough voting method so that the material in the material scene matches the maximum number of points of the material model, thereby determining the posture of the material.

9. The method for identifying and locating disordered materials based on 3D vision technology according to claim 8, characterized in that: The step S3.5 comprises: Step S3.2.1: Define a two-dimensional accumulator array (N m ,N angle ), where N m Represents the number of sampling points in the model, N angle represents the sampling step size of the rotation angle α; Step S3.2.2: For the reference point s in the material scene r , select all the points in the scene except the reference point s r Other points outside the r ,s i ), and calculate the feature F for each point pair s (s r ,s i ); the feature F s (s r ,s i ) includes the distance and relative direction between pairs of points; Step S3.2.3: The calculated feature F s As the key value, search in the global material model and find the corresponding material model point pair (m r , m i ), so that they have similar characteristics; Step S3.2.4: For each matched material model point pair (m r , m i ), calculate the rotation angle α required to align the material model with the material scene; for the calculated angle α, vote +1 in the corresponding cell in the accumulator array; when the whole process is completed, determine the optimal local coordinate system based on the position with the most votes in the accumulator array, and determine the best material model point and rotation angle combination (m r ,a), obtain the posture of the material in the scene.

10. A disordered material recognition and positioning system based on 3D vision technology, characterized in that: include Module M1: used to convert the 3D model of the material object into point cloud data, downsample the point cloud data, generate a template point cloud of the material object, and calculate the PPF feature of the template point cloud of the material object; Module M2: used to segment materials in disorderly stacked material scenes based on the acquired RGB image and depth image of the material scene using the SAM model, obtain the material point cloud of each instance, and calculate the PPF features of the material point cloud in the material scene; Module M3: used to match the PPF features of the material point cloud in the material scene with the PPF features of the template point cloud, obtain the 6D pose of each material, and realize the recognition and positioning of disordered materials through a single matching.

Images