A method and system for grasping based on structured light three-dimensional machine vision guidance

Abstract

This invention discloses a grasping method and system based on structured light 3D machine vision guidance. The method includes: acquiring a target point cloud of a target object; acquiring a 2D image of a work scene containing the target object, and then determining a 3D point cloud of the work scene based on the depth information of the 2D image; performing 3D point-pair feature template matching on the 3D point cloud and the target point cloud to obtain a transformation matrix from the target point cloud to the 3D point cloud; determining the first pose information of the target object based on the transformation matrix, and combining it with the second pose information of the grasping device, determining the grasping target through Monte Carlo tree search, and then executing the grasping process. This invention can improve the speed of object recognition by structured light 3D machine vision and increase grasping efficiency. The embodiments of this invention can efficiently guide object grasping and can be widely applied in the field of data processing technology.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a grasping method and system based on structured light 3D machine vision guidance. Background Technology

[0002] The structured light 3D machine vision-guided grasping system is an advanced system applied in the field of robotics. By combining structured light projection and 3D machine vision technology, it achieves high-precision target recognition and grasping operations. This system has broad application prospects in fields such as automated production and warehousing logistics. The technology utilizes a structured light projector to project coded light onto a target object and a camera to capture changes in light on the object's surface, thereby reconstructing the object's 3D shape. Through 3D machine vision technology, accurate recognition, shape reconstruction, and position localization of the target object can be achieved. It can also obtain high-precision shape and position information of the target object, providing precise guidance for subsequent grasping operations.

[0003] However, existing structured light 3D machine vision grasping systems still face some challenges in adapting to various environments. For example, ambient light interference can affect the system's accuracy and stability during structured light projection. Ambient light interference can cause changes in light intensity or noise, thus affecting the shape reconstruction and recognition of the target object. Furthermore, prior to the grasping process, the analysis of 3D point clouds typically requires processing large amounts of 3D point cloud data and performing complex algorithmic calculations and analyses. This results in slower system processing speeds, making it difficult to meet the requirements of real-time grasping operations. Especially in high-speed grasping, dynamic scenes, or rapidly changing environments, the system needs even faster processing speeds and real-time performance to ensure the success and stability of the grasping operation. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a grasping method and system based on structured light 3D machine vision guidance, which can efficiently guide object grasping.

[0005] On one hand, embodiments of the present invention provide a grasping method guided by structured light 3D machine vision, comprising:

[0006] Obtain the target point cloud of the target object;

[0007] Acquire a two-dimensional image of the work scene containing the target object, and then determine the three-dimensional point cloud of the work scene based on the depth information of the two-dimensional image;

[0008] Perform 3D point-pair feature template matching on the 3D point cloud and the target point cloud to obtain the transformation matrix from the target point cloud to the 3D point cloud;

[0009] The first pose information of the target object is determined based on the transformation matrix. Combined with the second pose information of the grasping device, the grasping target is determined by Monte Carlo tree search, and then the grasping process is executed.

[0010] Optionally, the target point cloud of the target object is acquired, including:

[0011] Acquire the first point cloud data of the target object from a structured light camera;

[0012] Remove point clouds that are irrelevant to the target object from the first point cloud data to obtain the target point cloud.

[0013] Optionally, a two-dimensional image of the work scene containing the target object is acquired, and then a three-dimensional point cloud of the work scene is determined based on the depth information of the two-dimensional image, including:

[0014] Acquire two-dimensional images of the work scene containing the target object from a structured light camera;

[0015] Based on the principle of structured light 3D imaging, the depth information of the 2D image is determined and a depth map is generated;

[0016] Based on the depth map, generate the second point cloud data of the work scene;

[0017] The second point cloud data is preprocessed to obtain a 3D point cloud of the working scene; the image preprocessing includes downsampling and filtering for noise reduction.

[0018] Optionally, 3D point-pair feature template matching is performed on the 3D point cloud and the target point cloud to obtain a transformation matrix from the target point cloud to the 3D point cloud, including:

[0019] Based on the relationship between the relative position and normal vector of each point in the point cloud and other points in the point cloud, the point pair features of each point in the point cloud are determined; wherein, the point pair features include the first point pair features of each point in the 3D point cloud and the second point pair features of each point in the target point cloud;

[0020] Vector pairing is performed on each first point pair feature and each second point pair feature to obtain the matching score between each point in the 3D point cloud and the target point cloud.

[0021] Based on the sum of the matching scores between each point in the target point cloud and all points in the 3D point cloud, the target matching point in the 3D point cloud for each point in the target point cloud is determined, and the matching point pair between the target point cloud and the 3D point cloud is obtained.

[0022] Based on the matching point pairs, construct the rotation matrix and translation vector;

[0023] Singular value decomposition is performed on the rotation matrix and translation vector, and the target rotation matrix and target translation vector are obtained through convergent iteration.

[0024] The transformation matrix from the target point cloud to the 3D point cloud is determined based on the target rotation matrix and the target translation vector.

[0025] Optionally, based on the relationship between the relative position and normal vector of each point in the point cloud and other points in the point cloud, the point pair features of each point in the point cloud are determined, including:

[0026] Based on the relative positions between point i and point j in the point cloud, the relative position vector is determined; and the first normal vector of point i and the second normal vector of point j are obtained; then the first angle between the relative position vector and the first normal vector is determined, and the second angle between the first normal vector and the second normal vector is determined.

[0027] Based on the relative position vector, the first included angle, and the second included angle, determine the point pair characteristics of point i and point j;

[0028] The expression for the point-pair feature is:

[0029] f ij =(v ij (cosθ,sinθ,cosφ,sinφ)

[0030] In the formula, f ij V represents the point pair features of points i and j. ij Let θ represent the relative position vector between points i and j, φ represent the first included angle, and φ represent the second included angle.

[0031] Optionally, singular value decomposition is performed on the rotation matrix and translation vector, and the target rotation matrix and target translation vector are obtained through convergent iteration, including:

[0032] The first centroid is determined based on the average sum of all points in the target point cloud, and the second centroid is determined based on the average sum of all points in the 3D point cloud.

[0033] Based on the first and second centroids, singular value decomposition is performed on the rotation matrix and translation vector, and the objective function is obtained by simplification based on the decomposition results.

[0034] The objective function is expressed as follows:

[0035]

[0036] In the formula, q′ i =q i -μ q q i Let i and μ represent points in a 3D point cloud. q p′ represents the second mass center; i =p i -μ p p i Let i and μ represent points in the target point cloud.p R represents the first centroid; R represents the rotation matrix; t represents the translation vector;

[0037] Based on the objective function, the target rotation matrix and target translation vector are obtained through convergent iteration.

[0038] Optionally, the crawling target is determined using the Monte Carlo tree search method, including:

[0039] The second pose information is used as the root node, and the first pose information is used as the child node;

[0040] Expand the child nodes and determine the target node based on the expansion results;

[0041] The capture value of a target node is determined based on the spatial information of its capture and placement.

[0042] The crawling value is evaluated through heuristic functions, thereby obtaining updated information;

[0043] In the path of the expansion of child nodes, the update information is propagated back and the node information is updated; each node of the expansion result is taken as the target node in turn, and then the step of determining the crawling value of the target node based on the spatial information of the target node crawling and placement is returned until the preset number of searches or time limit is reached.

[0044] The crawling target is determined based on the crawling value of each node after the update.

[0045] On the other hand, embodiments of the present invention provide a grasping system guided by structured light 3D machine vision, comprising:

[0046] The first module is used to acquire the target point cloud of the target object;

[0047] The second module is used to acquire a two-dimensional image of the work scene containing the target object, and then determine the three-dimensional point cloud of the work scene based on the depth information of the two-dimensional image.

[0048] The third module is used to perform 3D point-pair feature template matching between the 3D point cloud and the target point cloud to obtain the transformation matrix from the target point cloud to the 3D point cloud.

[0049] The fourth module is used to determine the first pose information of the target object based on the transformation matrix, and combined with the second pose information of the grasping device, to determine the grasping target through the Monte Carlo tree search method, and then execute the grasping process.

[0050] On the other hand, embodiments of the present invention provide an electronic device, including a processor and a memory;

[0051] Memory is used to store programs;

[0052] The processor executes the program as described above.

[0053] On the other hand, embodiments of the present invention provide a computer-readable storage medium storing a program that is executed by a processor to implement the method described above.

[0054] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0055] This invention first acquires a target point cloud of the target object; then acquires a 2D image of the work scene containing the target object, and determines a 3D point cloud of the work scene based on the depth information of the 2D image; performs 3D point-pair feature template matching on the 3D point cloud and the target point cloud to obtain a transformation matrix from the target point cloud to the 3D point cloud; based on the transformation matrix, determines the first pose information of the target object, and combines it with the second pose information of the grasping device, uses Monte Carlo tree search to determine the grasping target, and then executes the grasping process. This invention, based on the target point cloud of the target object and the 3D point cloud of the target object's work scene, determines the transformation matrix through feature template matching, which can improve the speed of structured light 3D machine vision object recognition. Furthermore, combining this with Monte Carlo tree search to determine the grasping target from the target object can improve grasping efficiency. This invention can efficiently guide object grasping. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 A flowchart illustrating a grasping method based on structured light 3D machine vision guidance provided in an embodiment of the present invention;

[0058] Figure 2 A schematic diagram of the overall process of the grasping method based on structured light 3D machine vision guidance provided in an embodiment of the present invention;

[0059] Figure 3 A schematic diagram illustrating the matching effect of the traditional PPF algorithm provided in an embodiment of the present invention;

[0060] Figure 4A schematic diagram illustrating the matching effect of the improved PPF algorithm provided in an embodiment of the present invention;

[0061] Figure 5 A flowchart illustrating the MCTS algorithm crawling strategy provided in this embodiment of the invention;

[0062] Figure 6 A schematic diagram of the voting score of the crawling object and the crawling pose depth provided in the embodiments of the present invention.

[0063] Figure 7 A schematic diagram of a grasping system guided by structured light 3D machine vision provided in an embodiment of the present invention;

[0064] Figure 8 This is a schematic diagram of the frame of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0066] First, it's important to explain the working principle of the structured light 3D machine vision-guided grasping system as follows: First, a structured light projector projects coded light rays onto the target object, forming textures or grids, and a camera captures the changes in light on the object's surface. Then, the system uses 3D machine vision algorithms to process the captured light information, reconstructing the 3D shape and position of the target object. Finally, based on the reconstructed information, the system performs target recognition and grasping planning, achieving accurate grasping of the target object.

[0067] The structured light 3D machine vision-guided grasping system has broad application prospects. In the field of automated production, this system can be used for the grasping and positioning of parts on automated assembly lines, improving production efficiency and quality. In the field of warehousing and logistics, this system can be used for robots to identify and grasp goods, realizing automated operations in smart warehouses. Furthermore, this system can also be applied to fields such as medical, security, and service robots, providing strong support for the development of robotics technology.

[0068] Currently, existing structured light 3D machine vision grasping systems are widely used in the fields of target recognition and grasping.

[0069] High-precision 3D shape reconstruction. It can acquire the 3D shape information of a target object with high accuracy, thus providing precise geometric data for subsequent target recognition and grasping operations.

[0070] Independent light source and camera. The positions of the light source and camera can be freely adjusted, which gives the system a certain degree of flexibility and adaptability, enabling it to cope with different application scenarios and target objects, and facilitating system layout and adjustment.

[0071] Non-contact operation. Information about the target object is obtained in a non-contact manner, without direct contact. This is significant for objects with special shapes or those that are fragile, as it can prevent damage or contamination, improving the safety and reliability of the operation.

[0072] On the one hand, such as Figure 1 As shown, an embodiment of the present invention provides a grasping method guided by structured light 3D machine vision, comprising:

[0073] S100: Obtain the target point cloud of the target object;

[0074] It should be noted that in some embodiments, step S100 may include: acquiring first point cloud data of the target object from the structured light camera; removing point clouds that are irrelevant to the target object from the first point cloud data to obtain the target point cloud.

[0075] In some specific embodiments, such as Figure 2 As shown, the structured light camera acquires the point cloud of the object to be captured (target object). Irrelevant point clouds are removed using software or other methods, and only the point cloud of the target object is retained. This file is called the point cloud model or point cloud template.

[0076] S200: Acquire a two-dimensional image of the work scene containing the target object, and then determine the three-dimensional point cloud of the work scene based on the depth information of the two-dimensional image.

[0077] It should be noted that in some embodiments, step S200 may include: acquiring a two-dimensional image of the work scene containing the target object from a structured light camera; determining the depth information of the two-dimensional image based on the principle of structured light three-dimensional imaging, and generating a depth map; generating second point cloud data of the work scene based on the depth map; performing image preprocessing on the second point cloud data to obtain a three-dimensional point cloud of the work scene; wherein, image preprocessing includes downsampling and filtering for noise reduction.

[0078] In some specific embodiments, such as Figure 2As shown, a structured light camera is deployed in the grasping work scene. The captured 2D images and 3D point clouds will undergo image preprocessing such as filtering and downsampling. Specifically, the object to be grasped should be within the camera's working field of view and working distance when the structured light camera is deployed in the grasping work scene. When the structured light camera is working, its onboard industrial 2D camera can capture 2D images of the current scene. Based on the principle of structured light 3D imaging, the depth information of the 2D image within the working field of view can be calculated to generate a depth map. The corresponding 3D point cloud data can then be generated using the depth map. The amount of 3D point cloud data is often quite large, and the processing of point cloud information generally requires image preprocessing operations such as downsampling and filtering for noise reduction.

[0079] S300. Perform 3D point-pair feature template matching on the 3D point cloud and the target point cloud to obtain the transformation matrix from the target point cloud to the 3D point cloud.

[0080] It should be noted that, in some embodiments, step S300 may include: determining point pair features for each point in the point cloud based on the relationship between the relative position and normal vectors of each point in the point cloud and other points in the point cloud; wherein, the point pair features include a first point pair feature for each point in the 3D point cloud and a second point pair feature for each point in the target point cloud; performing vector pairing on each first point pair feature and each second point pair feature to obtain a matching score between each point in the 3D point cloud and the target point cloud; determining the target matching point in the 3D point cloud for each point in the target point cloud based on the sum of the matching scores between each point in the target point cloud and all points in the 3D point cloud, thereby obtaining a matching point pair between the target point cloud and the 3D point cloud; constructing a rotation matrix and a translation vector based on the matching point pair; performing singular value decomposition on the rotation matrix and the translation vector, and obtaining the target rotation matrix and the target translation vector through convergence iteration; and determining the transformation matrix from the target point cloud to the 3D point cloud based on the target rotation matrix and the target translation vector.

[0081] Specifically, determining the point-pair features of each point in the point cloud based on the relationship between the relative position and normal vector of each point in the point cloud and other points in the point cloud can include: determining the first point-pair features of each point in the 3D point cloud based on the relationship between the relative position and normal vector of each point in the 3D point cloud and other points in the point cloud; and determining the second point-pair features of each point in the target point cloud based on the relationship between the relative position and normal vector of each point in the target point cloud and other points in the point cloud.

[0082] In some embodiments, the point-pair features of each point in the point cloud are determined based on the relationship between the relative position and normal vector of each point in the point cloud and other points in the point cloud. This includes: determining a relative position vector based on the relative position between point i and point j in the point cloud; obtaining a first normal vector of point i and a second normal vector of point j; further determining a first angle between the relative position vector and the first normal vector, and determining a second angle between the first normal vector and the second normal vector; and determining the point-pair features of point i and point j based on the relative position vector, the first angle, and the second angle. The expression for the point-pair features is:

[0083] f ij =(v ij (cosθ,sinθ,cosφ,sinφ)

[0084] In the formula, f ij V represents the point pair features of points i and j. ij Let θ represent the relative position vector between points i and j, φ represent the first included angle, and φ represent the second included angle.

[0085] In some embodiments, singular value decomposition is performed on the rotation matrix and translation vector, and the target rotation matrix and target translation vector are obtained through convergent iteration, including: determining the first centroid based on the average summation of all points in the target point cloud, and determining the second centroid based on the average summation of all points in the 3D point cloud; performing singular value decomposition on the rotation matrix and translation vector based on the first and second centroids, and obtaining the objective function based on the simplified decomposition results; wherein, the expression of the objective function is:

[0086]

[0087] In the formula, q′ i =q i -μ q q i Let i and μ represent points in a 3D point cloud. q p′ represents the second mass center; i =p i -μ p p i Let i and μ represent points in the target point cloud. p R represents the first centroid; R represents the rotation matrix; t represents the translation vector;

[0088] Based on the objective function, the target rotation matrix and target translation vector are obtained through convergent iteration.

[0089] In some specific embodiments, a three-dimensional point pair features (PPF) template matching algorithm can be used to estimate the rigid body transformation relationship between the two point clouds obtained in the aforementioned steps, obtain the pose information of the target object, and thus enable the robotic arm to reach the position of the target object for grasping.

[0090] The 3D point-pair feature template matching algorithm includes two processes: coarse matching and fine matching. In coarse matching, it is first assumed that both the scene to be matched and the model used for matching are composed of a finite number of directed points.

[0091] For a point p i First, it is necessary to calculate the relationship between the relative position of the point and other points and the normal vector, and construct a vector describing the features of the point, that is, the point-pair feature.

[0092] Suppose there is a point p i and point p j Their relative position vector is v ij =p j -p i The normal vectors are n i and n j Then their point-pair eigenvectors can be represented as:

[0093] f ij =(v ij ,cosθ,sinθ,cosφ,sinφ) (1)

[0094] Where θ and φ represent v, respectively. ij and n i The angle between them, and n i With n j The angle between them can be calculated using the following formula.

[0095]

[0096]

[0097] cosφ=n i ·n j (4)

[0098] sinφ=||n i ×n j || (5)

[0099] For two point clouds P and Q, we can calculate the point-pair feature vectors for each point in them, then pair these vectors together and calculate their similarity scores. Specifically, for point p... i and point qk Their point pairs feature vectors are f ij and g kl The similarity score (match score) between them can be calculated using the following formula;

[0100] m ik =(f ij -g kl ) 2 (6)

[0101] Where m ik Point p i and q k The matching scores between the two point clouds can be summed to obtain the point pair feature similarity score between the two point clouds.

[0102] After calculating the feature similarity scores of the point pairs, it is necessary to find the best match between the two point clouds. The sum of the matching scores between each point in point cloud P and all points in point cloud Q can be calculated using the following formula:

[0103]

[0104] Where n is the number of points in point cloud Q, and σ is a parameter used to control the weight distribution. Ultimately, the optimal matching point (target matching point) q of each point in point cloud P in point cloud Q can be found. π(i) This maximizes the total matching score.

[0105]

[0106] in, It is the permutation and combination of all points in the point cloud Q.

[0107] Then estimate the rigid body transformation:

[0108] After finding the optimal match between two point clouds, the rigid body transformation between them can be estimated using the point pair eigenvectors. Specifically, the rotation matrix R and translation vector t can be calculated using the following formula:

[0109] R = R y (θ)R z (φ)R x (α) (9)

[0110] t=qR p (10)

[0111] Where α is a parameter in the feature vector of a point pair, it can be expressed as:

[0112] α=atant2(v ij ·(n i ·nj ),v ij ·n i (11)

[0113] R x ,R y and R z The rotation matrices q and p are the rotations about the x, y, and z axes, respectively, and the matching point pairs p are the matching points. i and q π(i) The coordinates.

[0114] Based on the coarse matching, relatively accurate R and t are obtained. Then, a greedy algorithm is used to directly find the midpoint p in the point cloud P. i The point q that is closest to the point cloud Q i Using these points as corresponding points, we then continue to obtain the R and t matrices between the two point clouds. We iteratively calculate the above two steps until convergence.

[0115] Right now:

[0116] q i =Rp i +t (12)

[0117] Due to noise and mismatches, the above equation does not always hold true. Therefore, the objective function to be minimized is:

[0118]

[0119] Solving for R and t using Singular Value Decomposition (SVD):

[0120] Define the centroids of the two point clouds as The following steps are taken to process equation (9):

[0121]

[0122] Let p′ i =p i -μ p ,q′ i =q i -μ q The objective function simplifies to

[0123]

[0124] The goal of this paper on point cloud matching is to make the algorithm function converge.

[0125] Let R * and t * To find the optimal solution, the optimization problem is solved in two steps:

[0126] 1.

[0127] 2. t * =μ q -Rμ p

[0128] Will Expanded to:

[0129]

[0130] make SVD decomposition yields

[0131] W=U∑V T (16)

[0132] When W is full rank, we have:

[0133]

[0134] And corresponding to a unique U, V combination, the corresponding:

[0135] R * =UV T (17)

[0136] t * =μ q -Rμ p (18)

[0137] It should be noted that, compared to traditional point-to-point feature algorithms that rely solely on calculating the normal vector features of the point cloud model and all objects in the scene, this invention reduces the possibility of false identification. This is mainly reflected in the fact that traditional PPF algorithms calculate the normal vectors of all surfaces, but in real-world testing, such as... Figure 3 In this case, both objects have flat surfaces, and the area of ​​the circular object's point cloud is smaller than that of the square object's point cloud. Therefore, the circular object's point cloud will be contained within the square object's point cloud. Their normal vector features are more similar than when there is only one circular object surface.

[0138] In this invention, an edge extraction algorithm on two-dimensional images adds a shape or edge feature to the point-pair features, improving the recognition effect in complex point clouds and reducing the possibility of recognition errors caused by overlapping or similar point cloud structures. The algorithm's performance is as follows: Figure 4 As shown in the figure. This step can be extended to the recognition of objects of arbitrary shapes, and reduces the number of subsequent point-pair feature calculations, thus improving the recognition speed. The specific implementation process is shown in Algorithm 1 in Table 1:

[0139]

[0140]

[0141] By comparing other PPF algorithms with the improved algorithm for some objects, the results are shown in Table 2 (mean square error of matching results between different point-pair feature algorithms for rectangles, squares, and circles) and Table 3 (average matching time of point-pair features for rectangles, squares, and circles under different algorithms).

[0142] Table 2

[0143]

[0144] Table 3

[0145]

[0146] In Table 2, the mathematical meaning of RMSE is: the deviation between the final matched pose and the matched object in the scene point cloud. Its mathematical formula is shown in (18). It can be used to evaluate the quality of the matching result. A value close to 0 indicates that the matching deviation is smaller and the model matching result is more consistent with the target pose in the scene; a larger value indicates that the matching deviation is larger and the model matching result is farther away from the target pose.

[0147]

[0148] Table 3 compares the time taken for point-to-point matching among different point-to-point feature algorithms; shorter time indicates faster matching speed. Experimental results show that our algorithm performs well in this experiment. Without affecting the centroid pose of the target object, it can match point-to-point features in a short time, thus improving the overall computational efficiency of the algorithm.

[0149] S400: Based on the transformation matrix, determine the first pose information of the target object, combine it with the second pose information of the grasping device, determine the grasping target through Monte Carlo tree search, and then execute the grasping process.

[0150] It should be noted that in some embodiments, the Monte Carlo tree search method is used to determine the crawling target, including: using the second pose information as the root node and the first pose information as the child node; expanding the child nodes and determining the target node based on the expansion result; determining the crawling value of the target node based on the spatial information of the target node's crawling and placement; evaluating the crawling value through a heuristic function to obtain updated information; backpropagating the updated information along the path of the child node expansion to update the node information; sequentially using each node of the expansion result as the target node, and then returning to the step of determining the crawling value of the target node based on the spatial information of the target node's crawling and placement, until a preset number of searches or time limit is reached; and determining the crawling target based on the updated crawling value of each node.

[0151] In some specific embodiments, such as Figure 2 As shown, after completing target point cloud recognition in the scene, the system can obtain the visual result of the target, that is, the transformation matrix from the point cloud template to the target point cloud. It should be noted that after completing target point cloud recognition in the scene, meaning the point cloud model has correctly matched the target point cloud in the scene, the system can obtain the visual result of the matched target. The visual result includes the following information: 1. Position information: representing the coordinates of the center point of the target object in the three-dimensional coordinate system, which can be used to determine the object's position and location. 2. Pose information: representing the orientation and pose of the target object in three-dimensional space. It usually includes rotation angle, rotation axis, or quaternions representing the object's pose changes. 3. Matching score: an index used to evaluate the similarity between the target object and the template. It can represent the reliability and accuracy of the match; a higher matching score indicates a greater probability of matching. The visual result is a 4x4 transformation matrix, also called a homogeneous transformation matrix or pose matrix, which consists of a rotation matrix and a translation vector. At this point, the system uses the mathematical transformation relationship between the robotic arm coordinate system and the camera coordinate system to transform the visual result into the robotic arm coordinate system, allowing the robotic arm to move to the target position for grasping. At this point, the robotic arm can move directly to the target location to grasp the object. This invention proposes a Monte Carlo Tree Search (MCTS) grasping strategy to improve grasping efficiency when there are many objects in the scene. For example... Figure 5 As shown, it can be defined as follows:

[0152] 1. Selection. The robot's pose information can be regarded as the root node, and the matching targets that have been scored and are to be grasped can be regarded as unexplored child nodes node'. The tree is traversed downwards recursively through the child nodes.

[0153] 2. Expand this child node. Based on the placement pose information, it can be used as a child node of node, and simulate the unexplored node (i.e., the target node).

[0154] 3. Simulate the selected node. Based on the spatial information of grasping and placing:

[0155]

[0156] The score, which matches the target, is used to comprehensively evaluate the crawling value of the node. At this point, a heuristic function UCB1 (Upper Confidence Bounds) can be defined to evaluate the quality of the simulation results. The UCB formula is as follows:

[0157]

[0158] in, It is the node estimate. n j is the number of times a node has been visited; n is the total number of times its parent node has been visited. The number of times the algorithm runs is set under certain conditions, when UCB1 max Update The subscript 'max' indicates the maximum value.

[0159] 4. Backpropagation updates node information. The value of the simulation result of the child node is updated to the state of that node, and the state of all parent nodes is updated in turn.

[0160] 5. Repeat the above process for all child nodes until the predetermined number of searches or time limit is reached.

[0161] 6. Finally, select the optimal crawling target based on the value of the node status and execute the crawling process.

[0162] like Figure 6 As shown, the target matching score and the Z-axis height of the grasping pose are displayed before the grasping action. This also represents the height of the camera from the object, in millimeters (mm). The spatial distances between all corresponding object positions and their placement positions are obtained, and the results are shown in Table 4 (distances and scores between rectangles, squares, circles, and their placement positions).

[0163] Table 4

[0164] Distance Score Score / Distance*100 Pane 1 317.065 0.9979 0.31473 Pane 2 184.977 0.8836 0.47768 Circular column 1 278.846 0.9763 0.35012 Circular column 2 234.9 0.9620 0.40953 Circular column 3 340.203 0.9526 0.27983 Diamond 317.548 1.000 0.31491

[0165] The probability of "winning" in each decision is represented by a score per unit distance, and the calculation results are shown in Table 4. According to the MCTS decision-making process, after each round of decision-making, the object from the previous round is considered to have been captured and will not appear in the next round's decision. The simulation results in the program are as follows:

[0166] First round decision: Pane 2, Circular column 2 wins;

[0167] Second round decision: Circular column 1, Diamond wins;

[0168] Third round decision: Pane 1 wins;

[0169] Therefore, the crawling order should be Pane 2, Circular column 2, Circular column 1, Diamond, Pane 1 in that order. The program algorithm flow is shown in Table 5.

[0170] Table 5

[0171]

[0172]

[0173] This grasping strategy performs better when grasping a large number of objects. After the algorithm determines the optimal grasping result, it feeds back to the robotic arm, instructing it to move to the priority target, grasp it, and place it according to the corresponding position based on the previous visual result. The placement position is also determined during object recognition. In other words, the system achieves both visual grasping and visual placement.

[0174] In summary, this invention addresses the process of a structured light 3D machine vision grasping system, aiming to improve the stability and efficiency of the system in the 3D recognition and matching process, and to enhance the system's generalization ability. Existing structured light 3D machine vision grasping systems require processing large amounts of 3D data and performing complex algorithm calculations and analyses. This results in slow system processing speeds, making it difficult to meet the requirements of real-time grasping operations. Especially in applications requiring rapid response, such as robot grasping and production line operations, faster processing speeds and real-time performance are needed to ensure system performance and efficiency. Furthermore, they often lack generalization ability and adaptability when facing different types of target objects and scenes. The system typically needs to be trained and its parameters adjusted for specific target objects; for unknown objects or objects of different shapes, the system may not be able to accurately identify and grasp them. Therefore, this invention makes three innovations to improve the system's working efficiency.

[0175] (1) Improve the stability of point cloud matching. By extracting edge points and calculating features, a more compact and information-rich feature representation can be obtained. Edge points have strong shape and geometric features, and the extracted features can more accurately characterize the edge shape of the target object. This helps to improve the accuracy and robustness of recognition. In addition, this method can also filter out noise and redundant points in the point cloud, retaining the edge information of the target object. By reducing noise and redundancy, the recognition algorithm can focus more on the edge features of the target object, improving the reliability and stability of recognition. Moreover, the edge features of the same object should have similar feature descriptors in point cloud data from different viewpoints or positions. Through the consistency of edge features, the stability of matching can be enhanced, so that the target object can be accurately matched in different scenes. Table 1 shows the point-pair feature matching error results of the improved algorithm.

[0176] (2) Improve the speed of target point cloud recognition and matching. Edge points are regions with significant changes in the point cloud, representing the contours and boundaries of the target object. By combining two-dimensional images to extract edge points, the information of the two-dimensional images is effectively utilized and processed in the depth map. Some redundant point cloud data can be removed, thereby reducing the number of point clouds. In the recognition stage, the algorithm matches the point cloud data to be recognized with the feature template. The matching process can be judged by calculating the similarity or distance between the edge points in the point cloud to be recognized and the feature template. The number of point cloud data directly affects the computational complexity of the algorithm. Reducing the number of point clouds can reduce the computational burden of the algorithm and improve the recognition speed. Especially in large-scale point cloud data or real-time applications, reducing the number of point clouds can significantly improve the real-time performance and response speed of the system. Table 2 shows the point-to-feature matching time of the improved algorithm.

[0177] (3) A robotic arm grasping selection strategy is proposed. This strategy selects the most suitable grasping order based on the shape, size, center of gravity, and other characteristics of the target object. In this invention, the parameters that can be used include: the distance from the grasped object to the placed target, point cloud matching score, and a manually assigned order. All of these factors can be initialized with weights, allowing the system to calculate the grasping strategy during the grasping process. The grasping selection strategy can also consider the characteristics and limitations of different scenarios and objects as needed. Different objects have different shapes, center of gravity distributions, and grasping requirements, while different scenarios may involve factors such as occlusion and complex backgrounds. The grasping selection strategy can flexibly adjust the grasping strategy according to different situations and requirements, adapting to different scenarios and objects, thereby improving the system's adaptability and flexibility.

[0178] On the other hand, such as Figure 7 As shown, an embodiment of the present invention provides a grasping system 500 guided by structured light 3D machine vision, comprising: a first module 510 for acquiring a target point cloud of a target object; a second module 520 for acquiring a 2D image of a work scene containing the target object, and then determining a 3D point cloud of the work scene based on the depth information of the 2D image; a third module 530 for performing 3D point-to-point feature template matching on the 3D point cloud and the target point cloud to obtain a transformation matrix from the target point cloud to the 3D point cloud; and a fourth module 540 for determining the first pose information of the target object based on the transformation matrix, combining it with the second pose information of the grasping device, determining the grasping target through Monte Carlo tree search, and then executing the grasping process.

[0179] The content of the method embodiments of the present invention is applicable to the device embodiments. The specific functions implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above methods.

[0180] like Figure 8As shown, another aspect of the present invention provides an electronic device 600, including a processor 610 and a memory 620;

[0181] Memory 620 is used to store programs;

[0182] The processor 610 executes the program as described above.

[0183] The content of the method embodiments of the present invention is applicable to the embodiments of the present electronic device. The specific functions implemented by the embodiments of the present electronic device are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above methods.

[0184] Another aspect of this invention provides a computer-readable storage medium storing a program that is executed by a processor to implement the method described above.

[0185] The content of the method embodiments of the present invention is applicable to the computer-readable storage medium embodiments. The specific functions implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above methods.

[0186] This invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned method.

[0187] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0188] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0189] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0190] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution means, apparatus, or device (such as a computer-based device, a processor-including device, or other means that can fetch and execute instructions from, or in conjunction with, an instruction execution means, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution means, apparatus, or device.

[0191] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0192] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution device. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0193] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0194] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0195] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A grasping method based on structured light-guided 3D machine vision, characterized in that, include: Obtain the target point cloud of the target object; A two-dimensional image of the work scene containing the target object is acquired, and then the three-dimensional point cloud of the work scene is determined based on the depth information of the two-dimensional image; Perform 3D point-pair feature template matching on the 3D point cloud and the target point cloud to obtain the transformation matrix from the target point cloud to the 3D point cloud; Based on the transformation matrix, the first pose information of the target object is determined. Combined with the second pose information of the grasping device, the grasping target is determined by the Monte Carlo tree search method, and then the grasping process is executed. The step of performing 3D point-pair feature template matching on the 3D point cloud and the target point cloud to obtain the transformation matrix from the target point cloud to the 3D point cloud includes: Based on the relationship between the relative position and normal vector of each point in the point cloud and other points in the point cloud, the point pair features of each point in the point cloud are determined; wherein, the point pair features include the first point pair features of each point in the 3D point cloud and the second point pair features of each point in the target point cloud; Vector pairing is performed on each first point pair feature and each second point pair feature to obtain the matching score between each point of the 3D point cloud and the target point cloud; Based on the sum of the matching scores between each point in the target point cloud and all points in the three-dimensional point cloud, the target matching point in the three-dimensional point cloud for each point in the target point cloud is determined, and the matching point pair between the target point cloud and the three-dimensional point cloud is obtained. Based on the matching point pairs, construct a rotation matrix and a translation vector; Singular value decomposition is performed on the rotation matrix and the translation vector, and the target rotation matrix and the target translation vector are obtained through convergent iteration. The transformation matrix from the target point cloud to the 3D point cloud is determined based on the target rotation matrix and the target translation vector; Specifically, based on the relative positions and normal vector relationships between each point in the point cloud and other points in the point cloud, the point pair features of each point in the point cloud are determined, including: Based on the relationship between the relative position and normal vector of each point in the 3D point cloud and other points, the first point pair feature of each point in the 3D point cloud is determined; Based on the relationship between the relative position and normal vector of each point in the target point cloud and other points, the second point pair feature of each point in the target point cloud is determined; The step of determining the point pair features of each point in the point cloud based on the relationship between the relative position and normal vector of each point in the point cloud and other points in the point cloud includes: Based on the midpoint of the point cloud With point Determine the relative position vector between them; and obtain the points. The first normal vector and the point The second normal vector; then determine the first angle between the relative position vector and the first normal vector, and determine the second angle between the first normal vector and the second normal vector; The point is determined based on the relative position vector, the first included angle, and the second included angle. and the points mentioned Point-to-point features; The expression for the point pair feature is as follows: In the formula, Point and points Point-to-point features, Point and points The relative position vector, Indicates the first included angle. Indicates the second included angle; Specifically, for both the 3D point cloud and the target point cloud, the feature vectors of each point are calculated, and then these vectors are paired to calculate their matching scores. Specifically, for each point in the target point cloud... eigenvectors and the midpoint of the 3D point cloud eigenvectors ,point With point The matching score between them is calculated using the following formula; in, Point and Match scores between them.

2. The grasping method based on structured light 3D machine vision guidance according to claim 1, characterized in that, The acquisition of the target point cloud of the target object includes: Acquire the first point cloud data of the target object from a structured light camera; The target point cloud is obtained by removing point clouds that are irrelevant to the target object from the first point cloud data.

3. The grasping method based on structured light 3D machine vision guidance according to claim 1, characterized in that, The step of acquiring a two-dimensional image of the work scene containing the target object, and then determining the three-dimensional point cloud of the work scene based on the depth information of the two-dimensional image, includes: A two-dimensional image of the work scene containing the target object is acquired from a structured light camera; Based on the principle of structured light 3D imaging, the depth information of the 2D image is determined, and a depth map is generated; Based on the depth map, generate the second point cloud data of the working scene; The second point cloud data is subjected to image preprocessing to obtain the three-dimensional point cloud of the working scene; wherein, the image preprocessing includes downsampling and filtering for noise reduction.

4. The grasping method based on structured light 3D machine vision guidance according to claim 1, characterized in that, The step of performing singular value decomposition on the rotation matrix and the translation vector, and obtaining the target rotation matrix and the target translation vector through convergent iteration, includes: The first centroid is determined based on the average sum of all points in the target point cloud, and the second centroid is determined based on the average sum of all points in the three-dimensional point cloud. Based on the first centroid and the second centroid, singular value decomposition is performed on the rotation matrix and the translation vector, and the objective function is obtained by simplification based on the decomposition results. The expression for the objective function is: In the formula, , Represents points in a 3D point cloud , Indicates the second mass center; , Represents points in the target point cloud , Indicates the first mass; Represents the rotation matrix; Represents the translation vector; Based on the objective function, the target rotation matrix and target translation vector are obtained through convergent iteration.

5. The grasping method based on structured light 3D machine vision guidance according to claim 1, characterized in that, The method of determining the crawling target using Monte Carlo tree search includes: The second pose information is used as the root node, and the first pose information is used as the child node; Expand the child nodes and determine the target node based on the expansion results; The capture value of the target node is determined based on the spatial information of the target node capture and placement. The crawling value is evaluated using a heuristic function, thereby obtaining updated information; The update information is propagated back along the extended path of the child node to update the node information; each node of the extended result is taken as the target node in turn, and then the step of determining the crawling value of the target node based on the spatial information of the target node's crawling and placement is returned until the preset number of searches or time limit is reached. The crawling target is determined based on the crawling value of each node after the update.

6. A grasping system guided by structured light 3D machine vision, characterized in that, The system applied to the grasping method based on structured light 3D machine vision guidance as described in claim 1 includes: The first module is used to acquire the target point cloud of the target object; The second module is used to acquire a two-dimensional image of the work scene containing the target object, and then determine the three-dimensional point cloud of the work scene based on the depth information of the two-dimensional image. The third module is used to perform three-dimensional point-pair feature template matching on the three-dimensional point cloud and the target point cloud to obtain the transformation matrix from the target point cloud to the three-dimensional point cloud; The fourth module is used to determine the first pose information of the target object based on the transformation matrix, combine it with the second pose information of the grasping device, determine the grasping target through Monte Carlo tree search, and then execute the grasping process.

7. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 5.

Images