Industrial control method and device
By matching the point cloud of the target object and computing the rigid body transformation matrix, the problem of unstable position estimation of rotary symmetric objects is solved, and the safety and accuracy of manipulator grasping is improved.
Patent Information
- Application Number
- CN202411158666.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-12-27
AI Technical Summary
The existing 3D visual guidance technology is difficult to stably identify the object posture when grabbing rotary symmetric workpieces, resulting in the robot being prone to impact or damage to the material when grasping, reducing the safety of the work.
By coarse and fine matching of the initial point cloud and the target point cloud of the target object, a rigid body transformation matrix is calculated and the point cloud is aligned based on the matrix so that the robot can capture the target object and avoid pose estimation instability.
It improves work safety, reduces the risk of collision between the robot and the target object, enhances the accuracy and robustness of 3D matching, and improves the accuracy of grabbing workpieces.
Smart Images
Figure CN119205475A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of industrial control, and in particular, relates to an industrial control method and device. Background Art
[0002] 3D vision guidance technology has been widely used in logistics, automobiles, industrial intelligent manufacturing and other industries. Vision-guided manipulators can be used to grab materials to replace manual operations. In the related technology, there is a method of grabbing materials through 3D registration algorithms, but this method is prone to identifying different rotational postures during the grabbing and loading of the same batch of rotationally symmetrical workpieces, resulting in the manipulator generating grabbing actions with different rotational postures according to the posture given by vision, which is prone to collision or damage to materials, and the operation safety is low. Summary of the invention
[0003] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes an industrial control method and device, which avoids the problem of unstable pose estimation for rotationally symmetric objects, thereby avoiding collision between the manipulator and the target object or damage to the material, and improving the safety of the operation.
[0004] In a first aspect, the present application provides an industrial control method, the method comprising:
[0005] Processing the collected initial point cloud of the target object at the target angle and the target point cloud corresponding to the target object to obtain a first point cloud corresponding to the target object;
[0006] Calculating a target weighted distance between first point cloud data in the first point cloud and target point cloud data corresponding to the first point cloud data in the target point cloud;
[0007] Based on the target weighted distance corresponding to each of the first point cloud data, a rigid body transformation matrix between the first point cloud and the target point cloud is obtained by solving;
[0008] Processing the first point cloud based on the rigid body transformation matrix to obtain a second point cloud corresponding to the target object;
[0009] Based on the second point cloud, control the grabbing of the target object.
[0010] According to the industrial control method provided in the embodiment of the present application, by coarsely matching the initial point cloud and the target point cloud of the target object, and performing fine matching based on the coarse matching result, a rigid body transformation matrix between the first point cloud corresponding to the target object and the target point cloud is obtained. Therefore, the first point cloud can be aligned to the target point cloud based on the rigid body transformation matrix for the robot to grasp the target object, thereby avoiding the problem of unstable pose estimation for rotationally symmetric objects, thereby avoiding collision between the manipulator and the target object or damage to materials, and improving operation safety.
[0011] In an industrial control method according to an embodiment of the present application, the calculating and obtaining a target weighted distance between first point cloud data in the first point cloud and target point cloud data corresponding to the first point cloud data in the target point cloud comprises:
[0012] Calculating a target distance between a plane where target first point cloud data in a plurality of the first point cloud data and target point cloud data corresponding to the target first point cloud data are located;
[0013] Calculate the weight corresponding to the first target point cloud data based on the target distance corresponding to the first target point cloud data and the median of the target distances corresponding to the plurality of first point cloud data;
[0014] Based on the target distance corresponding to the target first point cloud data and the weight corresponding to the target first point cloud data, a target weighted distance corresponding to the target first point cloud data is calculated.
[0015] In an industrial control method according to an embodiment of the present application, the rigid body transformation matrix between the first point cloud and the target point cloud is obtained based on the target weighted distance corresponding to each of the first point cloud data, including:
[0016] Constructing an algorithm error function based on the first point cloud, the target point cloud and the target weighted distance corresponding to each of the first point cloud data;
[0017] Based on the algorithm error function, multiple rotation angles and multiple translation amounts corresponding to the rigid body transformation matrix are obtained by solving;
[0018] The rigid body transformation matrix is obtained based on the multiple rotation angles and the multiple translation amounts.
[0019] The industrial control method of an embodiment of the present application, based on the algorithm error function, solves and obtains multiple rotation angles and multiple translation amounts corresponding to the rigid body transformation matrix, including:
[0020] Determine a target rotation angle among the multiple rotation angles as a fixed value, and solve the remaining rotation angles of a target number other than the target rotation angle among the multiple rotation angles and the multiple translation amounts based on the fixed value;
[0021] In the case where the remaining rotation angle of the target number satisfies the target constraint condition, the function value of the algorithm error function is calculated based on the remaining rotation angle, the target rotation angle and the multiple translation amounts; the remaining rotation angle of the target number satisfies the target constraint condition including: the target number is 0 or the remaining rotation angle satisfies the angle constraint condition;
[0022] The remaining rotation angle and the target rotation angle corresponding to the minimum function value in at least one function value are determined as multiple rotation angles corresponding to the rigid body transformation matrix, and the multiple translation amounts corresponding to the minimum function value are determined as multiple translation amounts corresponding to the rigid body transformation matrix.
[0023] In an industrial control method according to an embodiment of the present application, the first point cloud is processed based on the rigid body transformation matrix to obtain a second point cloud corresponding to the target object, including:
[0024] Performing transformation processing on the first point cloud based on the rigid body transformation matrix to obtain a third point cloud corresponding to the target object;
[0025] In the case where the difference between the third point cloud data in the third point cloud and the target point cloud data corresponding to the third point cloud data in the target point cloud is greater than a target threshold, the third point cloud is determined as the first point cloud, and a rigid body transformation matrix is obtained based on the updated first point cloud until the iteration condition is met, thereby obtaining a second point cloud corresponding to the target object.
[0026] An industrial control method according to an embodiment of the present application, wherein the collected initial point cloud of the target object at the target angle and the target point cloud corresponding to the target object are processed to obtain a first point cloud corresponding to the target object, including:
[0027] Acquire candidate pose data between initial point cloud data in the initial point cloud and target point cloud data corresponding to the initial point cloud data in the target point cloud; the candidate pose data includes parameter information of transforming the initial point cloud data to the target point cloud data;
[0028] When the rotation angle corresponding to the candidate posture data satisfies the angle constraint condition, determining the candidate posture data as the target posture data;
[0029] The initial point cloud data is transformed based on the target posture data to obtain the first point cloud.
[0030] In an industrial control method according to an embodiment of the present application, before transforming the initial point cloud data based on the target posture data to obtain the first point cloud, the method further includes:
[0031] When the rotation angles corresponding to the candidate posture data do not satisfy the angle constraint condition, calculating the distance between the center point of the minimum bounding box corresponding to the initial point cloud and the center point of the minimum bounding box corresponding to the target point cloud in the target direction;
[0032] The distance is determined as the translation amount of the target posture data in the target direction, and the rotation angle corresponding to the target posture data is determined as 0.
[0033] In a second aspect, the present application provides an industrial control device, the device comprising:
[0034] A first processing module is used to process the collected initial point cloud of the target object at the target angle and the target point cloud corresponding to the target object to obtain a first point cloud corresponding to the target object;
[0035] A second processing module, configured to calculate a target weighted distance between first point cloud data in the first point cloud and target point cloud data corresponding to the first point cloud data in the target point cloud;
[0036] A third processing module, configured to obtain a rigid body transformation matrix between the first point cloud and the target point cloud based on the target weighted distance corresponding to each of the first point cloud data;
[0037] a fourth processing module, configured to process the first point cloud based on the rigid body transformation matrix to obtain a second point cloud corresponding to the target object;
[0038] A fifth processing module is used to control the grabbing of the target object based on the second point cloud.
[0039] According to the industrial control device provided in the embodiment of the present application, a rigid body transformation matrix between the first point cloud corresponding to the target object and the target point cloud is obtained by coarsely matching the initial point cloud and the target point cloud of the target object, and fine matching is performed based on the coarse matching result. Therefore, the first point cloud can be aligned to the target point cloud based on the rigid body transformation matrix for the robot to grasp the target object, thereby avoiding the problem of unstable pose estimation for rotationally symmetric objects, thereby avoiding collision between the manipulator and the target object or damage to materials, and improving operation safety.
[0040] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the industrial control method as described in the first aspect above is implemented.
[0041] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the industrial control method as described in the first aspect above.
[0042] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the industrial control method as described in the first aspect above.
[0043] The above one or more technical solutions in the embodiments of the present application have at least one of the following technical effects:
[0044] By roughly matching the initial point cloud and the target point cloud of the target object, and performing fine matching based on the rough matching results, the rigid body transformation matrix between the first point cloud corresponding to the target object and the target point cloud is obtained. The first point cloud can be aligned to the target point cloud based on the rigid body transformation matrix so that the robot can grasp the target object, avoiding the problem of unstable pose estimation for rotationally symmetric objects, thereby avoiding collision between the manipulator and the target object or damage to the material, and improving operation safety.
[0045] Furthermore, by calculating the distance between the target first point cloud data and the target point cloud data, and then calculating the weight corresponding to the target first point cloud data, the influence of outlier noise points on the subsequent registration process can be effectively reduced, the robustness of the registration algorithm is improved, and the precision and accuracy of 3D matching are improved.
[0046] Furthermore, by using a linear solution method with angle constraints to solve multiple rotation angles and multiple translation amounts corresponding to the rigid body transformation matrix, the solution speed is faster, avoiding the situation where the solution takes a long time and affects the production cycle, and improving the solution efficiency and accuracy.
[0047] Furthermore, the first point cloud is transformed by solving the rigid body transformation matrix to obtain a third point cloud. If there is still a deviation between the third point cloud and the target point cloud, iterative optimization can be continued so that the second point cloud finally solved can completely overlap with the target point cloud, thereby improving the precision and accuracy of subsequent grasping of workpieces.
[0048] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:
[0050] Figure 1It is a flowchart of the industrial control method provided by the embodiment of the present application;
[0051] Figure 2 It is one of the principle schematic diagrams of the industrial control method provided in the embodiment of the present application;
[0052] Figure 3 This is the second principle schematic diagram of the industrial control method provided in the embodiment of the present application;
[0053] Figure 4 It is a structural schematic diagram of an industrial control device provided in an embodiment of the present application;
[0054] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0055] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.
[0056] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0057] The industrial control method, industrial control device, electronic device and readable storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.
[0058] The industrial control method may be applied to a terminal, and may be specifically executed by hardware or software in the terminal.
[0059] The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad). It should also be understood that in some embodiments, the terminal may not be a portable communication device, but a desktop computer with a touch-sensitive surface (e.g., a touch screen display and / or a touch pad).
[0060] In the following various embodiments, a terminal including a display and a touch-sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, a mouse and a joystick.
[0061] The industrial control method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the industrial control method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The industrial control method provided in the embodiment of the present application is described below using the electronic device as an example of the execution subject.
[0062] like Figure 1 As shown, the industrial control method includes: step 110 , step 120 , step 130 , step 140 and step 150 .
[0063] It should be noted that the industrial control method can be applied to industries such as logistics, automobiles, and industrial intelligent manufacturing. For example, the industrial control method can be used to control robots or manipulators to grab auto parts, sort express deliveries, or load and unload materials on a production line.
[0064] Step 110 : Process the collected initial point cloud of the target object at the target angle and the target point cloud corresponding to the target object to obtain a first point cloud corresponding to the target object.
[0065] In this step, the target object may be materials to be transported during an industrial production process, for example, auto parts or other materials such as express delivery.
[0066] For example, in the automotive industry, the target objects are objects with a high degree of rotational symmetry, that is, after the workpiece is rotated a certain angle around a certain rotation axis, the appearance of the workpiece is the same as before the rotation.
[0067] During the rotation of the target object, the target object is collected, for example, three-dimensional coordinate information of the surface of the target object may be collected.
[0068] The initial point cloud is a set of point cloud data of the target object at the target angle.
[0069] The point cloud includes the three-dimensional coordinates and intensity values of the sampling points on the surface of the target object.
[0070] A point cloud is a collection of data points in space that represents a 3D shape or object.
[0071] The target point cloud is the template point cloud data corresponding to the target object. In the process of 3D registration of the initial point cloud, the target point cloud is the reference for the registration of the initial point cloud.
[0072] By processing the initial point cloud and the target point cloud, a first point cloud corresponding to the target object can be obtained.
[0073] For example, the initial point cloud and the target point cloud may be roughly matched to obtain the first point cloud.
[0074] In some embodiments, step 110 may include:
[0075] Acquire candidate pose data between initial point cloud data in the initial point cloud and target point cloud data corresponding to the initial point cloud data in the target point cloud;
[0076] When the rotation angle corresponding to the candidate posture data satisfies the angle constraint condition, the candidate posture data is determined as the target posture data;
[0077] The initial point cloud data is transformed based on the target posture data to obtain a first point cloud.
[0078] In this embodiment, if Figure 3 As shown, the initial point cloud (source point cloud) includes a plurality of initial point cloud data.
[0079] The target point cloud includes multiple target point cloud data.
[0080] During the actual execution process, the initial point cloud and the target point cloud can be roughly matched. For example, the PPF (Point Pair Features) point pair feature pre-alignment method can be used for rough matching, or various methods such as matching based on local features and random sampling consistency, matching based on NDT (Normal Distributions Transform) probability distribution, or matching based on deep learning can be used. The choice can be based on user needs and is not limited in this application.
[0081] After roughly matching the initial point cloud and the target point cloud, multiple matching point pairs can be obtained.
[0082] Each matching point pair includes an initial point cloud data and a target point cloud data corresponding to the initial point cloud data.
[0083] The candidate pose data includes parameter information of the transformation from the initial point cloud data to the target point cloud data, and the candidate pose data is used to characterize the transformation from the initial point cloud data to the target point cloud data.
[0084] The candidate posture data can be expressed in a 6-degree-of-freedom form, for example, the candidate posture data includes a rotation matrix and a translation matrix, which can convert the rotation matrix into rotation angles along the X-axis, Y-axis and Z-axis, and convert the translation matrix into translation amounts along the X-axis, Y-axis and Z-axis directions.
[0085] Angle constraints can be user-defined, for example, the rotation angle can be constrained to be between -10 degrees and 10 degrees.
[0086] The target posture data can be used as the initial posture data in the subsequent fine matching process.
[0087] After the target posture data is determined, the initial point cloud data may be transformed based on the target posture data to align the initial point cloud data to the target point cloud data.
[0088] By performing the above operation on each initial point cloud data, each initial point cloud data in the initial point cloud can be aligned to the target point cloud data corresponding to the initial point cloud data, thereby obtaining a first point cloud.
[0089] In some embodiments, before transforming the initial point cloud data based on the target posture data to obtain the first point cloud, the method further includes:
[0090] When the rotation angles corresponding to the candidate posture data do not satisfy the angle constraint conditions, the distance between the center point of the minimum bounding box corresponding to the initial point cloud and the center point of the minimum bounding box corresponding to the target point cloud in the target direction is calculated;
[0091] The distance is determined as the translation amount of the target posture data in the target direction, and the rotation angle corresponding to the target posture data is determined as 0.
[0092] In this embodiment, when the rotation angles corresponding to the candidate posture data do not satisfy the angle constraint condition, the posture data between the initial point cloud and the target point cloud can be reconstructed.
[0093] The minimum bounding box is the optimal bounding space corresponding to the discrete point cloud data set.
[0094] After the minimum bounding box is determined, the center point of the minimum bounding box can be obtained, and then the distance between the center point of the minimum bounding box corresponding to the initial point cloud and the center point of the minimum bounding box corresponding to the target point cloud in the target direction is calculated.
[0095] The target direction may include an X-axis direction, a Y-axis direction, and a Z-axis direction.
[0096] The distance between the two center points in the X-axis direction, the Y-axis direction, and the Z-axis direction can be calculated.
[0097] The distances in the three directions obtained above can be used as the translation amounts of the target posture data in the X-axis direction, the Y-axis direction and the Z-axis direction, respectively, and the rotation angle corresponding to the target posture data is determined to be 0.
[0098] According to the industrial control method provided in the embodiment of the present application, by roughly matching the initial point cloud and the target point cloud, the initial point cloud can be pre-aligned to the target point cloud, thereby improving the precision and accuracy of subsequent matching results, while improving the matching efficiency and reducing commercial use costs.
[0099] Step 120 : Calculate and obtain a target weighted distance between the first point cloud data in the first point cloud and the target point cloud data corresponding to the first point cloud data in the target point cloud.
[0100] In this step, the first point cloud includes a plurality of first point cloud data.
[0101] By performing rough matching on the initial point cloud and the target point cloud, a plurality of matching point pairs can be obtained, each matching point pair including first point cloud data and target point cloud data corresponding to the first point cloud data.
[0102] The target weighted distance is used to measure the influence of the first point cloud data on the subsequent registration process.
[0103] In some embodiments, step 120 may include:
[0104] Calculating a target distance between a target first point cloud data and a plane where target point cloud data corresponding to the target first point cloud data are located in a plurality of first point cloud data;
[0105] Based on the target distance corresponding to the first point cloud data of the target and the median of the target distances corresponding to the plurality of first point cloud data, a weight corresponding to the first point cloud data of the target is calculated;
[0106] Based on the target distance corresponding to the target first point cloud data and the weight corresponding to the target first point cloud data, the target weighted distance corresponding to the target first point cloud data is calculated.
[0107] In this embodiment, the target first point cloud data is any first point cloud data among a plurality of first point cloud data.
[0108] The target distance is used to characterize the distance from the target first point cloud data to the target plane, and the target plane is the plane where the target point cloud data corresponding to the target first point cloud data is located.
[0109] The target distance corresponding to the target first point cloud data can be obtained by calculating the distance between the target first point cloud data and the target point cloud data and the normal of the plane where the target point cloud data is located.
[0110] For each first point cloud data, the target distance corresponding to the first point cloud data can be calculated, and then the median value of multiple target distances can be obtained.
[0111] When the median values of multiple target distances are known, the weight corresponding to the first point cloud data of the target can be calculated according to the exponential function corresponding to the median value.
[0112] In the actual execution process, the weight corresponding to the first point cloud data of the target can be calculated based on the Welsch weight function, Tukey weight function, Fair weight function or Logistic weight function. The following takes the Welsch weight function as an example to illustrate the calculation method of the weight:
[0113]
[0114] Among them, x is the target distance corresponding to the first point cloud data of the target, v is the median of multiple target distances, exp() is an exponential function with e as the base, and the schematic diagram of the function is as follows Figure 2 shown.
[0115] Figure 2 A schematic diagram of the function with median values of 0.05, 0.5 and 2 is shown.
[0116] When the target distance corresponding to the first target point cloud data is larger, the weight corresponding to the first target point cloud data is smaller, which means that the influence of the point cloud data on the subsequent registration process is smaller.
[0117] According to the industrial control method provided in the embodiment of the present application, by calculating the distance between the target first point cloud data and the target point cloud data, and then calculating the weight corresponding to the target first point cloud data, the influence of outlier noise points on the subsequent alignment process can be effectively reduced, the robustness of the alignment algorithm is improved, and the precision and accuracy of 3D matching are improved.
[0118] Step 130: Based on the target weighted distance corresponding to each first point cloud data, a rigid body transformation matrix between the first point cloud and the target point cloud is obtained by solving.
[0119] In this step, the rigid body transformation matrix between the first point cloud and the target point cloud is used to characterize the transformation required to transform the first point cloud into the target point cloud.
[0120] The rigid body transformation matrix includes the rotation matrix and the translation matrix.
[0121] Among them, multiple rotation angles can be obtained based on the rotation matrix, and multiple translation amounts can be obtained based on the translation matrix.
[0122] That is, the first point cloud is transformed based on the rigid body transformation matrix, so that the first point cloud can be aligned to the target point cloud.
[0123] Based on the target weighted distance corresponding to each first point cloud data, an iterative solution can be performed to obtain the final rigid body transformation matrix.
[0124] In some embodiments, step 130 may include:
[0125] Constructing an algorithm error function based on the first point cloud, the target point cloud and the target weighted distance corresponding to each first point cloud data;
[0126] Based on the algorithm error function, multiple rotation angles and multiple translation amounts corresponding to the rigid body transformation matrix are solved;
[0127] Based on multiple rotation angles and multiple translation amounts, a rigid body transformation matrix is obtained.
[0128] In this embodiment, the algorithm error function is the sum of squares of vertical distances from all first point cloud data in the first point cloud to the plane where the corresponding target point cloud data is located.
[0129] The algorithm error function may include the rigid body transformation matrix to be solved.
[0130] The multiple rotation angles corresponding to the rigid body transformation matrix include a rotation angle along the X-axis direction, a rotation angle along the Y-axis direction, and a rotation angle along the Z-axis direction.
[0131] The multiple translation amounts corresponding to the rigid body transformation matrix include a translation amount along the X-axis direction, a translation amount along the Y-axis direction, and a translation amount along the Z-axis direction.
[0132] In the actual execution process, multiple rotation angles and multiple translation amounts corresponding to the rigid body transformation matrix can be solved through an iterative algorithm.
[0133] For example, an iterative solution may be performed through a point-to-plane ICP (Iterative Closest Point) algorithm to obtain multiple rotation angles and multiple translation amounts.
[0134] In the actual execution process, the algorithm error function shown below can be constructed:
[0135]
[0136] Among them, F error is the function value of the algorithm error function, s i is the first point cloud, d i is the target point cloud, n i is the normal corresponding to the target point cloud, M is the rigid body transformation matrix, w i is the weight.
[0137] In some embodiments, solving a plurality of rotation angles and a plurality of translation amounts corresponding to a rigid body transformation matrix based on an algorithm error function may include:
[0138] Determine a target rotation angle among the multiple rotation angles as a fixed value, and solve the remaining rotation angles of the target number and multiple translation amounts among the multiple rotation angles except the target rotation angle based on the fixed value;
[0139] When the remaining rotation angle of the target number satisfies the target constraint condition, the function value of the algorithm error function is calculated based on the remaining rotation angle, the target rotation angle and the plurality of translation amounts;
[0140] The remaining rotation angle and the target rotation angle corresponding to the minimum function value in at least one function value are determined as multiple rotation angles corresponding to the rigid body transformation matrix, and the multiple translation amounts corresponding to the minimum function value are determined as multiple translation amounts corresponding to the rigid body transformation matrix.
[0141] In this embodiment, the target rotation angle is one or more rotation angles among a plurality of rotation angles.
[0142] For example, when multiple rotation angles include α, β and γ, the target rotation angle can be any one of α, β and γ, or the target rotation angle can be any two of α, β and γ, or the target rotation angle can be α, β and γ.
[0143] Selecting one or more rotation angles from multiple rotation angles as target rotation angles and fixing the target rotation angles to a preset value or a known value can reduce the number of variables that need to be optimized.
[0144] In the actual execution process, an objective function to be solved can be constructed, and the remaining rotation angles to be solved and multiple translation amounts can be processed based on fixed values to obtain a new objective function.
[0145] The objective function can then be decomposed based on a decomposition algorithm to obtain the residual rotation angle and multiple translation amounts.
[0146] The number of targets can be 2, 1, or 0.
[0147] For example, when the number of target rotation angles is 1, the target number is 2; when the number of target rotation angles is 2, the target number is 1; and when the number of target rotation angles is 3, the target number is 0.
[0148] The remaining rotation angle of the target number satisfies the target constraint condition, which may include: the target number is 0 or the remaining rotation angle satisfies the angle constraint condition.
[0149] When the multiple rotation angles are all fixed values, each rotation angle satisfies the angle constraint condition.
[0150] The angle constraint condition can be customized based on the user. The angle constraint condition is used to constrain the rotation angle within a set range. For example, the angle constraint condition can include the rotation angle in the X-axis direction being between -10 degrees and 10 degrees, the rotation angle in the Y-axis direction being between -15 degrees and 15 degrees, and the rotation angle in the Z-axis direction being between -10 degrees and 10 degrees; or it can be other constraints, which are not limited in this application.
[0151] Substituting the multiple remaining rotation angles, target rotation angles, and multiple translation amounts obtained by solving into the algorithm error function, the function value of the algorithm error function can be obtained.
[0152] After calculating and obtaining multiple function values that satisfy the constraint conditions, the minimum function value can be determined from the multiple function values, and the result corresponding to the minimum function value is determined as the optimal solution of the algorithm error function.
[0153] In the actual execution process, the rotation angles corresponding to the rigid body transformation matrix M can be set to α, β and γ in the XYZ directions respectively. Taking the ICP (Iterative Closest Point) algorithm for iterative solution as an example, ICP is a step-by-step iterative optimization algorithm. The rotation angle corresponding to the rigid body transformation obtained in each iteration is small. The trigonometric function corresponding to the rotation angle can be mathematically approximated as sinα=α, sinβ=β, sinγ=γ, cosα=1, cosβ=1, cosγ=1, then the rigid body transformation matrix M can be approximated as:
[0154]
[0155] Among them, T is the translation component, R' is the approximate rotation component, α is the rotation angle of the rigid body transformation matrix in the X direction, β is the rotation angle of the rigid body transformation matrix in the Y direction, γ is the rotation angle of the rigid body transformation matrix in the Z direction, t x is the translation of the rigid body transformation matrix in the X direction, t y is the translation of the rigid body transformation matrix in the Y direction, t z is the translation of the rigid body transformation matrix in the Z direction.
[0156] Based on the above formula, we can get:
[0157]
[0158] Assuming there are N pairs of associated points, we can 1≤i≤N is converted into a matrix expression:
[0159]
[0160] Among them, w is the weight,
[0161]
[0162] x=[αβλt x t y t z ] T
[0163]
[0164] And there are angle constraints on α, β and γ in the x vector, and the linear solution problem is specifically:
[0165] Coefficient matrix A∈R Nx6 , b∈R Nx1 , x=[αβλt x t y t z ] T , α∈[a1,b1],β∈[a2,b2],γ∈[a3,b3], the objective function is: The specific solution method is:
[0166] Step 1: Fix a single unknown number as a boundary value, for example, fix α=a1, then subtract the column corresponding to the fixed unknown number in the coefficient matrix A from the matrix b and multiply it by the fixed value a1 of the unknown number, and remove the corresponding column of the coefficient matrix A, thereby obtaining a new objective function, as shown in the following formula:
[0167] A'x'=b',
[0168]
[0169] x'=[βλt x t y t z ] T
[0170]
[0171] Step 2: Solve the above formula by SVD (Singular Value Decomposition) to obtain β and γ, as well as t x ,t y and t z , and determine whether the values of β and γ obtained by the solution meet the angle constraint conditions. If β and γ meet the angle constraint conditions, a1, β, γ, t x ,t y and t z Substitute into the objective function to obtain the function value and corresponding result;
[0172] Step 3: Fix α, β and γ as boundary values respectively, and solve the 6 degrees of freedom under all fixed single boundary values similar to Step 1, and record the function values and corresponding results that meet the angle constraint conditions;
[0173] Step 4: Randomly fix two variables in α, β and γ to a certain boundary value, similar to Step 1, solve all the 6 degrees of freedom under the fixed two boundary values, and record the function values and corresponding results that meet the angle constraint conditions;
[0174] Step 5: Fix α, β and γ to a certain boundary value at the same time, similar to Step 1, solve the 6 degrees of freedom under all three fixed boundary values, record the function value and the corresponding result. It can be understood that when α, β and γ are all fixed values, α, β and γ all meet the angle constraint conditions;
[0175] Step 6: Compare all the function values that satisfy the angle constraints, and select the result corresponding to the minimum function value as the optimal solution of the original objective function.
[0176] In some embodiments, a nonlinear optimization method with boundary constraints may also be used to solve multiple rotation angles and multiple translation amounts corresponding to the rigid body transformation matrix, which will not be described in detail here.
[0177] According to the industrial control method provided in the embodiment of the present application, a linear solution method with angle constraints is used to solve multiple rotation angles and multiple translation amounts corresponding to the rigid body transformation matrix. The solution speed is fast, avoiding the situation where the solution takes a long time and affects the production rhythm, thereby improving the solution efficiency and solution accuracy.
[0178] Step 140: Process the first point cloud based on the rigid body transformation matrix to obtain a second point cloud corresponding to the target object.
[0179] In this step, when the rigid body transformation matrix is obtained, the first point cloud can be transformed based on the rigid body transformation matrix to be aligned to the target point cloud.
[0180] In some embodiments, after the first point cloud is transformed, it is also possible to determine whether the point cloud after the transformation meets the conditions. For example, if there is still a deviation between the point cloud after the transformation and the target point cloud, iterative solution can be performed again to obtain a point cloud that completely overlaps with the target point cloud.
[0181] The second point cloud is a point cloud obtained by processing the first point cloud, and the second point cloud includes a plurality of second point cloud data.
[0182] The second point cloud is the point cloud aligned with the target point cloud.
[0183] In some embodiments, step 140 may include:
[0184] Transform the first point cloud based on the rigid body transformation matrix to obtain a third point cloud corresponding to the target object;
[0185] When the difference between the third point cloud data in the third point cloud and the target point cloud data corresponding to the third point cloud data in the target point cloud is greater than the target threshold, the third point cloud is determined as the first point cloud, and the rigid body transformation matrix is obtained based on the updated first point cloud until the iteration condition is met, and the second point cloud corresponding to the target object is obtained.
[0186] In this embodiment, the third point cloud includes a plurality of third point cloud data, and each third point cloud data has corresponding target point cloud data in the target point cloud.
[0187] The third point cloud data may overlap with the target point cloud data, or there may be a deviation between the third point cloud data and the target point cloud data.
[0188] The difference between the third point cloud data and the target point cloud data may include an angular deviation and a translational deviation therebetween.
[0189] The target threshold can be customized by the user and is not limited in this application.
[0190] When the difference between the third point cloud data and the target point cloud data is greater than the target threshold, the third point cloud can be determined as the first point cloud, and steps 120 to 140 are repeated until the iteration stop condition is met, such as when the set maximum number of iterations is reached, the change in the rigid body transformation matrix before and after two iterations is less than the first threshold, or the root mean square error of the error metric is less than the second threshold, a new second point cloud is obtained.
[0191] For example, after each iteration, a rigid body transformation matrix is obtained, and the change in the rigid body transformation matrix between two iterations is the difference between the transformation postures generated by the two iterations, for example, it can be the difference between the 6 degrees of freedom obtained by the two iterations.
[0192] The root mean square error of the error metric is in,
[0193] According to the industrial control method provided in the embodiment of the present application, the first point cloud is transformed by solving the rigid body transformation matrix to obtain a third point cloud. If there is still a deviation between the third point cloud and the target point cloud, iterative optimization can be continued so that the second point cloud finally solved can completely overlap with the target point cloud, thereby improving the precision and accuracy of subsequent grasping of workpieces.
[0194] Step 150: Based on the second point cloud, control the grabbing of the target object.
[0195] In this step, features of the target object may be extracted from the second point cloud. For example, the shape, size, and posture of the target object may be extracted.
[0196] Based on the extracted features of the target object, the target object can be identified from the second point cloud.
[0197] According to the position and posture of the target object, the robot's path from the current position to the target object grasping position can be planned, and according to the path planning results, the robot is controlled to grasp the target object.
[0198] According to the industrial control method provided in the embodiment of the present application, by coarsely matching the initial point cloud and the target point cloud of the target object, and performing fine matching based on the coarse matching result, a rigid body transformation matrix between the first point cloud corresponding to the target object and the target point cloud is obtained. Therefore, the first point cloud can be aligned to the target point cloud based on the rigid body transformation matrix for the robot to grasp the target object, thereby avoiding the problem of unstable pose estimation for rotationally symmetric objects, thereby avoiding collision between the manipulator and the target object or damage to materials, and improving operation safety.
[0199] The industrial control device provided in the present application is described below. The industrial control device described below and the industrial control method described above can be referenced to each other.
[0200] The industrial control method provided in the embodiment of the present application can be executed by an industrial control device. In the embodiment of the present application, an industrial control device executing the industrial control method is taken as an example to illustrate the industrial control device provided in the embodiment of the present application.
[0201] An embodiment of the present application also provides an industrial control device.
[0202] like Figure 4 As shown, the industrial control device includes: a first processing module 410 , a second processing module 420 , a third processing module 430 , a fourth processing module 440 and a fifth processing module 450 .
[0203] A first processing module 410 is used to process the collected initial point cloud of the target object at the target angle and the target point cloud corresponding to the target object to obtain a first point cloud corresponding to the target object;
[0204] The second processing module 420 is used to calculate a target weighted distance between the first point cloud data in the first point cloud and the target point cloud data corresponding to the first point cloud data in the target point cloud;
[0205] The third processing module 430 is used to solve and obtain a rigid body transformation matrix between the first point cloud and the target point cloud based on the target weighted distance corresponding to each first point cloud data;
[0206] A fourth processing module 440 is used to process the first point cloud based on the rigid body transformation matrix to obtain a second point cloud corresponding to the target object;
[0207] The fifth processing module 450 is used to control the grabbing of the target object based on the second point cloud.
[0208] According to the industrial control device provided in the embodiment of the present application, a rigid body transformation matrix between the first point cloud corresponding to the target object and the target point cloud is obtained by coarsely matching the initial point cloud and the target point cloud of the target object, and fine matching is performed based on the coarse matching result. Therefore, the first point cloud can be aligned to the target point cloud based on the rigid body transformation matrix for the robot to grasp the target object, thereby avoiding the problem of unstable pose estimation for rotationally symmetric objects, thereby avoiding collision between the manipulator and the target object or damage to materials, and improving operation safety.
[0209] In some embodiments, the second processing module 420 may also be used to:
[0210] Calculating a target distance between a target first point cloud data and a plane where target point cloud data corresponding to the target first point cloud data are located in a plurality of first point cloud data;
[0211] Based on the target distance corresponding to the first point cloud data of the target and the median of the target distances corresponding to the plurality of first point cloud data, a weight corresponding to the first point cloud data of the target is calculated;
[0212] Based on the target distance corresponding to the target first point cloud data and the weight corresponding to the target first point cloud data, the target weighted distance corresponding to the target first point cloud data is calculated.
[0213] In some embodiments, the third processing module 430 may also be used to:
[0214] Constructing an algorithm error function based on the first point cloud, the target point cloud and the target weighted distance corresponding to each first point cloud data;
[0215] Based on the algorithm error function, multiple rotation angles and multiple translation amounts corresponding to the rigid body transformation matrix are solved;
[0216] Based on multiple rotation angles and multiple translation amounts, a rigid body transformation matrix is obtained.
[0217] In some embodiments, the third processing module 430 may also be used to:
[0218] Determine a target rotation angle among the multiple rotation angles as a fixed value, and solve the remaining rotation angles of the target number and multiple translation amounts among the multiple rotation angles except the target rotation angle based on the fixed value;
[0219] When the remaining rotation angle of the target number satisfies the target constraint condition, the function value of the algorithm error function is calculated based on the remaining rotation angle, the target rotation angle and the multiple translation amounts; the remaining rotation angle of the target number satisfies the target constraint condition including: the target number is 0 or the remaining rotation angle satisfies the angle constraint condition;
[0220] The remaining rotation angle and the target rotation angle corresponding to the minimum function value in at least one function value are determined as multiple rotation angles corresponding to the rigid body transformation matrix, and the multiple translation amounts corresponding to the minimum function value are determined as multiple translation amounts corresponding to the rigid body transformation matrix.
[0221] In some embodiments, the fourth processing module 440 may also be used to:
[0222] Transform the first point cloud based on the rigid body transformation matrix to obtain a third point cloud corresponding to the target object;
[0223] When the difference between the third point cloud data in the third point cloud and the target point cloud data corresponding to the third point cloud data in the target point cloud is greater than the target threshold, the third point cloud is determined as the first point cloud, and the rigid body transformation matrix is obtained based on the updated first point cloud until the iteration condition is met, and the second point cloud corresponding to the target object is obtained.
[0224] In some embodiments, the first processing module 410 may also be used to:
[0225] Acquire candidate pose data between initial point cloud data in the initial point cloud and target point cloud data corresponding to the initial point cloud data in the target point cloud; the candidate pose data includes parameter information of transforming the initial point cloud data to the target point cloud data;
[0226] When the rotation angle corresponding to the candidate posture data satisfies the angle constraint condition, the candidate posture data is determined as the target posture data;
[0227] The initial point cloud data is transformed based on the target posture data to obtain a first point cloud.
[0228] In some embodiments, the industrial control method may further include a sixth processing module for calculating the distance in the target direction between the center point of the minimum bounding box corresponding to the initial point cloud and the center point of the minimum bounding box corresponding to the target point cloud when the rotation angles corresponding to the candidate posture data do not satisfy the angle constraint condition before the initial point cloud data is transformed based on the target posture data to obtain the first point cloud;
[0229] The distance is determined as the translation amount of the target posture data in the target direction, and the rotation angle corresponding to the target posture data is determined as 0.
[0230] The industrial control device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
[0231] The industrial control device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.
[0232] The industrial control device provided in the embodiment of the present application can achieve Figures 1 to 3 To avoid repetition, the various processes implemented by the method embodiment are not described here.
[0233] In some embodiments, Figure 5 As shown, an embodiment of the present application also provides an electronic device 500, including a processor 501, a memory 502, and a computer program stored in the memory 502 and executable on the processor 501. When the program is executed by the processor 501, the various processes of the above-mentioned industrial control method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.
[0234] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.
[0235] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the various processes of the above-mentioned industrial control method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0236] On the other hand, the present application also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the various processes of the above-mentioned industrial control method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0237] On the other hand, an embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned industrial control method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0238] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
[0239] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0240] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution can be essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.
[0241] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. An industrial control method, characterized in that: include: Processing the collected initial point cloud of the target object at the target angle and the target point cloud corresponding to the target object to obtain a first point cloud corresponding to the target object; Calculating a target weighted distance between first point cloud data in the first point cloud and target point cloud data corresponding to the first point cloud data in the target point cloud; Based on the target weighted distance corresponding to each of the first point cloud data, a rigid body transformation matrix between the first point cloud and the target point cloud is obtained by solving; Processing the first point cloud based on the rigid body transformation matrix to obtain a second point cloud corresponding to the target object; Based on the second point cloud, control the grabbing of the target object.
2. The industrial control method according to claim 1, characterized in that: The step of calculating a target weighted distance between first point cloud data in the first point cloud and target point cloud data corresponding to the first point cloud data in the target point cloud includes: Calculating a target distance between a target first point cloud data in a plurality of the first point cloud data and a plane where target point cloud data corresponding to the target first point cloud data are located; Calculate the weight corresponding to the first target point cloud data based on the target distance corresponding to the first target point cloud data and the median of the target distances corresponding to the plurality of first point cloud data; Based on the target distance corresponding to the target first point cloud data and the weight corresponding to the target first point cloud data, a target weighted distance corresponding to the target first point cloud data is calculated.
3. The industrial control method according to claim 1, characterized in that: The step of solving a rigid body transformation matrix between the first point cloud and the target point cloud based on the target weighted distance corresponding to each of the first point cloud data includes: Constructing an algorithm error function based on the first point cloud, the target point cloud and the target weighted distance corresponding to each of the first point cloud data; Based on the algorithm error function, multiple rotation angles and multiple translation amounts corresponding to the rigid body transformation matrix are obtained by solving; The rigid body transformation matrix is obtained based on the multiple rotation angles and the multiple translation amounts.
4. The industrial control method according to claim 3, characterized in that: The method of solving the plurality of rotation angles and the plurality of translation amounts corresponding to the rigid body transformation matrix based on the algorithm error function includes: Determine a target rotation angle among the multiple rotation angles as a fixed value, and solve the remaining rotation angles of a target number other than the target rotation angle among the multiple rotation angles and the multiple translation amounts based on the fixed value; In the case where the remaining rotation angle of the target number satisfies the target constraint condition, the function value of the algorithm error function is calculated based on the remaining rotation angle, the target rotation angle and the multiple translation amounts; the remaining rotation angle of the target number satisfies the target constraint condition including: the target number is 0 or the remaining rotation angle satisfies the angle constraint condition; The remaining rotation angle and the target rotation angle corresponding to the minimum function value in at least one function value are determined as multiple rotation angles corresponding to the rigid body transformation matrix, and the multiple translation amounts corresponding to the minimum function value are determined as multiple translation amounts corresponding to the rigid body transformation matrix.
5. The industrial control method according to any one of claims 1 to 4, characterized in that: The step of processing the first point cloud based on the rigid body transformation matrix to obtain a second point cloud corresponding to the target object includes: Performing transformation processing on the first point cloud based on the rigid body transformation matrix to obtain a third point cloud corresponding to the target object; In the case where the difference between the third point cloud data in the third point cloud and the target point cloud data corresponding to the third point cloud data in the target point cloud is greater than a target threshold, the third point cloud is determined as the first point cloud, and a rigid body transformation matrix is obtained based on the updated first point cloud until the iteration condition is met, thereby obtaining a second point cloud corresponding to the target object.
6. The industrial control method according to any one of claims 1 to 4, characterized in that: The step of processing the collected initial point cloud of the target object at the target angle and the target point cloud corresponding to the target object to obtain a first point cloud corresponding to the target object includes: Acquire candidate pose data between initial point cloud data in the initial point cloud and target point cloud data corresponding to the initial point cloud data in the target point cloud; the candidate pose data includes parameter information of transforming the initial point cloud data to the target point cloud data; When the rotation angle corresponding to the candidate posture data satisfies the angle constraint condition, determining the candidate posture data as the target posture data; The initial point cloud data is transformed based on the target posture data to obtain the first point cloud.
7. The industrial control method according to claim 6, characterized in that: Before transforming the initial point cloud data based on the target posture data to obtain the first point cloud, the method further includes: When the rotation angles corresponding to the candidate posture data do not satisfy the angle constraint condition, calculating the distance between the center point of the minimum bounding box corresponding to the initial point cloud and the center point of the minimum bounding box corresponding to the target point cloud in the target direction; The distance is determined as the translation amount of the target posture data in the target direction, and the rotation angle corresponding to the target posture data is determined as 0.
8. An industrial control device, characterized in that: include: A first processing module is used to process the collected initial point cloud of the target object at the target angle and the target point cloud corresponding to the target object to obtain a first point cloud corresponding to the target object; A second processing module, configured to calculate a target weighted distance between first point cloud data in the first point cloud and target point cloud data corresponding to the first point cloud data in the target point cloud; A third processing module, configured to obtain a rigid body transformation matrix between the first point cloud and the target point cloud based on the target weighted distance corresponding to each of the first point cloud data; a fourth processing module, configured to process the first point cloud based on the rigid body transformation matrix to obtain a second point cloud corresponding to the target object; A fifth processing module is used to control the grabbing of the target object based on the second point cloud.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the industrial control method according to any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the industrial control method according to any one of claims 1 to 7 is implemented.
11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the industrial control method according to any one of claims 1 to 7 is implemented.