Flexible material identification methods, devices, equipment and storage media

By using spring diagram non-rigid body registration and multi-view consistency constraints, the system identifies the area of ​​minimum deformation of flexible materials for grasping, solving the problem that traditional 3D vision grasping technology cannot adapt to flexible materials, and achieving efficient and low-damage grasping results.

CN122694945BActive Publication Date: 2026-11-17SHENZHEN JIZHI INTELLIGENT TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202611185222.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2026-08-06
Publication Date
2026-11-17
Estimated Expiration
2046-08-06

AI Technical Summary

Technical Problem

Existing 3D vision grasping technology cannot effectively identify and grasp flexible materials because traditional algorithms assume that the material is a rigid body and cannot adapt to the deformation and complex posture of flexible materials.

Method used

The spring diagram non-rigid body registration method is adopted. By constructing an energy function and multi-view consistency constraints, the flexible material is allowed to undergo local bending deformation during the registration process. The region with the smallest deformation is identified as the gripping position, and the gripping direction is determined.

Benefits of technology

It improves the accuracy of gripping flexible materials, reduces material damage, and solves the problem of failure of traditional rigid ICP registration on flexible materials.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN122694945B_ABST
    Figure CN122694945B_ABST
Patent Text Reader

Abstract

This invention relates to the field of 3D visual recognition and discloses a method for recognizing flexible materials, comprising: acquiring a measured point cloud of the flexible material; constructing an energy function, which includes a registration deviation term, a tensile penalty term, a bending penalty term, and a multi-view consistency constraint term; modeling the template point cloud of the flexible material as a spring diagram, and registering the non-rigid spring diagram to the measured point cloud of the flexible material by minimizing the energy function to obtain a registration result; obtaining a 3D model of the flexible material based on the registration result and the measured point cloud; and identifying the degree of local deformation of the flexible material based on the 3D model and selecting the region with the smallest deformation as the grasping position. This invention enables accurate recognition and grasping of flexible materials.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of 3D visual recognition, and more particularly to methods, apparatus, equipment and storage media for flexible material recognition. Background Technology

[0002] In packaging production lines, the automated handling of flexible materials (such as network cables, data cables, and power cords) faces unique 3D recognition challenges. Unlike rigid materials, flexible materials exhibit highly irregular shapes in their disordered stacking state, such as bending, tangling, crossing, and drooping. Traditional point cloud matching algorithms based on the assumption of rigid materials (such as ICP iterative nearest point) completely fail. Taking network cables as an example, the posture of the same batch of materials in the basket may have hundreds of variations, and the stacking state of the remaining materials changes after each piece is removed.

[0003] Existing 3D vision grasping technologies are mostly designed for rigid materials, assuming that the material is a rigid body with no deformation tolerance. When applied to flexible materials, they cannot effectively identify and grasp flexible materials. Summary of the Invention

[0004] This invention provides a flexible material identification method, apparatus, device, and storage medium to overcome the shortcomings of existing methods that cannot accurately identify flexible materials, and to achieve accurate identification and grasping of flexible materials.

[0005] In a first aspect, embodiments of the present invention provide a flexible material identification method, comprising: Collect measured point clouds of flexible materials; Construct an energy function, which includes a registration deviation term, a stretching penalty term, a bending penalty term, and a multi-view consistency constraint term; The template point cloud of the flexible material is modeled as a spring diagram. By minimizing the energy function, the non-rigid spring diagram is registered to the measured point cloud of the flexible material to obtain the registration result. Based on the registration results and measured point cloud, a three-dimensional model of the flexible material is obtained; Based on the three-dimensional model, the degree of local deformation of the flexible material is identified, and the area with the smallest deformation is selected as the gripping position, including: based on the three-dimensional model, identifying the end position of the flexible material as a candidate gripping point; Obtain the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface; The degree of local deformation of the candidate gripping point is calculated based on the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface. The candidate gripping point with the smallest local deformation is selected as the gripping position, and the tangent direction of the gripping position is determined as the gripping direction.

[0006] In a second aspect, embodiments of the present invention provide a flexible material identification device, comprising: The data acquisition module is used to acquire measured point clouds of flexible materials. The building module is used to construct the energy function, which includes a registration deviation term, a stretching penalty term, a bending penalty term, and a multi-view consistency constraint term. The registration module is used to model the template point cloud of flexible material into a spring diagram. By minimizing the energy function, the non-rigid spring diagram is registered to the measured point cloud of the flexible material to obtain the registration result. The acquisition module is used to acquire a three-dimensional model of the flexible material based on the registration results and the measured point cloud. The identification module is used to identify the degree of local deformation of flexible materials based on a three-dimensional model and select the area with the smallest deformation as the gripping position, including: identifying the end position of the flexible material based on the three-dimensional model as a candidate gripping point; Obtain the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface; The degree of local deformation of the candidate gripping point is calculated based on the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface. The candidate gripping point with the smallest local deformation is selected as the gripping position, and the tangent direction of the gripping position is determined as the gripping direction.

[0007] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the flexible material identification method described above.

[0008] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described flexible material identification method.

[0009] The aforementioned flexible material identification method, device, equipment, and storage medium, through spring diagram non-rigid body registration, allow flexible materials to undergo local bending deformation during the registration process, solving the problem of complete failure of traditional rigid ICP (Iterative Closest Point Algorithm) registration on flexible materials and improving the accuracy of flexible material grasping; by selecting the area with the smallest deformation as the grasping position through the degree of local deformation, it avoids clamping in the drooping and bending area of ​​the flexible material, which can reduce the damage to the flexible material during grasping. Attached Figure Description

[0010] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0011] Figure 1 This is a flowchart of a flexible material identification method according to an embodiment of the present invention; Figure 2 yes Figure 1 A flowchart of step S105; Figure 3 yes Figure 1 A flowchart of step S103; Figure 4 This is a flowchart of a flexible material identification device according to an embodiment of the present invention; Figure 5 This is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0012] To make the technical problems solved, the technical solutions, and the beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0013] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0015] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0016] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0017] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0018] In one embodiment, please refer to Figure 1 A flexible material identification method is provided, comprising the following steps: Step S101: Collect the measured point cloud of the flexible material; Step S102: Construct the energy function, which includes a registration deviation term, a stretching penalty term, a bending penalty term, and a multi-view consistency constraint term; Step S103: Model the template point cloud of the flexible material as a spring diagram. By minimizing the energy function, register the non-rigid spring diagram to the measured point cloud of the flexible material to obtain the registration result. Step S104: Based on the registration results and the measured point cloud, obtain a three-dimensional model of the flexible material; Step S105: Based on the 3D model, identify the degree of local deformation of the flexible material and select the area with the smallest deformation as the gripping position; including: Step S1051: Based on the 3D model, identify the end positions of the flexible material as candidate gripping points; Step S1052: Obtain the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface; Step S1053: Calculate the degree of local deformation of the candidate gripping point based on the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface; Step S1054: Select the candidate gripping point with the smallest local deformation as the gripping position, and determine the tangent direction of the gripping position as the gripping direction.

[0019] The measured point cloud is point cloud data obtained by capturing images of flexible materials in actual grasping scenarios using a 3D camera. Unlike the template point cloud, the measured point cloud reflects the actual shape of the flexible material at the current moment, including complex deformation states such as bending, entanglement, stacking, and sagging. The template point cloud is a 3D point cloud of the flexible material in a standard shape, serving as the initial reference for subsequent non-rigid body registration.

[0020] As an example, in step S101, measured point clouds of the flexible material are acquired from different perspectives using multiple cameras. These measured point clouds are not single multi-view point clouds, but rather collections of point clouds corresponding to multiple perspectives, with each perspective independently providing surface information of the flexible material in that direction. In this embodiment, four Orbbec Gemini series 3D cameras are used, including two Gemini 305 wide-angle cameras and two Gemini 335 high-precision cameras (one installed as a hand-eye camera at the robot's end effector, and the other as a global camera installed above the material basket). Point cloud data of the flexible material is acquired simultaneously from four different directions. Because flexible materials (such as network cables, data cables, power cables, etc.) have highly irregular shapes when stacked in a disordered state, a single camera cannot completely acquire all surface information of the material, resulting in occlusion and blind spots. Multi-camera acquisition can effectively compensate for the information loss from a single perspective, effectively eliminate occlusion blind spots under a single perspective, provide sufficient surface information for subsequent non-rigid body registration, and improve registration accuracy and robustness.

[0021] The energy function is used to quantify the degree of deformation of the spring diagram during the registration process. The smaller the value of the energy function, the more reasonable the deformation of the spring diagram and the closer it is to the measured point cloud.

[0022] As an example, in step S102, an energy function for non-rigid body registration is constructed. This energy function includes four terms: registration deviation term, stretching penalty term, bending penalty term, and multi-view consistency constraint term. The registration deviation term constrains the spatial distance between vertices on the spring diagram and corresponding points in the measured point cloud, ensuring that the registered template point cloud fits the measured point cloud. A smaller registration deviation indicates a higher degree of matching between the template and the measured flexible material. The stretching penalty term constrains the distance variation between adjacent vertices in the spring diagram. When the template point cloud deforms to fit the measured point cloud, the distance between adjacent vertices should not be excessively stretched or compressed, thus maintaining the local geometry of the flexible material. The bending penalty term constrains the range of local curvature variation. Although flexible materials can be bent, the degree of bending should conform to the physical properties of the flexible material, and unreasonable bending should not occur. The multi-view consistency constraint term constrains the consistency of the point clouds of the same flexible material under multiple camera views in a unified coordinate system, ensuring that the same flexible material region observed by each of the four cameras can match each other after registration, eliminating inconsistencies between multiple views.

[0023] By constructing a multi-element energy function that includes registration bias, tension penalty, bending penalty, and multi-view consistency constraints, multi-dimensional constraints on the deformation of flexible materials are achieved. Compared with the rigid ICP method in the prior art that only considers registration bias, this invention allows flexible materials to undergo local bending deformation during the registration process. At the same time, the tension and bending penalty terms prevent excessive deformation, making the registration result more consistent with the actual physical properties of the flexible material. The multi-view consistency constraint term ensures the consistency of multi-camera data fusion and avoids registration failure caused by multi-view data conflicts.

[0024] The template point cloud is a 3D point cloud of the flexible material in its standard form, serving as the initial reference for registration. The spring diagram is a deformable model obtained by modeling the template point cloud as an elastic diagram structure. Non-rigid registration uses the spring diagram constructed after downsampling the template point cloud as the initial form, and the measured point cloud as the target. By minimizing the energy function, the positions of each vertex on the spring diagram are iteratively optimized. The registration result refers to the final set of positions reached by each vertex on the spring diagram after optimization, that is, the state of the template point cloud after non-rigid registration fitting onto the measured point cloud. The registration result contains the actual shape information of the flexible material in the grasping scenario—the new positions of each vertex reflect the bending, twisting, and other deformation states of the material in space.

[0025] In one embodiment, in step S103, the template point cloud is modeled as a spring diagram. By minimizing the energy function, the non-rigid spring diagram is registered to the measured point cloud to obtain the registration result. In this embodiment, the template point cloud is 3D scan data of a single cable in its natural straight state, serving as the initial reference shape for registration. The template point cloud has complete geometric information of the flexible material surface, but does not include bending deformations in the actual grasping scenario. Furthermore, the template point cloud is not fixed during the registration process, but rather serves as the initial shape of the spring diagram, gradually deforming through energy function optimization to fit the measured point cloud.

[0026] Among them, the three-dimensional model is the three-dimensional model of the flexible material in the world coordinate system obtained after registration. This three-dimensional model contains all the geometric information of the surface of the flexible material—including bending shape, spatial orientation, local curvature changes, etc., and is a complete digital description of the current actual shape of the flexible material.

[0027] As an example, in step S104, after registration, the vertices on the spring diagram have deformed to fit the measured point cloud. At this point, each vertex is fused with the measured point cloud data collected by the four cameras to obtain a complete 3D model of the flexible material in a unified world coordinate system. By fusing the registered deformed template with the measured point cloud, a complete 3D model of the flexible material is obtained, providing a comprehensive geometric basis for subsequent gripping point selection. Four-camera fusion effectively eliminates single-view occlusion and blind spots, enabling the acquisition of the complete 3D shape of the flexible material even in scenarios with severe material stacking.

[0028] Among them, the degree of local deformation is an indicator used to quantify the deformation intensity of different regions on a three-dimensional model.

[0029] As an example, in step S105, the degree of local deformation of the flexible material is obtained based on the three-dimensional model, and the area with the smallest deformation is selected as the gripping position. For flexible materials, the degree of deformation varies greatly in different regions—the two ends usually have smaller deformation and regular shape, while the middle area often sags and bends severely due to gravity. The larger the deformation, the easier it is to slip or damage the material when gripping. Therefore, the area with the smallest deformation should be selected for gripping first.

[0030] Please see Figure 2 As an example, step S105, based on the 3D model, identifies the degree of local deformation of the flexible material and selects the area with the smallest deformation as the gripping position, including: Step S1051: Based on the 3D model, identify the end positions of the flexible material as candidate gripping points; Step S1052: Obtain the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface; Step S1053: Calculate the degree of local deformation of the candidate gripping point based on the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface; Step S1054: Select the candidate gripping point with the smallest local deformation as the gripping position, and determine the tangent direction of the gripping position as the gripping direction.

[0031] Among these, the end positions are the two ends of the flexible material, which usually have the least deformation and the most regular shape, making them the most suitable positions for gripping. Candidate gripping points refer to potential gripping locations on the 3D model.

[0032] As an example, in step S1051, after obtaining the complete three-dimensional model of the flexible material, the three-dimensional model is first analyzed to identify the locations of the two ends of the flexible material. In this embodiment, for flexible materials with obvious ends, such as network cables, the two ends usually have the smallest deformation and the most regular shape, so the two end locations are used as candidate gripping points. The identification of the end locations can be achieved through diameter abrupt change detection (obvious diameter change at the network cable connector) or color marking identification (cable ends usually have color markings). It should be noted that if the flexible material does not have obvious ends (such as ring-shaped materials), the candidate gripping points can also be selected from other areas with small local curvature on the three-dimensional model. This embodiment uses the ends as an example for illustration, but it is not a limitation on the range of candidate gripping points.

[0033] The local curvature describes the degree of bending at the candidate gripping point. The support surface is the bearing plane on which the flexible material is placed. The distance from the candidate gripping point to the support surface refers to the perpendicular distance from that point to the support surface along the direction of gravity in three-dimensional space. This distance quantifies the degree of sagging caused by gravity in that area: the smaller the distance, the closer the area is to the support surface, the less affected by gravity, and the smaller the deformation; the larger the distance, the more severe the suspension in the area, and the more significant the sagging deformation. For points located on the support surface, this distance is zero.

[0034] As an example, in step S1052, after determining the candidate gripping points, two key deformation evaluation indicators are calculated for each candidate gripping point: the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface. Specifically, on the 3D model, with the candidate gripping point p as the center, several points within its neighborhood are fitted to the surface, and the average curvature of the point is calculated. The larger the curvature value, the more severe the bending and deformation in the region; the smaller the curvature value, the closer the region is to a straight state, and the less severe the deformation. For the ends of the mesh, since the ends are usually relatively straight, their local curvature is generally small. More specifically, based on the spatial relationship between the 3D model and the support surface, the vertical distance from the candidate gripping point to the support surface along the direction of gravity is calculated. The smaller the distance value, the closer the region is to the support surface, the less affected by gravity, and the smaller the deformation; the larger the distance value, the more severe the suspension and the more significant the sagging deformation in the region.

[0035] This embodiment simultaneously acquires two indicators: local curvature and distance to the support surface. It comprehensively evaluates the deformation state of candidate gripping points from two dimensions: the degree of bending and the degree of sag, avoiding the one-sidedness of evaluation based on a single indicator. For example, although an end may have a small curvature, if it is far from the support surface, it is still unsuitable as a gripping point; conversely, although a region may be close to the support surface, if its curvature is large, it is also unsuitable for gripping. The two indicators complement each other, making the deformation assessment more comprehensive and reasonable.

[0036] As an example, in step S1053, the local curvature obtained in step S1052 and the distance from the candidate residential point to the support surface are fused to obtain the degree of local deformation of the candidate grab point. The calculation formula is as follows: Deformation(p) = κ(p) + h(p) Where κ(p) is the local curvature of the candidate gripping point P, and h(p) is the distance from the candidate gripping point to the support surface.

[0037] As an example, in step S1054, after calculating the degree of local deformation of all candidate gripping points, the degree of local deformation of each candidate gripping point is compared, and the candidate gripping point with the smallest degree of local deformation is selected as the gripping position. After determining the gripping position, the gripping direction also needs to be determined. The gripping direction is determined along the tangent direction of the material at that gripping position. The tangent direction reflects the local orientation of the flexible material at that point. When the grippers hold the material along the tangent direction, the contact between the grippers and the material surface is most uniform, the gripping force distribution is most reasonable, and this effectively reduces bending moments and lowers the risk of damage to the flexible material.

[0038] Please see Figure 3 In one embodiment, step S103, which involves modeling the template point cloud of the flexible material as a spring diagram and registering the non-rigid spring diagram to the measured point cloud of the flexible material by minimizing the energy function, includes: Step S1031: Downsample the template point cloud of the flexible material into several vertices, and construct a spring graph by connecting the K nearest neighbors of each vertex; Step S1032: With the goal of minimizing the energy function, the spring diagram non-rigid body is registered to the multi-view measured point cloud of the flexible material by iteratively updating each vertex.

[0039] As an example, in step S1031, the template point cloud is 3D scan data of a single cable in its naturally straight state, containing tens of thousands of dense 3D coordinate points. Directly using all points for registration and optimization would be computationally intensive and fail to meet the real-time requirements of the production line. This embodiment first uniformly downsamples the template point cloud, selecting 200 representative points as vertices. These vertices are evenly distributed on the material surface, enabling a complete description of the overall shape and geometric features of the flexible material with a smaller amount of data. After downsampling, a spring graph structure G=(V,E) is constructed using these 200 vertices. The vertex set V consists of the 200 downsampled vertices, each representing a local location on the flexible material surface. The edge set E is obtained by connecting each vertex to its four closest Euclidean distance vertices using the K-nearest neighbor algorithm (K=4).

[0040] Specifically, each edge is equivalent to a virtual spring, and the original Euclidean distance between adjacent vertices is the natural length of the spring. The resulting spring graph has the following characteristics: when a vertex is displaced under external force, the connected springs generate a restoring force, pulling the vertex back from excessive deviation, thus ensuring smooth, continuous, and reasonable overall deformation. The natural length of the spring determines the "most comfortable" relative distance between vertices; any deviation will incur a penalty in the energy function. This embodiment uses K=4 because: 4-nearest neighbor connections ensure the connectivity of the graph structure while avoiding excessive computational complexity due to too many edges, and 4-nearest neighbors are sufficient to describe the local geometric topology of the material surface.

[0041] This embodiment achieves a balance between registration accuracy and computational efficiency by constructing a spring graph through downsampling and K-nearest neighbor connections. Downsampling to 200 vertices significantly reduces the number of variables involved in optimization, resulting in a substantial improvement in computational efficiency. Simultaneously, the even distribution of 200 vertices is sufficient to describe the overall bending morphology and local geometric features of flexible materials, preventing the loss of crucial shape information due to an insufficient number of vertices. The K=4 adjacency relationship ensures graph connectivity and local topological description capabilities while avoiding the optimization burden caused by an excessive number of edges.

[0042] As an example, in step S1032, after the spring diagram is constructed, using the current form of the flexible material as the initial state and the measured point cloud as the target, the spatial position of each vertex on the spring diagram is iteratively optimized by minimizing the energy function. This embodiment uses the Gauss-Newton iterative method. Specifically, in each iteration, the gradient direction of the current energy function is calculated, and the position of each vertex is updated along the gradient descent direction, gradually reducing the total energy E. The iteration terminates when the energy change is less than 0.01% or the number of iterations exceeds 50. After the iteration is complete, each vertex reaches its final position, thus obtaining the registration result.

[0043] By downsampling mapping and energy function-driven registration, standard template point clouds are adapted to the actual deformation state of flexible materials. The spring diagram structure balances deformation flexibility and shape retention, enabling the registration process to adapt to various complex deformation states of flexible materials while avoiding unreasonable deformation. The final registration results accurately describe the bending trajectory and local geometric features of the flexible material in space, providing a reliable data foundation for subsequent 3D model reconstruction and gripping point optimization.

[0044] In one embodiment, the energy function is: E = + α· + β· + γ· in, For the registration deviation term, , The first point cloud in the template i The position of each point The first measured point cloud corresponding to the template point cloud i The position of each point For the registered template point cloud, the first i The position of each point; For stretching penalty items, , The first point cloud in the template j One point, For the registered template point cloud, the first j The position of each point; For the bending penalty item, , The first point cloud in the template i The local curvature angle at each point, For the registered template point cloud, the first i Local curvature angle at each point; For multi-perspective consistency constraints, , For the c-th camera view, the template point cloud is the first... i The position of the points after registration. It is the first i The position of a point in the world coordinate system This represents the transformation from the c-th camera to the world coordinate system; α, β, and γ are weighting coefficients.

[0045] As an example, the energy function includes a registration deviation term, a stretching penalty term, a bending penalty term, and a multi-view consistency constraint. α is 0.5, β is 0.3, and γ is 0.2. The registration deviation term has the highest weight to ensure fitting accuracy as the primary goal. The stretching and bending penalty terms prevent excessive deformation. The multi-view consistency constraint term coordinates the data acquired by multiple cameras.

[0046] In one embodiment, step S1054, selecting the candidate gripping point with the smallest local deformation as the gripping location, includes: Candidate grab points with local curvature greater than the curvature threshold are removed, and the candidate grab point with the smallest deformation evaluation value is selected from the remaining candidate grab points as the grab position.

[0047] The curvature threshold is a preset numerical limit used to determine whether a certain area on a 3D model is unsuitable as a gripping location due to excessive curvature.

[0048] As an example, in step S1054, candidate gripping points with local curvature greater than a curvature threshold are removed from the candidate gripping points, and the candidate gripping point with the smallest deformation evaluation value is selected as the gripping position from the remaining candidate gripping points. By setting a curvature threshold to pre-remove candidate gripping points with severe bending, gripping in excessively bent areas is effectively prevented, further reducing the gripping damage rate.

[0049] In one embodiment, before obtaining the three-dimensional model of the flexible material based on the registration results and the measured point cloud in step S104, the method further includes: Step S001: Calculate the recognition confidence based on the registration results; Step S002: If the identification confidence level is greater than or equal to the preset threshold, proceed to the step of obtaining a three-dimensional model of the flexible material based on the registration result and the measured point cloud. After calculating the recognition confidence based on the registration results, the following steps are also included: Step S003: If the recognition confidence level is less than the preset threshold, then proceed to the step of collecting the measured point cloud of the flexible material from different perspectives using multiple cameras.

[0050] Among them, the identification confidence level is a quantitative indicator used to measure the reliability of the registration result, reflecting the accuracy of this registration.

[0051] As an example, in step S001, after completing the non-rigid body registration of the spring diagram, the registration result is obtained, namely the final position of each vertex after deformation and the values ​​of each component of the energy function. The registration deviation term reflects the overall fit deviation between the spring diagram and the measured point cloud. Specifically, the recognition confidence is calculated based on the registration deviation term, using the following formula: Confidence = Where N is the number of vertices in the spring diagram, used to calculate the average registration error per vertex; If Confidence < 0.7, mark it as "low confidence"; This is the average registration error for each vertex; T is a preset threshold for the confidence level of identification.

[0052] As an example, in step S002, the calculated recognition confidence score is compared with a preset threshold. If the recognition confidence score is greater than or equal to the preset threshold, it indicates that the registration error is within an acceptable range, and the quality of the registration result meets the requirements for subsequent processing. At this point, the registration result is determined to be reliable, and the process proceeds to the step of obtaining a 3D model of the flexible material based on the registration result and the measured point cloud. By setting a preset threshold for the recognition confidence score as a quality threshold for entering subsequent steps, it ensures that only registration results that meet certain quality requirements are used for 3D model reconstruction and grasp point selection. This screening mechanism effectively prevents low-quality registration results from being passed downstream, avoids making grasping decisions based on erroneous data, and improves the overall reliability of the system.

[0053] Furthermore, the setting of the preset threshold needs to balance recognition accuracy and production line efficiency: Setting the threshold too high (e.g., 0.9), while ensuring high-quality registration results for subsequent steps, will lead to a large number of registrations being judged as unqualified and triggering retakes, reducing production line efficiency; setting the threshold too low (e.g., 0.5), although reducing the retake rate, may result in low-quality registration results leading to large deviations in the 3D model, thus affecting the success rate of capture. In this embodiment, a threshold of 0.7 is chosen as a balance between registration quality and production line efficiency, but it can also be selected according to actual needs.

[0054] As an example, in step S003, if the confidence level is less than the preset threshold, it indicates that the registration error is too large and the reliability of the registration result is insufficient. Possible reasons for registration failure include: the material stacking is too complex, leading to multi-view data conflicts; poor lighting conditions during shooting, resulting in a decrease in point cloud quality; and material deformation exceeding the adaptability range of the spring diagram. In this case, instead of proceeding to step S104, step S101 is re-executed, with four cameras again acquiring the measured point cloud of the flexible material from different perspectives, and the subsequent non-rigid body registration process is re-executed. If the confidence level is still lower than the preset threshold after multiple re-shoots, the process can be switched to manual confirmation mode, whereby a person manually judges the current state of the material basket and decides whether to intervene.

[0055] In this embodiment, a re-shooting mechanism is automatically triggered when the registration quality fails to meet the standards, enabling the system to self-correct. Compared to directly using low-quality registration results for subsequent processing, the re-shooting mechanism avoids invalid capture based on erroneous data, reducing capture failures and material damage. Furthermore, the re-shooting mechanism is fully automated, requiring no manual intervention, and only switches to manual confirmation mode after multiple consecutive failures, maximizing the automated operation efficiency of the production line.

[0056] In one embodiment, after obtaining the three-dimensional model of the flexible material based on the registration results and the measured point cloud in step S104, the method further includes: Step S004: Based on the three-dimensional model, calculate the principal axis direction and center of gravity position of the flexible material through principal component analysis to obtain the pose data of the flexible material.

[0057] Principal Component Analysis (PCA) is a statistical method used to identify the direction of maximum variance in a dataset.

[0058] As an example, in step S004, after obtaining the complete 3D model of the flexible material, principal component analysis (PCA) is performed on all point cloud data in the 3D model to extract the position data of the flexible material. Specifically, for 3D point cloud data, PCA can calculate three mutually orthogonal principal directions. The first principal direction corresponds to the direction with the largest variance in the point cloud distribution, which is the direction in which the flexible material extends the longest in 3D space, and is called the principal axis direction of the material. For slender flexible materials like meshes, the principal axis direction is the overall direction of the material from one end to the other. The second principal direction corresponds to the direction with the second largest variance in the point cloud distribution and is orthogonal to the first principal direction, and is called the secondary axis direction of the material. The third principal direction corresponds to the direction with the smallest variance in the point cloud distribution and is orthogonal to both the first and second principal directions, usually corresponding to the thickness direction of the material. The centroid position is the geometric center coordinates (x, y, z) of all points in the 3D model, obtained by averaging the coordinates of all points. Thus, the pose data of the flexible material can be obtained: position data: centroid coordinates (x, y, z); attitude data: determined by the first and second principal directions. It should be noted that this pose data describes the overall spatial state of the flexible material, not the precise pose of the grasping point. Because flexible materials exhibit localized bending and deformation, their overall principal axis direction only reflects the general orientation of the material and cannot be directly used for precise grasping. The main purpose of this pose data is to provide the robot with the approximate position and orientation of the flexible material in space, guiding the robot's end effector to move near the material and providing a coarse positioning reference for subsequent precise grasping based on the grasping point.

[0059] In this embodiment, principal component analysis (PCA) is used to obtain the pose data of the flexible material, providing the robot with overall spatial distribution information of the material. This pose data serves as a coarse positioning guide, enabling the robot to quickly move its end effector to the vicinity of the flexible material, narrowing the search range for the gripping point and improving gripping efficiency. Simultaneously, this pose data can also be used for preliminary planning of the robot's gripping path, helping the robot avoid the edges of the material basket or other obstacles. PCA calculations are simple and efficient; a 3D model with 200 vertices can be completed in milliseconds, without placing an additional burden on the production line cycle time.

[0060] The embodiments of the present invention have the following beneficial effects: (1) The spring diagram non-rigid body registration method is adopted to model the template point cloud as a spring diagram with elastic constraints. The deformation of the spring diagram is driven by an energy function that includes registration deviation, stretching penalty, bending penalty and multi-view consistency constraint, so that the material can undergo local bending deformation during the registration process. This solves the fundamental defect of the traditional rigid ICP registration that assumes the material is rigid and has no deformation.

[0061] (2) The measured point cloud of flexible material is collected synchronously from multiple perspectives by four 3D cameras (two Gemini 305 wide-angle cameras + two Gemini 335 high-precision cameras). A multi-view consistency constraint term is introduced into the energy function to ensure that the same material area observed by the four cameras can match each other after registration. Compared with the single-view scheme, the four-camera fusion effectively eliminates the occlusion and blind spot problems under the single view.

[0062] (3) By identifying the end position on the three-dimensional model of the flexible material, the local curvature of each candidate gripping point and the distance to the support surface are calculated. The degree of local deformation of each region is comprehensively evaluated, and a curvature threshold is set to remove areas with severe bending. Finally, the region with the smallest deformation is selected as the gripping position, and the gripping direction is determined along the tangent direction of that point. Compared with the traditional method of using the center of gravity or geometric center of the material as the gripping point, this invention selects the region with the smallest deformation and the most regular shape for gripping, which effectively ensures the integrity of the flexible material during the gripping process.

[0063] Please see Figure 4 In one embodiment, a flexible material identification device is provided, which corresponds one-to-one with the flexible material identification method described in the above embodiments. The flexible material identification device includes: The acquisition module 301 is used to acquire the measured point cloud of flexible materials; Module 302 is used to construct the energy function, which includes a registration deviation term, a stretching penalty term, a bending penalty term, and a multi-view consistency constraint term. The registration module 303 is used to model the template point cloud of the flexible material into a spring diagram, and to register the non-rigid spring diagram to the measured point cloud of the flexible material by minimizing the energy function, thereby obtaining the registration result. The acquisition module 304 is used to acquire a three-dimensional model of the flexible material based on the registration results and the measured point cloud. The recognition module 305 is used to identify the degree of local deformation of flexible materials based on a three-dimensional model and select the area with the smallest deformation as the gripping position. The identification module 305 is also used for: Based on the 3D model, the end positions of the flexible material are identified as candidate gripping points; Obtain the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface; The degree of local deformation of the candidate gripping point is calculated based on the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface. The candidate gripping point with the smallest local deformation is selected as the gripping position, and the tangent direction of the gripping position is determined as the gripping direction.

[0064] Optionally, in one embodiment, the registration module 303 is further configured to: The template point cloud of the flexible material is downsampled into several vertices, and a spring graph is constructed based on the K-nearest neighbor connections between the vertices. With the goal of minimizing the energy function, the spring diagram non-rigid body is registered to the measured point cloud of the flexible material by iteratively updating each vertex.

[0065] Optionally, in one embodiment, the identification module 305 is further configured to: Candidate grab points with local curvature greater than the curvature threshold are removed, and the candidate grab point with the smallest deformation evaluation value is selected from the remaining candidate grab points as the grab position.

[0066] Optionally, in one embodiment, the flexible material identification device further includes an identification confidence module, used to: calculate the identification confidence based on the registration result; If the confidence level is greater than or equal to the preset threshold, then proceed to the step of obtaining a three-dimensional model of the flexible material based on the registration results and the measured point cloud. After calculating the recognition confidence based on the registration results, the following steps are also included: If the confidence level is less than the preset threshold, the process proceeds to the step of collecting measured point clouds of the flexible material from different perspectives using multiple cameras.

[0067] Optionally, in one embodiment, the flexible material recognition device further includes a pose calculation module, which calculates the principal axis direction and center of gravity position of the flexible material through principal component analysis based on the three-dimensional model to obtain the pose data of the flexible material.

[0068] Specific limitations regarding the flexible material identification device can be found in the limitations of the flexible material identification method described above, and will not be repeated here. Each module in the aforementioned flexible material identification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0069] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores measured point clouds, 3D models, etc. The network interface communicates with external terminals via a network. When executed by the processor, the computer program implements a flexible material identification method.

[0070] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the flexible material identification method described in the above embodiment. To avoid repetition, further details are omitted here. The computer-readable storage medium can be non-volatile or volatile.

[0071] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0072] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0073] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A flexible material identification method, characterized in that, include: Collect measured point clouds of flexible materials; Construct an energy function, which includes a registration deviation term, a stretching penalty term, a bending penalty term, and a multi-view consistency constraint term; the energy function is: E = + a· + b; + c· in, For the registration deviation term, , The first point cloud in the template i The position of each point The first measured point cloud corresponding to the template point cloud i The position of each point For the registered template point cloud, the first i The position of each point; For stretching penalty items, , The first point cloud in the template j One point, For the registered template point cloud, the first j The position of each point; For the bending penalty item, , The first point cloud in the template i The local curvature angle at each point, For the registered template point cloud, the first i Local curvature angle at each point; For multi-perspective consistency constraints, , For the c-th camera view, the template point cloud is the first... i The position of the points after registration. It is the first i The position of a point in the world coordinate system This represents the transformation from the world coordinate system to the c-th camera coordinate system; α, β, and γ are weighting coefficients. The template point cloud of the flexible material is downsampled into several vertices, and a spring diagram is constructed using K-nearest neighbor connections between each vertex. With the goal of minimizing the energy function, each vertex is iteratively updated, and the spring diagram is non-rigidly registered to the measured point cloud of the flexible material to obtain the registration result. Based on the registration results and the measured point cloud, a three-dimensional model of the flexible material is obtained; Based on the three-dimensional model, the degree of local deformation of the flexible material is identified, and the area with the smallest deformation is selected as the gripping position, including: Based on the three-dimensional model, the end positions of the flexible material are identified as candidate gripping points; Obtain the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface; The degree of local deformation of the candidate gripping point is calculated based on the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface. The candidate gripping point with the smallest local deformation is selected as the gripping position, and the tangent direction of the gripping position is determined as the gripping direction.

2. The flexible material identification method as described in claim 1, characterized in that, The step of selecting the candidate grasping point with the smallest local deformation as the grasping position includes: Candidate grab points with local curvature greater than the curvature threshold are removed, and the candidate grab point with the smallest local deformation is selected from the remaining candidate grab points as the grab position.

3. The flexible material identification method as described in claim 1, characterized in that, Before obtaining the 3D model of the flexible material based on the registration results and measured point cloud, the following steps are also included: Calculate the recognition confidence level based on the registration results; If the recognition confidence level is greater than or equal to a preset threshold, then proceed to the step of obtaining a three-dimensional model of the flexible material based on the registration result and the measured point cloud. After calculating the recognition confidence based on the registration results, the method further includes: If the recognition confidence level is less than the preset threshold, then proceed to the step of collecting the measured point cloud of the flexible material.

4. The flexible material identification method as described in claim 1, characterized in that, After obtaining the three-dimensional model of the flexible material based on the registration result and the measured point cloud, the process further includes: Based on the three-dimensional model, the principal axis direction and center of gravity position of the flexible material are calculated by principal component analysis to obtain the pose data of the flexible material.

5. A flexible material identification device, characterized in that, include: The data acquisition module is used to acquire measured point clouds of flexible materials. A construction module is used to construct an energy function, which includes a registration deviation term, a stretching penalty term, a bending penalty term, and a multi-view consistency constraint term; the energy function is: E = + a· + b; + c· in, For the registration deviation term, , The first point cloud in the template i The position of each point The first measured point cloud corresponding to the template point cloud i The position of each point For the registered template point cloud, the first i The position of each point; For stretching penalty items, , The first point cloud in the template j One point, For the registered template point cloud, the first j The position of each point; For the bending penalty item, , The first point cloud in the template i The local curvature angle at each point, For the registered template point cloud, the first i Local curvature angle at each point; For multi-perspective consistency constraints, , For the c-th camera view, the template point cloud is the first... i The position of the points after registration. It is the first i The position of a point in the world coordinate system This represents the transformation from the world coordinate system to the c-th camera coordinate system; α, β, and γ are weighting coefficients. The registration module is used to downsample the template point cloud of the flexible material into several vertices, construct a spring diagram by connecting the K nearest neighbors between each vertex, and register the spring diagram to the measured point cloud of the flexible material by iteratively updating each vertex with the goal of minimizing the energy function, thereby obtaining the registration result. The acquisition module is used to acquire a three-dimensional model of the flexible material based on the registration result and the measured point cloud. The recognition module is used to identify the degree of local deformation of the flexible material based on the three-dimensional model, and select the area with the smallest deformation as the gripping position, including: Based on the three-dimensional model, the end positions of the flexible material are identified as candidate gripping points; Obtain the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface; The degree of local deformation of the candidate gripping point is calculated based on the local curvature of the candidate gripping point and the distance from the candidate gripping point to the support surface. The candidate gripping point with the smallest local deformation is selected as the gripping position, and the tangent direction of the gripping position is determined as the gripping direction.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the flexible material identification method according to any one of claims 1-4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the flexible material identification method as described in any one of claims 1-4.

Citation Information

Patent Citations

  • Method and system for tracking deformation of flexible object by using RGBD camera

    CN114155256A

  • Mining area earth surface three-dimensional movement extraction method and system based on improved CPD non-rigid point cloud registration

    CN122115515A