A method and storage device for extracting glue coating trajectory from 3D vision of shoe sole

By determining the starting point and end point of the sole segmentation in the 3D visual extraction of the glue coating trajectory method, the three-dimensional coordinate system, projection and curvature fitting are established, the problem of low glue coating accuracy in the existing technology is solved, and the accuracy of the sole and upper is achieved Accurate acquisition of bonding and glue coating trajectory.

CN114049366BActive Publication Date: 2024-11-22FUJIAN YILI INTELLIGENT TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111261784.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-28
Publication Date
2024-11-22
Estimated Expiration
2041-10-28

AI Technical Summary

Technical Problem

In the prior art, when the highest point of the sole side wall is directly used as the glue coating track point, the glue coating accuracy is not high, which affects the fit between the sole and the upper.

Method used

The 3D visual extraction and coating trajectory method of sole is used to determine the starting point and end point of the sole, establish a three-dimensional coordinate system, and make a tangent plane along the x-axis direction, project the highest point of the shoe wall, and obtain the curvature of the point cloud within the preset range. Divide the fit and take the point with the largest change rate as the fit point between the sole and the upper.

Benefits of technology

Accurately determine the fit point between the sole and the upper, and then obtain an accurate glue application trajectory, improving the accuracy of glue application.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114049366B_ABST
    Figure CN114049366B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of gluing technology, and in particular to a method and storage device for extracting gluing trajectories from a 3D sole visual system. The method comprises the following steps: determining a segmentation start point and a segmentation end point of the sole; establishing a three-dimensional coordinate system, and leveling all sole point clouds to the extent that the length direction of the shoe is parallel to the x-axis of the three-dimensional coordinate system, the width direction of the shoe is parallel to the y-axis of the three-dimensional coordinate system, and the height direction of the shoe is parallel to the z-axis of the three-dimensional coordinate system; making a cutting plane from the segmentation start point to the segmentation end point along the x-axis direction, and obtaining the highest point of the shoe wall by the projection of the cutting plane on the yz plane; obtaining a preset number of point clouds in a preset range area of ​​the highest point of the shoe wall, performing curvature division and fitting of the preset number of point clouds into n segments, obtaining the change rate of each segment, and taking the point with the largest change rate as the fitting point of the sole and the upper. By means of the above method, the true fitting point of the sole and the upper can be determined, and an accurate gluing trajectory can be obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of gluing technology, and in particular to a method for extracting gluing tracks from a shoe sole using 3D vision and a storage device. Background Art

[0002] As people's living standards improve, their clothing becomes more diverse, including the styles of shoes. The highest point of the side wall of the sole is not entirely the fitting point between the sole and the upper. Using the highest point as the gluing trajectory point will have a greater impact on the accuracy of gluing. How to obtain the accurate fitting point for accurate gluing has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] Therefore, it is necessary to provide a method for extracting the gluing trajectory of the sole by 3D vision, so as to solve the technical problem that the gluing accuracy is greatly affected by directly using the highest point of the side wall of the sole as the gluing trajectory point. The specific technical solution is as follows:

[0004] A method for extracting glue coating tracks from a shoe sole using 3D vision, comprising the following steps:

[0005] Determine a segmentation starting point and a segmentation end point of the sole, wherein the segmentation starting point includes: the minimum point of the heel, and the segmentation end point includes: the maximum point of the toe;

[0006] Establish a three-dimensional coordinate system, and level all the sole point clouds until the length direction of the shoe is parallel to the x-axis of the three-dimensional coordinate system, the width direction of the shoe is parallel to the y-axis of the three-dimensional coordinate system, and the height direction of the shoe is parallel to the z-axis of the three-dimensional coordinate system;

[0007] A cutting plane is made from the segmentation starting point to the segmentation end point along the x-axis direction, and the highest point of the shoe wall is obtained by projecting the cutting plane on the yz plane;

[0008] A preset number of point clouds are obtained in a preset range area of ​​the highest point of the shoe wall, and the preset number of point clouds are divided and fitted into n segments by curvature, and the change rate of each segment is obtained, and the point with the largest change rate is taken as the fitting point of the sole and the upper.

[0009] Furthermore, the “determining the segmentation starting point and segmentation end point of the shoe sole” specifically includes the following steps:

[0010] The sole point cloud data is converted into a depth map, and the segmentation boundary lines of the toe and heel are located by establishing a search caliper to determine the segmentation starting point and end point of the sole.

[0011] Furthermore, after the “making a cutting plane from the segmentation starting point to the segmentation end point along the x-axis direction”, the method further includes the following steps:

[0012] There is a point cloud intersection between the cutting plane and the sole point cloud. The point cloud intersection is mapped to a two-dimensional plane to form a two-dimensional contour intersection line, the gluing trajectory of the cutting plane is extracted, and the two-dimensional information is converted into a three-dimensional space gluing trajectory through the trajectory points.

[0013] Furthermore, the "leveling of all sole point clouds until the length direction of the shoe is parallel to the x-axis of the three-dimensional coordinate system" specifically includes the steps of: obtaining the posture transformation of the randomly placed sole and the camera calibrated sole, obtaining the posture deviation, performing rigid body changes, and converting it into the posture during camera calibration.

[0014] In order to solve the above technical problems, a storage device is also provided, and the specific technical solution is as follows:

[0015] A storage device having an instruction set stored therein, wherein the instruction set is used to execute:

[0016] Determine a segmentation starting point and a segmentation end point of the sole, wherein the segmentation starting point includes: the minimum point of the heel, and the segmentation end point includes: the maximum point of the toe;

[0017] Establish a three-dimensional coordinate system, and level all the sole point clouds until the length direction of the shoe is parallel to the x-axis of the three-dimensional coordinate system, the width direction of the shoe is parallel to the y-axis of the three-dimensional coordinate system, and the height direction of the shoe is parallel to the z-axis of the three-dimensional coordinate system;

[0018] A cutting plane is made from the segmentation starting point to the segmentation end point along the x-axis direction, and the highest point of the shoe wall is obtained by projecting the cutting plane on the yz plane;

[0019] A preset number of point clouds are obtained in a preset range area of ​​the highest point of the shoe wall, and the preset number of point clouds are divided and fitted into n segments by curvature, and the change rate of each segment is obtained, and the point with the largest change rate is taken as the fitting point of the sole and the upper.

[0020] Furthermore, the instruction set is also used to execute: the step of “determining the segmentation starting point and segmentation end point of the shoe sole” specifically includes the steps of:

[0021] The sole point cloud data is converted into a depth map, and the segmentation boundary lines of the toe and heel are located by establishing a search caliper to determine the segmentation starting point and end point of the sole.

[0022] Furthermore, the instruction set is also used to execute: after the “making a cutting plane from the segmentation starting point to the segmentation end point along the x-axis direction”, the instruction set further includes the following steps:

[0023] There is a point cloud intersection between the cutting plane and the sole point cloud. The point cloud intersection is mapped to a two-dimensional plane to form a two-dimensional contour intersection line, the gluing trajectory of the cutting plane is extracted, and the two-dimensional information is converted into a three-dimensional space gluing trajectory through the trajectory points.

[0024] Furthermore, the instruction set is also used to execute: the "leveling of all sole point clouds until the length direction of the shoe is parallel to the x-axis of the three-dimensional coordinate system", which specifically also includes the steps of: obtaining the posture transformation of the randomly placed sole and the camera calibrated sole, and performing rigid body changes by obtaining the posture deviation, and converting it into the posture during camera calibration.

[0025] The beneficial effects of the present invention are: a method for extracting glue coating tracks from 3D vision of the sole, comprising the steps of: determining the segmentation starting point and segmentation end point of the sole, the segmentation starting point includes: the minimum point of the heel, the segmentation end point includes: the maximum point of the toe; establishing a three-dimensional coordinate system, leveling all sole point clouds to the extent that the length direction of the shoe is parallel to the x-axis of the three-dimensional coordinate system, the width direction of the shoe is parallel to the y-axis of the three-dimensional coordinate system, and the height direction of the shoe is parallel to the z-axis of the three-dimensional coordinate system; making a cutting plane from the segmentation starting point to the segmentation end point along the x-axis direction, and obtaining the highest point of the shoe wall by the projection of the cutting plane on the yz plane; obtaining a preset number of point clouds in a preset range area of ​​the highest point of the shoe wall, performing curvature division and fitting of the preset number of point clouds into n segments, obtaining the change rate of each segment, and taking the point with the largest change rate as the fitting point of the sole and the upper. By the above method, the true fitting point of the sole and the upper can be determined, and then an accurate glue coating track can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of a method for extracting glue coating tracks from a shoe sole using 3D vision according to a specific implementation method;

[0027] Figure 2 This is a schematic diagram of the boundary search principle described in the specific implementation method;

[0028] Figure 3 It is a schematic diagram of the projection threshold segmentation principle described in the specific implementation method;

[0029] Figure 4 This is a schematic diagram of the rigidity change of the point cloud described in the specific implementation method;

[0030] Figure 5 This is a schematic diagram of the cutting and gluing track points described in the specific implementation method;

[0031] Figure 6 A schematic diagram for obtaining the minimum point and the maximum point of the heel as described in the specific implementation method;

[0032] Figure 7 The gluing track described in the specific implementation method converts the two-dimensional information into a three-dimensional gluing track schematic diagram through the track points;

[0033] Figure 8 A module schematic diagram of a storage device described in a specific implementation manner.

[0034] Description of reference numerals:

[0035] 800. Storage device. DETAILED DESCRIPTION

[0036] In order to explain the technical content, structural features, achieved objectives and effects of the technical solution in detail, the following is a detailed description in conjunction with specific embodiments and accompanying drawings.

[0037] See also Figures 1 to 7 In this embodiment, a method for extracting glue coating tracks from a shoe sole using 3D vision can be applied to a storage device, and the storage device includes but is not limited to: a personal computer, a server, a general-purpose computer, a special-purpose computer, a network device, an embedded device, a programmable device, etc.

[0038] The specific technical solutions are as follows:

[0039] Step S101: determining a segmentation start point and a segmentation end point of the sole, wherein the segmentation start point includes: a minimum point of the heel, and the segmentation end point includes: a maximum point of the toe.

[0040] The smallest and largest points of the heel are as follows: Figure 6 As shown in the figure, the image obtained from 3D vision includes the sole and the background area. To process the sole, it is necessary to locate the sole, take the dividing line of the heel as the minimum point, and the toe as the maximum point. Figure 6 The two dividing lines shown in d.

[0041] The “determining the segmentation starting point and segmentation end point of the shoe sole” specifically includes the following steps:

[0042] The sole point cloud data is converted into a depth map, and the segmentation boundary lines of the toe and heel are located by establishing a search caliper to determine the segmentation starting point and end point of the sole.

[0043] The following combination Figure 2 and Figure 3 Further explanation:

[0044] By identifying the toe and heel areas, the start and end points of the segmentation are distinguished. Due to the consistency of the background, the camera obtains the point cloud data of the sole. By converting the point cloud into a 16-bit depth map, the boundary between the sole and the background is enhanced to facilitate information extraction. A search caliper is established to locate the boundary edge. Figure 3 As shown, these boundary points are then used to bridge a straight line, which represents the boundary line of the object. Since the toe and the sole are both arc-shaped, the smallest point of the heel and the largest point of the toe are selected as the starting and ending points of the segmentation.

[0045] like Figure 2The search caliper searches for edges by the contrast of the image. The contrast is the initial ratio of black to white in the image, that is, the gradient from black to white. The larger the ratio, the greater the gradient from black to white, and the richer the color expression. In order to reduce processing time and storage, and in some cases enhance edge information, the search caliper area is projected from the two-dimensional image to the one-dimensional image such as Figure 3 As shown in Figure 2. The edge lines that do not meet the minimum contrast are eliminated by contrast thresholding.

[0046] The calculation formula of contrast ratio is:

[0047] Among them: δ(i,j) = |ij|, that is, the grayscale difference between adjacent pixels; P δ (i, j) is the pixel distribution probability when the grayscale difference between adjacent pixels is δ.

[0048] After determining the segmentation start point and segmentation end point of the shoe sole, execute step S102: establish a three-dimensional coordinate system, and level all the shoe sole point clouds until the shoe length direction is parallel to the x-axis of the three-dimensional coordinate system, the shoe width direction is parallel to the y-axis of the three-dimensional coordinate system, and the shoe height direction is parallel to the z-axis of the three-dimensional coordinate system. The "leveling all the shoe sole point clouds until the shoe length direction is parallel to the x-axis of the three-dimensional coordinate system" specifically includes the following steps:

[0049] Get the posture transformation between the randomly placed sole and the camera-calibrated sole, obtain the posture deviation, perform rigid body transformation, and convert it into the posture during camera calibration.

[0050] Step S103: Make a cutting plane from the segmentation starting point to the segmentation end point along the x-axis direction, and obtain the highest point of the shoe wall by projecting the cutting plane on the yz plane. After the step of "making a cutting plane from the segmentation starting point to the segmentation end point along the x-axis direction", the following steps are specifically included:

[0051] The cutting plane and the sole point cloud have a point cloud intersection, and the point cloud intersection is mapped to a two-dimensional plane to form a two-dimensional contour intersection line, and the gluing trajectory of the cutting plane is extracted, and the two-dimensional information is converted into a three-dimensional space gluing trajectory through the trajectory points. Figure 7 As shown, the two-dimensional contour intersection is as follows Figure 7 The curve in the middle of the right image obtains the left and right highest points of the curve. The curve represents the XZ information. By setting the spacing of the cutting plane, the Y information can be obtained, and then the three-dimensional information of the space can be obtained.

[0052] Step S104: obtaining a preset number of point clouds in a preset range area of ​​the highest point of the shoe wall, dividing the preset number of point clouds into n segments by curvature fitting, obtaining the change rate of each segment, and taking the point with the largest change rate as the fitting point between the sole and the upper.

[0053] The following combination Figure 4and Figure 5 , and specifically explain step S102 to step S104:

[0054] The start and end points of the sole segmentation are determined by calipers to limit the spacing of the cuts. Since the soles are placed randomly by workers, the consistency of the orientation cannot be guaranteed, so a rigid body transformation of the point cloud data is required. First, the posture deviation between the randomly placed soles and the calibrated soles is obtained, and all points of the sole point cloud are leveled through rigid body transformation. The point cloud leveling methods for the sole include the minimum circumscribed rectangle, finding the main axis of the sole point cloud, and adjusting the overall point cloud posture by fitting the sole point cloud plane. Since the background obtained by the camera is relatively single and has good consistency, the sole point cloud plane is fitted to obtain the point cloud posture for adjustment.

[0055] Formally, the goal of pose estimation is to estimate the transformation This transformation minimizes the sum of the squared distances between each point p on the object model P and the corresponding point q in the scene model Q:

[0056]

[0057] In the equation above, a homogeneous representation of the points is used to allow matrix-vector multiplication. Pose estimation is usually solved using a robust and outlier-tolerant method, such as RANSAC. A common way to handle this problem is based on feature correspondences, where the following is iterated:

[0058] (1) Find n ≥ 3 random sampling points in P and their corresponding points in Q through nearest neighbor matching of the invariant feature descriptor.

[0059] (2) Use n sample correspondences to estimate the hypothesized transformation

[0060] (3) Apply the hypothesized transformation to the source point cloud P.

[0061] (4) Find the inliers by performing a spatial nearest neighbor search between the transformed source point cloud and the target point cloud Q using a Euclidean distance threshold. If the number of inliers is too small, return to step (1).

[0062] (5) Re-estimate the hypothesized transformation using the inlier correspondence

[0063] (6) Calculate the distance between corresponding pairs of inner points If the distance is the minimum so far, then as the final conversion.

[0064] In many cases, the algorithm can be optimized by stopping if ε falls below a predefined convergence threshold. Otherwise, the algorithm runs for a specified number of iterations.

[0065] After the first step of each iteration, the above RANSAC pose estimation process is modified by implementing low-level geometric constraints. Specifically, the ratio between the side lengths of the virtual polygon formed by the n sample points on the object and scene models is checked. The point corresponding to the feature sample is denoted as p i , q i , the side length of the object polygon is as follows d p,i =||p i+1 modn-p i ||, the same method is used to calculate the side length d of the scene polygon q,i Then, the relative dissimilarity vector δ is calculated by the ratio between the edge lengths of the n polygons:

[0066]

[0067] If the two polygons match perfectly, then δ=0.

[0068] The maximum prefetch deviation is lower than a certain threshold t poly :

[0069] ||δ|| ∞ ≤t poly

[0070] Therefore, the modification to RANSAC is to insert the following steps between step (1) and step (2):

[0071] Calculate the dissimilarity vector δ between the edge lengths of the sampled polygons, if ||δ|| ∞ ≤t poly Return to step (1). This verification step is much cheaper than implementing the full geometric constraints using steps (2) to (6). poly The closer it is to 1, the fewer iterations are needed; however, it increases the risk of excluding the correct location when noise is present.

[0072] Assuming that this step does not filter out hypothetical poses of the correctly aligned model, one can expect the same success rate, but in a much shorter time. Obviously, this assumption only holds for fairly accurate geometric observations, so that the polygons are indeed equidistant. If the sensor used exhibits large depth errors or produces large distortion effects, the threshold needs to be set higher to account for these inaccuracies. Given the quality of current sensors, we expect this problem to be limited in scope. Using t poly =0.25, thereby allowing a maximum edge length difference of 25%.

[0073] Our modification of RANSAC is based on a simple criterion that false hypotheses should be filtered out immediately, saving time in generating a larger number of hypothesized poses. The method can be considered hierarchical in its outlier rejection stage, so it introduces a preliminary low-level polygon-based rejection of unwanted poses before the usual inlier-based rejection stage.

[0074] Given the expected success rate ρ and the expected inline score ω, the formula for calculating the required number of RANSAC iterations k is:

[0075]

[0076] In general, in step (1), n ​​= 3 points are sampled, the inline score is estimated ω = 0.05, the expected success rate ρ = 0.99, k = 3700, the Euclidean convergence threshold ε = 0.01m, and the required number of inline points is set to 50% of the total number of object model points.

[0077] For rigid body changes, the pose describes a rigid 3D transformation, i.e. a transformation consisting of arbitrary translations and rotations, with 6 parameters: TransX, TransY and TransZ specify translations along the x, y and z axes respectively, while RotX, RotY and RotZ represent rotations around the x, y and z axes respectively. The main difficulty lies in rotation, and 3D rotations around arbitrary axes can be represented by 3 parameters in many ways. The rotation transformation is shown below.

[0078] R gba =R x (RotX)·R y (RotY)·R z (RotZ)

[0079] Where: R gba Expressed as "Yaw-Pitch-Roll".

[0080] The corresponding homogeneous transformation matrix is ​​as follows:

[0081]

[0082] Through the rigid change of the sole point cloud, it is converted into a direction parallel to the X-axis of the long axis of the sole, and the minimum circumscribed rectangle of the sole is established. Along the X-direction, a cutting plane is made from the starting point of the heel to the toe. Each time it moves along the X-axis by a specified length, the cutting plane and the sole point cloud have a point cloud intersection. The point cloud intersection is mapped to a two-dimensional plane to form an XLD two-dimensional contour intersection line. Figure 5 The gluing trajectory of the cutting plane is extracted, and the two-dimensional information is converted into a three-dimensional gluing trajectory through the trajectory points.

[0083] Depend on Figure 5From the projection of the mid-tangent plane on the yz plane, it can be seen that the highest point of the shoe wall is at point A, while the actual area where the sole and the upper are glued together is not the highest point, but near the highest point, where the slope suddenly changes greatly.

[0084] 1000 point clouds are selected from the areas near the highest point and behind it for curvature division and fitting into n segments, and the change rate of each segment is obtained according to the formula (X).

[0085]

[0086] Where: Δθ represents the rate of change of the inclination angles of two adjacent segments, Represents the inclination angle between the nth segment and the n-1th segment. Obtain the change rate of each segment, and take the point with the largest change rate as the fitting point between the sole and the upper.

[0087] Through the above method, the real fitting point between the sole and the upper can be determined, thereby obtaining an accurate gluing trajectory.

[0088] See also Figures 2 to 8 In this embodiment, a specific implementation of a storage device 800 is as follows:

[0089] A storage device 800 stores an instruction set, wherein the instruction set is used to execute:

[0090] Determine a segmentation starting point and a segmentation end point of the sole, wherein the segmentation starting point includes: the minimum point of the heel, and the segmentation end point includes: the maximum point of the toe;

[0091] Establish a three-dimensional coordinate system, and level all the sole point clouds until the length direction of the shoe is parallel to the x-axis of the three-dimensional coordinate system, the width direction of the shoe is parallel to the y-axis of the three-dimensional coordinate system, and the height direction of the shoe is parallel to the z-axis of the three-dimensional coordinate system;

[0092] A cutting plane is made from the segmentation starting point to the segmentation end point along the x-axis direction, and the highest point of the shoe wall is obtained by projecting the cutting plane on the yz plane;

[0093] A preset number of point clouds are obtained in a preset range area of ​​the highest point of the shoe wall, and the preset number of point clouds are divided and fitted into n segments by curvature, and the change rate of each segment is obtained, and the point with the largest change rate is taken as the fitting point of the sole and the upper.

[0094] The smallest and largest points of the heel are as follows: Figure 6 As shown in the figure, the image obtained from 3D vision includes the sole and the background area. To process the sole, it is necessary to locate the sole, take the dividing line of the heel as the minimum point, and the toe as the maximum point. Figure 6 (d) The two dividing lines shown.

[0095] The following combination Figure 2 and Figure 3Further explanation:

[0096] By identifying the toe and heel areas, the start and end points of the segmentation are distinguished. Due to the consistency of the background, the camera obtains the point cloud data of the sole. By converting the point cloud into a 16-bit depth map, the boundary between the sole and the background is enhanced to facilitate information extraction. A search caliper is established to locate the boundary edge. Figure 3 As shown, these boundary points are then used to bridge a straight line, which represents the boundary line of the object. Since the toe and the sole are both arc-shaped, the smallest point of the heel and the largest point of the toe are selected as the starting and ending points of the segmentation.

[0097] like Figure 2 The search caliper searches for edges by the contrast of the image. The contrast is the initial ratio of black to white in the image, that is, the gradient from black to white. The larger the ratio, the greater the gradient from black to white, and the richer the color expression. In order to reduce processing time and storage, and in some cases enhance edge information, the search caliper area is projected from the two-dimensional image to the one-dimensional image such as Figure 3 As shown in Figure 2. The edge lines that do not meet the minimum contrast are eliminated by contrast thresholding.

[0098] The calculation formula of contrast ratio is:

[0099] Among them: δ(i,j) = |ij|, that is, the grayscale difference between adjacent pixels; P δ (i, j) is the pixel distribution probability when the grayscale difference between adjacent pixels is δ.

[0100] After determining the start and end points of the shoe sole segmentation, a three-dimensional coordinate system is established, and all the shoe sole point clouds are leveled so that the shoe length direction is parallel to the x-axis of the three-dimensional coordinate system, the shoe width direction is parallel to the y-axis of the three-dimensional coordinate system, and the shoe height direction is parallel to the z-axis of the three-dimensional coordinate system. The "leveling of all the shoe sole point clouds so that the shoe length direction is parallel to the x-axis of the three-dimensional coordinate system" specifically includes the following steps:

[0101] Get the posture transformation between the randomly placed sole and the camera-calibrated sole, obtain the posture deviation, perform rigid body transformation, and convert it into the posture during camera calibration.

[0102] Furthermore, the instruction set is also used to execute: after the “making a cutting plane from the segmentation starting point to the segmentation end point along the x-axis direction”, the instruction set further includes the following steps:

[0103] The cutting plane and the sole point cloud have a point cloud intersection, and the point cloud intersection is mapped to a two-dimensional plane to form a two-dimensional contour intersection line, and the gluing trajectory of the cutting plane is extracted, and the two-dimensional information is converted into a three-dimensional space gluing trajectory through the trajectory points. Figure 7 As shown, the two-dimensional contour intersection is as follows Figure 7 The curve in the middle of the right image obtains the left and right highest points of the curve. The curve represents the XZ information. By setting the spacing of the cutting plane, the Y information can be obtained, and then the three-dimensional information of the space can be obtained.

[0104] Furthermore, the instruction set is also used to execute: the "leveling of all sole point clouds until the length direction of the shoe is parallel to the x-axis of the three-dimensional coordinate system", which specifically also includes the steps of: obtaining the posture transformation of the randomly placed sole and the camera calibrated sole, and performing rigid body changes by obtaining the posture deviation, and converting it into the posture during camera calibration.

[0105] The following combination Figure 4 and Figure 5 , expand the specific instructions:

[0106] The start and end points of the shoe sole segmentation are determined by calipers to limit the spacing of the cuts. Since the soles are randomly placed by workers, the consistency of the orientation cannot be guaranteed, so a rigid body transformation of the point cloud data is required. First, the posture deviation between the randomly placed soles and the calibrated soles is obtained, and all points of the sole point cloud are leveled through rigid body transformation, as shown in Figure X. The point cloud leveling methods for the sole include the minimum circumscribed rectangle, finding the main axis of the sole point cloud, and adjusting the overall point cloud posture by fitting the sole point cloud plane. Since the background obtained by the camera is relatively single and has good consistency, the sole point cloud plane is fitted to obtain the point cloud posture for adjustment.

[0107] Formally, the goal of pose estimation is to estimate the transformation This transformation minimizes the sum of the squared distances between each point p on the object model P and the corresponding point q in the scene model Q:

[0108]

[0109] In the equation above, a homogeneous representation of the points is used to allow matrix-vector multiplication. Pose estimation is usually solved using a robust and outlier-tolerant method, such as RANSAC. A common way to handle this problem is based on feature correspondences, where the following is iterated:

[0110] (1) Find n ≥ 3 random sampling points in P and their corresponding points in Q through nearest neighbor matching of the invariant feature descriptor.

[0111] (2) Use n sample correspondences to estimate the hypothesized transformation

[0112] (3) Apply the hypothesized transformation to the source point cloud P.

[0113] (4) Find the inliers by performing a spatial nearest neighbor search between the transformed source point cloud and the target point cloud Q using a Euclidean distance threshold. If the number of inliers is too small, return to step (1).

[0114] (5) Re-estimate the hypothesized transformation using the inlier correspondence

[0115] (6) Calculate the distance between corresponding pairs of inner points If the distance is the minimum so far, then as the final conversion.

[0116] In many cases, the algorithm can be optimized by stopping if ε falls below a predefined convergence threshold. Otherwise, the algorithm runs for a specified number of iterations.

[0117] After the first step of each iteration, the above RANSAC pose estimation process is modified by implementing low-level geometric constraints. Specifically, the ratio between the side lengths of the virtual polygon formed by the n sample points on the object and scene models is checked. The point corresponding to the feature sample is denoted as p i , q i , the side length of the object polygon is as follows d p,i =||p i+1 modn-p i ||, the same method is used to calculate the side length d of the scene polygon q,i Then, the relative dissimilarity vector δ is calculated by the ratio between the edge lengths of the n polygons:

[0118]

[0119] If the two polygons match perfectly, then δ=0.

[0120] The maximum prefetch deviation is lower than a certain threshold t poly :

[0121] ||δ|| ∞ ≤t poly

[0122] Therefore, the modification to RANSAC is to insert the following steps between step (1) and step (2):

[0123] Calculate the dissimilarity vector δ between the edge lengths of the sampled polygons, if ||δ|| ∞ ≤t poly Return to step (1). This verification step is much cheaper than implementing the full geometric constraints using steps (2) to (6). poly The closer it is to 1, the fewer iterations are needed; however, it increases the risk of excluding the correct location when noise is present.

[0124] Assuming that this step does not filter out hypothetical poses of the correctly aligned model, one can expect the same success rate, but in a much shorter time. Obviously, this assumption only holds for fairly accurate geometric observations, so that the polygons are indeed equidistant. If the sensor used exhibits large depth errors or produces large distortion effects, the threshold needs to be set higher to account for these inaccuracies. Given the quality of current sensors, we expect this problem to be limited in scope. Using t poly =0.25, thereby allowing a maximum edge length difference of 25%.

[0125] Our modification of RANSAC is based on a simple criterion that false hypotheses should be filtered out immediately, saving time in generating a larger number of hypothesized poses. The method can be considered hierarchical in its outlier rejection stage, so it introduces a preliminary low-level polygon-based rejection of unwanted poses before the usual inlier-based rejection stage.

[0126] Given the expected success rate ρ and the expected inline score ω, the formula for calculating the required number of RANSAC iterations k is:

[0127]

[0128] In general, in step (1), n ​​= 3 points are sampled, the inline score is estimated ω = 0.05, the expected success rate ρ = 0.99, k = 3700, the Euclidean convergence threshold ε = 0.01m, and the required number of inline points is set to 50% of the total number of object model points.

[0129] For rigid body changes, the pose describes a rigid 3D transformation, i.e. a transformation consisting of arbitrary translations and rotations, with 6 parameters: TransX, TransY and TransZ specify translations along the x, y and z axes respectively, while RotX, RotY and RotZ represent rotations around the x, y and z axes respectively. The main difficulty lies in rotation, and 3D rotations around arbitrary axes can be represented by 3 parameters in many ways. The rotation transformation is shown below.

[0130] R gba =R x (RotX)·R y (RotY)·R z (RotZ)

[0131] Where: R gba Expressed as "Yaw-Pitch-Roll".

[0132] The corresponding homogeneous transformation matrix is ​​as follows:

[0133]

[0134] Through the rigid change of the sole point cloud, it is converted into a direction parallel to the X-axis of the long axis of the sole, and the minimum circumscribed rectangle of the sole is established. Along the X-direction, a cutting plane is made from the starting point of the heel to the toe. Each time it moves along the X-axis by a specified length, the cutting plane and the sole point cloud have a point cloud intersection. The point cloud intersection is mapped to a two-dimensional plane to form an XLD two-dimensional contour intersection line. Figure 5 The gluing trajectory of the cutting plane is extracted, and the two-dimensional information is converted into a three-dimensional gluing trajectory through the trajectory points.

[0135] Depend on Figure 5 From the projection of the mid-tangent plane on the yz plane, it can be seen that the highest point of the shoe wall is at point A, while the actual area where the sole and the upper are glued together is not the highest point, but near the highest point, where the slope suddenly changes greatly.

[0136] 1000 point clouds are selected from the areas near the highest point and behind it for curvature division and fitting into n segments, and the change rate of each segment is obtained according to the formula (X).

[0137]

[0138] Where: Δθ represents the rate of change of the inclination angles of two adjacent segments, Represents the inclination angle between the nth segment and the n-1th segment. Obtain the change rate of each segment, and take the point with the largest change rate as the fitting point between the sole and the upper.

[0139] By means of the storage device 800, the actual fitting point between the sole and the upper can be determined, thereby obtaining an accurate gluing trajectory.

[0140] It should be noted that, although the above embodiments have been described in this article, the patent protection scope of the present invention is not limited thereby. Therefore, based on the innovative concept of the present invention, changes and modifications made to the embodiments described herein, or equivalent structures or equivalent process changes made using the contents of the present invention specification and drawings, directly or indirectly applying the above technical solutions to other related technical fields, are all included in the patent protection scope of the present invention.

Claims

1. A method for extracting glue coating tracks from soles using 3D vision, characterized in that: Includes steps: Determine a segmentation starting point and a segmentation end point of the sole, wherein the segmentation starting point includes: the minimum point of the heel, and the segmentation end point includes: the maximum point of the toe; Establish a three-dimensional coordinate system, and level all the sole point clouds until the length direction of the shoe is parallel to the x-axis of the three-dimensional coordinate system, the width direction of the shoe is parallel to the y-axis of the three-dimensional coordinate system, and the height direction of the shoe is parallel to the z-axis of the three-dimensional coordinate system; A cutting plane is made from the segmentation starting point to the segmentation end point along the x-axis direction, and the highest point of the shoe wall is obtained by projecting the cutting plane on the yz plane; A preset number of point clouds are obtained in a preset range area of ​​the highest point of the shoe wall, and the preset number of point clouds are divided and fitted into n segments by curvature, and the change rate of each segment is obtained, and the point with the largest change rate is taken as the fitting point of the sole and the upper.

2. The method for extracting glue coating tracks from a 3D sole visual system according to claim 1, characterized in that: The "determining the segmentation starting point and segmentation end point of the sole" specifically includes the steps of: The sole point cloud data is converted into a depth map, and the segmentation boundary lines of the toe and heel are located by establishing a search caliper to determine the segmentation starting point and end point of the sole.

3. The method for extracting glue coating tracks from a 3D sole visual system according to claim 1, characterized in that: After the step of "making a cutting plane from the segmentation starting point to the segmentation end point along the x-axis direction", the method further includes the following steps: There is a point cloud intersection between the cutting plane and the sole point cloud. The point cloud intersection is mapped to a two-dimensional plane to form a two-dimensional contour intersection line, the gluing trajectory of the cutting plane is extracted, and the two-dimensional information is converted into a three-dimensional space gluing trajectory through the trajectory points.

4. The method for extracting glue coating tracks from a 3D sole visual system according to claim 1, characterized in that: The "leveling of all sole point clouds until the length direction of the shoe is parallel to the x-axis of the three-dimensional coordinate system" specifically includes the following steps: Get the posture transformation between the randomly placed sole and the camera-calibrated sole, obtain the posture deviation, perform rigid body transformation, and convert it into the posture during camera calibration.

5. A storage device having an instruction set stored therein, characterized in that: The instruction set is executed by the processor: Determine a segmentation starting point and a segmentation end point of the sole, wherein the segmentation starting point includes: the minimum point of the heel, and the segmentation end point includes: the maximum point of the toe; Establish a three-dimensional coordinate system, and level all the sole point clouds until the length direction of the shoe is parallel to the x-axis of the three-dimensional coordinate system, the width direction of the shoe is parallel to the y-axis of the three-dimensional coordinate system, and the height direction of the shoe is parallel to the z-axis of the three-dimensional coordinate system; A cutting plane is made from the segmentation starting point to the segmentation end point along the x-axis direction, and the highest point of the shoe wall is obtained by projecting the cutting plane on the yz plane; A preset number of point clouds are obtained in a preset range area of ​​the highest point of the shoe wall, and the preset number of point clouds are divided and fitted into n segments by curvature, and the change rate of each segment is obtained, and the point with the largest change rate is taken as the fitting point of the sole and the upper.

6. A storage device according to claim 5, characterized in that: The instruction set is also executed by the processor: the "determining the segmentation starting point and segmentation end point of the shoe sole" specifically includes the steps of: The sole point cloud data is converted into a depth map, and the segmentation boundary lines of the toe and heel are located by establishing a search caliper to determine the segmentation starting point and end point of the sole.

7. A storage device according to claim 5, characterized in that: The instruction set is also executed by the processor: after the "making a cutting plane from the segmentation starting point to the segmentation end point along the x-axis direction", the instruction set further includes the following steps: There is a point cloud intersection between the cutting plane and the sole point cloud. The point cloud intersection is mapped to a two-dimensional plane to form a two-dimensional contour intersection line, the gluing trajectory of the cutting plane is extracted, and the two-dimensional information is converted into a three-dimensional space gluing trajectory through the trajectory points.

8. A storage device according to claim 5, characterized in that: The instruction set is also executed by the processor: the "leveling of all sole point clouds to a point where the length direction of the shoe is parallel to the x-axis of the three-dimensional coordinate system" specifically includes the steps of: Get the posture transformation between the randomly placed sole and the camera-calibrated sole, obtain the posture deviation, perform rigid body transformation, and convert it into the posture during camera calibration.