A vision-based method for calibrating fabric sewing trajectory

Abstract

The present invention proposes a vision-based cloth sewing trajectory calibration method, comprising: intercepting an interested ROI region containing cloth; using an image processing algorithm to find the inflection point of the image contour in the ROI region; collecting a data set of cloth to be processed, collecting some pictures of each cloth for training a deep learning model, marking the inflection point of the edge of the cloth after the collection is completed, and training a deep learning key point detection model after the marking is completed to find the inflection point of the image contour; using a curvature search method and a deep learning method to jointly find the inflection point, after each inflection point is found, obtaining the number and distance of the inflection points found by the two methods, and judging whether to abandon the piece of cloth according to the number and distance of the inflection points; after finding the inflection point of the cloth contour, offsetting the trajectory of the cloth contour based on the inflection point; comparing the offset curve with a qualified template curve after debugging to detect the overall reliability of the algorithm.

Description

Technical Field

[0001] The invention relates to the technical field of industrial robots, and in particular to a vision-based cloth sewing trajectory calibration method. Background Art

[0002] In the field of robotic automated sewing, most of the sewing is done along the outer edge of the fabric. However, due to the accuracy of the robot, the calibration accuracy of the camera, and the mechanical error of the sewing mechanism, the actual movement route of the robot may deviate from the edge contour of the fabric. For example, the actual movement trajectory of the robot may deviate to the outside or inside of the fabric contour, which may result in substandard sewing process results.

[0003] Most of the existing robot sewing mechanisms do not have a trajectory calibrated or the calibration method is not robust, resulting in a discrepancy between the calibrated sewing trajectory and the ideal situation in actual sewing work. In the worst case, the sewing will not be able to be done, and in the worst case, it will cause the sewing mechanism and the robot to collide. Summary of the invention

[0004] The object of the present invention is to solve at least one of the technical drawbacks.

[0005] To this end, the purpose of the present invention is to propose a vision-based cloth sewing trajectory calibration method.

[0006] In order to achieve the above object, the embodiment of the present invention provides a method for calibrating a cloth sewing trajectory based on vision, comprising the following steps:

[0007] Step S1, intercepting the ROI region of interest containing the cloth;

[0008] Step S2, using an image processing algorithm to find the inflection point of the image contour in the ROI area, includes:

[0009] (1) Binarizing the ROI area in step S1;

[0010] (2) Find the edge contour of the material;

[0011] (3) Downsampling the edge contour of the material to make the contour point set sparse;

[0012] (4) Calculate the curvature between every three points of the edge contour of the cloth;

[0013] (5) Calculate the average of the curvature of each point obtained in the previous step and the curvature of the adjacent points, and then obtain the average curvature of each point. Set a threshold. When the average curvature of a point is greater than the threshold, the point is considered to be the inflection point of the contour.

[0014] Step S3, collecting a data set of fabrics to be processed, collecting some pictures of each fabric for training the deep learning model, marking the inflection points of the edges of the fabrics after the collection is completed, and training the deep learning key point detection model after the marking is completed. After the model is trained, the ROI area containing the fabric obtained in step S1 is input to find the inflection points of the image contour;

[0015] Step S4, using the curvature search method and the deep learning method to search for inflection points together, after searching for inflection points respectively, obtaining the number and distance of inflection points found by the two methods, and judging whether to abandon the piece of fabric according to the number and distance of inflection points;

[0016] Step S5, after finding the inflection point of the cloth contour, performing a trajectory offset of the cloth contour based on the inflection point;

[0017] Step S6, comparing the offset curve with the qualified template curve after debugging to detect the overall reliability of the algorithm.

[0018] Furthermore, in step S2 (4), the curvature between every three points of the cloth edge contour is calculated in sequence, including: calculating the radius of the circumscribed circle of every three points starting from the starting point in sequence, and then taking the inverse of the radius of the circumscribed circle as the curvature to obtain the curvature of each point on the contour.

[0019] Further, in step S2 (5),

[0020] The mean curvature at each point is given by:

[0021]

[0022] Among them, i is the index of a point on the contour, k is the point i as the middle point, and k / 2 points are selected before and after to calculate the average curvature. It is the first The curvature of a point, is the average curvature of the i-th point after the calculation, and then use The value is compared with the set threshold. When it is greater than a certain threshold, the index point is considered It is the corresponding inflection point of the contour.

[0023] Further, in step S4, after using the curvature search method and the deep learning method to jointly search for inflection points, the number of inflection points found by the two algorithms is determined. If they are not equal, the piece of cloth is abandoned; if they are equal, a starting inflection point is selected, and then the distance between the inflection points found by the two algorithms is determined in a counterclockwise or clockwise order. When the distance is less than a given threshold, it is considered that the inflection points found by the two algorithms are consistent, and then the middle value of each inflection point under the two algorithms is taken as the final inflection point; if the inflection point distance threshold under the two algorithms is greater than a given threshold, it is considered that the results of the two search algorithms are inconsistent due to the cloth, and the piece of cloth is abandoned.

[0024] Further, in step S5, after locating the inflection point of the contour curve, a starting inflection point is selected, and the remaining inflection points are selected in turn in a clockwise or counterclockwise direction, and then the contour of the cloth is divided into 4 sections with each inflection point as an endpoint, each contour section has a starting point and an end point, and then the distance from the starting point to the end point is calculated, and finally the points on each contour curve are offset according to the ratio of the distance from the starting point to the distance from the starting point to the end point.

[0025] Further, the following formula is used for offset:

[0026]

[0027] in, It is the ratio of the specified distance from the starting point to the distance from the starting point to the end point. The ratio of the distance from the starting point to the distance from the starting point to the end point is The offset at , It is the ratio of the distance from the i-th point to the starting point to the distance from the starting point to the end point in the contour segment. The first The above method is to set the offset for a contour curve, and the same applies to other contour curves.

[0028] Further, in the step S6, a template contour curve is selected, and then in each new cloth processing process, a new cloth contour curve is obtained after inflection point search and contour offset based on the inflection point. At this time, a series of distance values ​​are selected for each curve in the cloth contour curve, and the index values ​​of the midpoints of the template contour curve and the contour curve to be verified at each distance are calculated respectively. A threshold is selected, and then the index value is subtracted. If it is greater than the threshold, it means that the verification fails, otherwise the verification is successful.

[0029] Furthermore, the selection of the template contour curve includes: first finding the inflection point of the fabric contour curve, and then offsetting the fabric contour by a certain amount based on the inflection point, and performing actual sewing operations on the obtained contour curve. If the given offset is not suitable in the actual sewing operation, multiple adjustments are made until the actual production process requirements are met. At this time, the contour curve after the offset setting is saved, and this curve is the template contour curve.

[0030] According to the vision-based cloth sewing trajectory calibration method of an embodiment of the present invention, a deep learning key point detection and average curvature are combined to search for the inflection points of the cloth contour, and a verification algorithm based on the template contour curve and the contour curve to be verified is provided.

[0031] The present invention proposes a method combining deep learning key point detection and traditional curvature to search for the inflection point of the cloth contour in view of the flexible characteristics of the cloth. In the process of searching for the inflection point of the cloth curve contour, it can be decided whether to use the method combining deep learning and traditional curvature or to use the deep learning method alone according to the characteristics of the cloth. This algorithm greatly enhances the robustness of the algorithm. In addition, the present invention also proposes an algorithm for calibration based on the template contour curve. This method can effectively prevent sewing due to deformation of the cloth and feeding errors of the feeding mechanism. Because if sewing continues under such conditions, it is very likely that the sewing machine will collide with the robot or other mechanisms, so this situation must be prohibited.

[0032] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0034] Figure 1 is a flow chart of a method for calibrating a cloth sewing trajectory based on vision according to an embodiment of the present invention;

[0035] Figure 2 is an original cloth image according to an embodiment of the present invention;

[0036] Figure 3 For Figure 2 The image of the region of interest of the cut cloth;

[0037] Figure 4 For Figure 3 Images annotated with key point detection dataset;

[0038] Figure 5 For Figure 4An image that locates the inflection point of the contour curve;

[0039] Figure 6 is a flow chart of an overall algorithm according to an embodiment of the present invention;

[0040] Figure 7 is a flow chart of an offset algorithm based on contour inflection points according to an embodiment of the present invention;

[0041] Figure 8 4 is a flow chart of a template contour curve and a contour curve after offset verification algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0043] The present invention proposes a vision-based sewing trajectory calibration method, which is implemented on the basis of precise positioning of the inflection point of the cloth contour. After positioning the inflection point, the contour trajectory is offset and then compared with the selected nominal path. If the error is within the allowable range, sewing is performed, otherwise the piece of cloth is discarded.

[0044] In the process of searching for the inflection point of the cloth contour, a combination of traditional algorithms and deep learning algorithms is used to ensure the accuracy and stability of the inflection point search. Because the corners of some cloth are arc-shaped, the traditional curvature method proposed in this patent cannot find the inflection point for such images, because the curvature of each point on the arc where the inflection point is located is the same. For such images, a deep learning algorithm must be used separately to find the inflection point, and then for images like Figure 2 Such images can be processed using a combination of the curvature method and deep learning, which can prevent a certain algorithm from making inaccurate searches to a certain extent.

[0045] like Figure 1 and Figure 6 As shown, the vision-based cloth sewing trajectory calibration method of the embodiment of the present invention includes the following steps:

[0046] Step S1, intercepting the ROI region of interest containing the cloth.

[0047] Specifically, the region of interest (ROI) containing the fabric is captured, and the captured image is as follows: Figure 3 shown.

[0048] Step S2, using an image processing algorithm to find the inflection point of the image contour in the ROI area, includes:

[0049] (1) Binarizing the ROI area in step S1;

[0050] (2) Find the edge contour of the material;

[0051] (3) Downsampling the edge contour of the material to make the contour point set sparse;

[0052] (4) Calculating the curvature between every three points of the edge contour of the cloth in sequence, including: calculating the radius of the circumscribed circle of every three points starting from the starting point in sequence, and then taking the inverse of the radius of the circumscribed circle as the curvature, so as to obtain the curvature of each point on the contour.

[0053] (5) Calculate the curvature of each point obtained in the previous step and the average of the curvatures of the adjacent points, then obtain the average curvature of each point and set a threshold. When the average curvature of a point is greater than the threshold, the point is considered to be the inflection point of the contour.

[0054] The average curvature at each point is as described in formula (1):

[0055] (1)

[0056] Among them, i is the index of a point on the contour, k is the point i as the middle point, and k / 2 points are selected before and after to calculate the average curvature. It is the first The curvature of a point, is the average curvature of the i-th point after the calculation, and then use The value is compared with the set threshold. When it is greater than a certain threshold, the index point is considered It is the corresponding inflection point of the contour.

[0057] Step S3, collect the data set of the fabric to be processed. For example, if 10 kinds of fabric are processed, collect some pictures of each kind of fabric for deep learning model training. After the fabric is collected, the turning points of its edges are marked. After the marking is completed, a suitable deep learning key point detection model is trained for use. The key point detection and marking rules are as follows: Figure 4 As shown. During the annotation process, the inflection point of the cloth can be marked. After the model is trained, the inflection point of the image contour can be found by inputting the ROI region containing the cloth obtained in step S1.

[0058] Step S4, using the curvature search method and the deep learning method to jointly find the turning points. After finding the turning points respectively, the number and distance of the turning points found by the two methods are obtained, and it is determined whether to abandon the piece of cloth based on the number and distance of the turning points.

[0059] Specifically, we choose to use the curvature search method and deep learning method to find the inflection point. If the target image is as follows Figure 2 In that case, two methods can be selected to jointly find inflection points. After the two methods have found inflection points respectively, the number of inflection points found by the two methods is determined. If they are equal, the next step of determination is performed. If they are not equal, it means that one of the methods has found the wrong number, or both have found the wrong number. In either case, the piece of cloth should be abandoned, indicating that there is a problem with the piece of cloth that causes one of the algorithms to be inaccurate.

[0060] If the number of inflection points found by the two algorithms is equal, a starting inflection point is selected, and then the distance between the inflection points found by the two algorithms is determined in a counterclockwise or clockwise order. When the distance is less than a given threshold, the inflection points found by the two algorithms are considered to be consistent, and then the middle value of each inflection point under the two algorithms is taken as the final inflection point. If the inflection point distance threshold under the two algorithms is greater than a given threshold, it is considered that the results of the two search algorithms are inconsistent due to the cloth, and in this case the piece of cloth is abandoned.

[0061] If the target image is Figure 3 In that case, a deep learning algorithm must be used separately to find the inflection point, and the result is used directly to offset the curve contour.

[0062] Step S5, after finding the inflection point of the cloth contour, the trajectory of the cloth contour is offset based on the inflection point.

[0063] like Figure 5 and Figure 7 As shown, after locating the inflection point of the contour curve ( Figure 5 Select a starting inflection point, and then select the remaining inflection points in a clockwise or counterclockwise direction. For example, the inflection points are numbered 1, 2, 3, and 4, and then divide the outline of the fabric into 4 segments with each inflection point as the endpoint ( Figure 5 There are four inflection points in the middle, so it is divided into 4 segments. Generally, it is divided into as many segments as there are inflection points). Because the clockwise or counterclockwise direction is specified, each contour has a starting point and an end point. Then the distance from the starting point to the end point is calculated. Finally, the points on each contour curve are offset according to the ratio of the distance from the starting point to the distance from the starting point to the end point, as described in the following formula (2):

[0064] (2)

[0065] In the above formula It is the ratio of the specified distance from the starting point to the distance from the starting point to the end point. The ratio of the distance from the starting point to the distance from the starting point to the end point is The offset at , It is the ratio of the distance from the i-th point to the starting point to the distance from the starting point to the end point in the contour segment. The first The above method is to set the offset for a contour curve, and the same applies to other contour curves.

[0066] Step S6, comparing the offset curve with the qualified template curve after debugging to detect the overall reliability of the algorithm.

[0067] Steps S2 to S5 are based on the inflection point of the contour. The purpose of this step is to compare the offset curve with the qualified template curve after debugging, so as to detect the overall reliability of the algorithm. The algorithm flow chart of this step is as shown in the attached figure. Figure 8 shown.

[0068] The first step is to select the template contour curve. Based on the first two parts of the algorithm, the inflection point of the cloth contour curve is first found, and then the cloth contour is offset by a certain amount based on the inflection point. The obtained contour curve is used for actual sewing operations. If the given offset is not suitable in the actual sewing operation, it can be adjusted multiple times until it meets the actual production process requirements. At this time, the contour curve after the offset setting is saved, which is the template contour curve.

[0069] After selecting the template contour curve, in each new fabric processing process, a new fabric contour curve will be obtained after inflection point search and inflection point-based contour offset. At this time, for each curve in the fabric contour curve (in the inflection point-based offset process, the contour has been divided into several segments according to the number of inflection points, and the number of segments is equal to the number of inflection points), a series of distance values ​​can be selected. For example, if the distance from the starting point to the end point of a contour line is L, the distance value can be selected from L / 10, 2L / 10,,, 10L / 10, and ten points with equal intervals can be selected. If you want to apply the calibration algorithm more carefully, you can also divide it into 100 parts. After selecting a series of distance values, calculate the index value of the midpoint of the template contour curve and the contour curve to be verified at each distance, select a threshold, and then make the difference of the index value. If it is greater than the threshold, it means that the verification fails, otherwise it is successful.

[0070] According to the vision-based cloth sewing trajectory calibration method of an embodiment of the present invention, a deep learning key point detection and average curvature are combined to search for the inflection points of the cloth contour, and a verification algorithm based on the template contour curve and the contour curve to be verified is provided.

[0071] The present invention proposes a method combining deep learning key point detection and traditional curvature to search for the inflection point of the cloth contour in view of the flexible characteristics of the cloth. In the process of searching for the inflection point of the cloth curve contour, it can be decided whether to use the method combining deep learning and traditional curvature or to use the deep learning method alone according to the characteristics of the cloth. This algorithm greatly enhances the robustness of the algorithm. In addition, the present invention also proposes an algorithm for calibration based on the template contour curve. This method can effectively prevent sewing due to deformation of the cloth and feeding errors of the feeding mechanism. Because if sewing continues under such conditions, it is very likely that the sewing machine will collide with the robot or other mechanisms, so this situation must be prohibited.

[0072] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0073] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. Those skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention without departing from the principles and purpose of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for calibrating cloth sewing trajectory based on vision, characterized in that: The steps include: Step S1, intercepting the ROI region of interest containing the cloth; Step S2, using an image processing algorithm to find the inflection point of the image contour in the ROI area, includes: (1) performing binarization processing on the ROI area of ​​step S1; (2) Find the edge contour of the fabric; (3) Downsampling the edge contour of the cloth to make the contour point set sparse; (4) Calculate the curvature between every three points of the edge contour of the cloth in sequence; (5) Calculate the average of the curvature of each point obtained in the previous step and the curvature of the adjacent points, and then obtain the average curvature of each point. Set a threshold. When the average curvature of a point is greater than the threshold, the point is considered to be the inflection point of the contour. Step S3, collecting a data set of fabrics to be processed, collecting some pictures of each fabric for training the deep learning model, marking the inflection points of the edges of the fabrics after the collection is completed, and training the deep learning key point detection model after the marking is completed. After the model is trained, the ROI area containing the fabric obtained in step S1 is input to find the inflection points of the image contour; Step S4, using the curvature search method and the deep learning method to search for inflection points together, after searching for inflection points respectively, obtaining the number and distance of inflection points found by the two methods, and judging whether to abandon the piece of fabric according to the number and distance of inflection points; Step S5, after finding the inflection point of the cloth contour, performing a trajectory offset of the cloth contour based on the inflection point; Step S6, comparing the offset curve with the qualified template curve after debugging to detect the overall reliability of the algorithm.

2. The method for calibrating the cloth sewing trajectory based on vision as claimed in claim 1, characterized in that: In step S2 (4), the curvature between every three points of the cloth edge contour is calculated in sequence, including: calculating the radius of the circumscribed circle of every three points starting from the starting point in sequence, and then taking the inverse of the radius of the circumscribed circle as the curvature to obtain the curvature of each point on the contour.

3. The vision-based cloth sewing trajectory calibration method according to claim 1, characterized in that: In step S2 (5), The mean curvature at each point is given by: Among them, i is the index of a point on the contour, k is the point i as the middle point, and k / 2 points are selected before and after to calculate the average curvature, c i is the curvature of the i-th point on the contour, is the average curvature of the i-th point after the calculation, and then use The value of is compared with the set threshold. When it is greater than a certain threshold, the index point i is considered to be the corresponding contour inflection point.

4. The method for calibrating the cloth sewing trajectory based on vision as claimed in claim 1, characterized in that: In step S4, after using the curvature search method and the deep learning method to search for inflection points, the number of inflection points found by the two methods is determined, and if they are not equal, the piece of fabric is discarded; If they are equal, a starting inflection point is selected, and then the distance between the inflection points found by the two algorithms is determined in a counterclockwise or clockwise order. When the distance is less than a given threshold, the inflection points found by the two algorithms are considered to be consistent, and then the middle value of each inflection point under the two algorithms is taken as the final inflection point; if the inflection point distance threshold under the two algorithms is greater than a given threshold, it is considered that the results of the two search algorithms are inconsistent due to the cloth, and the piece of cloth is abandoned.

5. The vision-based cloth sewing trajectory calibration method according to claim 1, characterized in that: In step S5, after locating the inflection point of the contour curve, a starting inflection point is selected, and the remaining inflection points are selected in turn in a clockwise or counterclockwise direction, and then the contour of the cloth is divided into 4 sections with each inflection point as an endpoint, each contour section has a starting point and an end point, and then the distance from the starting point to the end point is calculated, and finally the points on each contour curve are offset according to the ratio of the distance from the starting point to the distance from the starting point to the end point.

6. The method for calibrating the cloth sewing trajectory based on vision as claimed in claim 5, characterized in that: The following formula is used for offset: Where p is the ratio of the distance from the starting point to the distance from the starting point to the end point, and r is the ratio of the distance from the starting point to the distance from the starting point to the end point. The offset at p is r, and p i is the ratio of the distance from the i-th point to the starting point to the distance from the starting point to the end point in the contour segment, r i is the offset of the i-th point on the contour, where 7. The vision-based cloth sewing trajectory calibration method according to claim 1, characterized in that: In the step S6, a template contour curve is selected, and then in each new cloth processing process, a new cloth contour curve is obtained after inflection point search and contour offset based on the inflection point. At this time, a series of distance values ​​are selected for each curve in the cloth contour curve, and the index values ​​of the midpoints of the template contour curve and the contour curve to be verified at each distance are calculated respectively. A threshold is selected, and then the index value is subtracted. If it is greater than the threshold, it means that the verification fails, otherwise the verification is successful.

8. The vision-based cloth sewing trajectory calibration method according to claim 7, characterized in that: The selection of the template contour curve includes: first finding the inflection point of the fabric contour curve, then offsetting the fabric contour by a certain amount based on the inflection point, and performing actual sewing operation on the obtained contour curve. If the given offset is not suitable in the actual sewing operation, multiple adjustments are made until the actual production process requirements are met. At this time, the contour curve after the offset setting is saved, and the curve is the template contour curve.

Images