An embroidery data acquisition device using an industrial area array camera

Abstract

The present invention relates to the field of image processing technology, and in particular to an embroidery data acquisition device using an industrial array camera, comprising: a stitch track acquisition module for forming a stitch track from all stitch points in a first image; a stitch track fusion module for matching the stitch track to a second image; a deformation data acquisition module for acquiring tension deformation areas and non-tension deformation areas in the embroidery process; a deformation error feedback module for reacquiring the stitch track on the second image according to the distribution of the tension deformation area along the stitch track, and reacquiring the tension deformation area and non-tension deformation area in the embroidery process using the stitch track on the reacquired second image. The embroidery data acquired by the present invention helps staff to timely discover and locate problems in embroidery over-layering.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to an embroidery data acquisition device using an industrial area array camera. Background Art

[0002] In modern embroidery production, in order to improve product quality and production efficiency, more and more factories have introduced cameras for real-time data collection. However, the embroidery image data directly collected by the camera has certain limitations. On the one hand, these images lack the accurate presentation of the stitch movement trajectory and texture distribution, which is not enough to intuitively describe the dynamic performance of the embroidery thread during the embroidery process. On the other hand, when the embroidery thread tension is not appropriate, mechanical motion errors or other embroidery-related problems cause the pattern to deform, the directly collected images cannot provide a clear basis for analysis, and it is difficult to quickly locate the problem.

[0003] This kind of data cannot directly reflect the current status of embroidery problems, which leads to the need for workers to rely on subjective judgment in actual operation and manually analyze the embroidery deformation by observing image data. This method is not only time-consuming and labor-intensive, but also the analysis accuracy may be affected due to human factors. The problem of low efficiency makes it difficult to detect and solve embroidery defects in a timely manner during the production process, further reducing the response speed of the production line and the consistency of the finished product quality. Summary of the invention

[0004] In order to solve the above problems, the present invention provides an embroidery data acquisition device using an industrial area array camera.

[0005] The embroidery data acquisition device using an industrial area array camera of the present invention adopts the following technical solution:

[0006] An embodiment of the present invention provides an embroidery data acquisition device using an industrial area array camera. The device can be installed on an embroidery machine or a main body bracket. When installed on the main body bracket, the device is used to collect embroidery data in a manual embroidery process; when installed on the embroidery machine, the device is used to collect embroidery data during automatic embroidery of the embroidery machine. The device includes: cloth, a first camera and a second camera. The first image acquired by the first camera is used to capture the embroidery process on the cloth in real time. The second camera is installed above the cloth, and the second image acquired by the second camera contains the embroidery pattern embroidered on the cloth. The acquisition device also includes an acquisition system, and the system includes:

[0007] A stitch track acquisition module, used for forming a stitch track from all stitch points in the first image, wherein the stitch track is composed of a plurality of independent tracks;

[0008] A stitch track fusion module, for matching the stitch track to the second image;

[0009] A deformation data collection module is used to collect tension deformation areas and non-tension deformation areas in the embroidery process;

[0010] The method for obtaining the tension deformation area and the non-tension deformation area comprises: clustering all stitch points in the stitch track into a plurality of stitch categories, recording the difference between the texture in the second image and the texture of the pattern template in the local area where the stitch category is located as the local embroidery deformation degree of the stitch category; for adjacent stitch points of stitch points in the stitch category on the stitch track, clustering the adjacent stitch points, and recording the obtained categories as adjacent stitch categories of the stitch category, and obtaining a local embroidery deformation degree corresponding to each adjacent stitch category, and using all stitch categories and the local embroidery deformation degrees of all adjacent stitch categories, dividing the local areas where all stitch categories are located into tension deformation areas and non-tension deformation areas;

[0011] A deformation error feedback module, used for re-obtaining a stitch track on the second image according to the distribution of the tension deformation area along the stitch track, wherein the stitch track on the second image has a minimum estimation error, and the estimation error of the stitch track is obtained by the distribution number of the tension deformation area on each independent track in the stitch track;

[0012] The tension deformation area and the non-tension deformation area existing in the embroidery process are re-collected using the stitch track on the re-acquired second image.

[0013] Preferably, the step of forming a stitch track from all stitch points in the first image comprises the following specific steps:

[0014] transforming each first image into a top view of the pin from a top-down perspective;

[0015] For the stitch top views of the first images acquired twice adjacently, the displacement vector formed by the stitch points in the stitch top view is obtained, and the displacement vectors obtained in all adjacent first images are spliced ​​end to end to form a stitch trajectory.

[0016] Preferably, the step of matching the stitch track to the second image includes the following specific steps:

[0017] For a first image captured when a second image is captured, the stitch points contained in the first image are within a stitch trajectory;

[0018] Corner point matching is performed on the stitch top view of the first image and the second image to obtain a number of corner point pairs; the stitch top view and the stitch trajectory are translated until the Euclidean distance between the two corner points in each corner point pair is minimized, so that the stitch trajectory is matched to the second image.

[0019] Preferably, for the adjacent stitch points of stitch points within a stitch category on the stitch track, the adjacent stitch points are clustered, and the obtained categories are recorded as the adjacent stitch categories of the stitch category, including the following specific steps:

[0020] Any stitch category is recorded as a target category, and any stitch point in the target category is recorded as a target stitch point; the adjacent stitch points include the previous adjacent stitch point and the next adjacent stitch point of the target stitch point in the stitch track;

[0021] The adjacent stitch categories of the target category include a first category and a second category, wherein the previous adjacent stitch points of all stitch points in the target category are clustered to obtain the first category; and the next adjacent stitch points of all stitch points in the target category are clustered to obtain the second category.

[0022] Preferably, the specific steps for obtaining the local embroidery deformation degree are as follows:

[0023] For the convex hull area where all stitch points in a stitch category or all stitch points in adjacent stitch categories are located, the second image is grayed to obtain a second grayscale image, and the pixels in the convex hull area on the second grayscale image constitute the first area. The pattern template is converted into a grayscale image, which is recorded as a third grayscale image. The pixels in the convex hull area on the third grayscale image constitute the second area, and the texture difference between the pixels in the first area and the second area is recorded as the degree of local embroidery deformation.

[0024] Preferably, the method of utilizing the local embroidery deformation degrees of all stitch categories and all adjacent stitch categories to divide the local areas where all stitch categories are located into tension deformation areas and non-tension deformation areas includes the following specific steps:

[0025] The average of the local embroidery deformation degrees corresponding to all adjacent stitch categories of each stitch category is recorded as the adjacent deformation degree of each stitch category; the difference between the local embroidery deformation degree and the adjacent deformation degree of each stitch category is recorded as the deformation difference of each stitch category;

[0026] Using the Otsu threshold segmentation algorithm, the deformation differences of all stitch categories are divided into two parts. The convex hull area of ​​the stitch category corresponding to the part with the smallest average deformation difference is recorded as the tension deformation area; the convex hull area of ​​the stitch category corresponding to the part with the largest average deformation difference is recorded as the non-tension deformation area.

[0027] Preferably, the specific steps of obtaining the estimated error are as follows:

[0028] The number of all tension-deformation regions on the same independent trajectory is recorded as the tension-deformation distribution characteristic of each independent trajectory;

[0029] The Otsu threshold segmentation algorithm is used to divide the tension deformation distribution characteristics of all independent trajectories into two parts. The part with the smallest average value of the tension deformation distribution characteristics is recorded as the first part. The number of independent trajectories in the first part is recorded as H. The ratio of H to the average value of the tension deformation distribution characteristics in the first part is obtained, and the ratio is used as the estimation error of the stitch trajectory.

[0030] Preferably, the step of re-obtaining the stitch track on the second image according to the distribution of the tension deformation area along the stitch track comprises the following specific steps:

[0031] For the several corner point pairs obtained by corner point matching of the stitch top view of the first image and the second image, when one corner point pair is retained, the stitch trajectory is re-matched to the second image, and the estimated error of the stitch trajectory is regained, which is recorded as the estimated error of each corner point pair. The N1 corner point pairs with the smallest estimated errors are retained, and the stitch trajectory is re-matched to the second image using the N1 corner point pairs to obtain the stitch trajectory with the smallest estimated error; N1 is a preset integer.

[0032] Preferably, the stitch point represents the position where the embroidery needle contacts the cloth in the first image.

[0033] The beneficial effects of the technical solution of the present invention are:

[0034] The stitch track acquisition module of the present invention forms a stitch track from all stitch points in the first image; the stitch track fusion module matches the stitch track to the second image. This process uses the stitch track of the second image to intuitively display the embroidery texture produced by the machine needle in different moving directions and moving distances, which is helpful for the staff to judge the problems of the stitches (such as too dense stitches, unreasonable stitch order, etc.) according to the deformation and stretching of the embroidery texture produced in different moving directions and moving distances, so as to make timely adjustments to the embroidery process.

[0035] Furthermore, the deformation data acquisition module of the present invention is used to acquire the tension deformation area and non-tension deformation area in the embroidery process. This process can further acquire the texture deformation area that may exist in the embroidery process based on the stitch track of the second image, so that the staff can timely discover and locate the deformation and stretching of the embroidery texture, including the deformation and stretching caused by the large tension of the embroidery thread, as well as the deformation caused by mechanical motion error and the over-tightening or over-loosening of the cloth. It is helpful to make timely and accurate adjustments to the embroidery process.

[0036] Finally, the deformation error feedback module of the present invention is used to regain the stitch trajectory on the second image and regain the tension deformation area and the non-tension deformation area according to the distribution of the tension deformation area along the stitch trajectory, thereby avoiding the problem of unreliable data acquisition results and inaccurate adjustment of the embroidery process caused by the interference of errors in the data acquisition process. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is a schematic diagram of the overall structure of an embroidery device for collecting embroidery data using an industrial area array camera provided by an embodiment of the present invention;

[0039] Figure 2 It is a top view of an overall embroidery device for collecting embroidery data using an industrial area array camera provided by an embodiment of the present invention;

[0040] Figure 3 It is an overall left view of an embroidery device for collecting embroidery data using an industrial area array camera provided by an embodiment of the present invention;

[0041] Figure 4 This is an overall front view of an embroidery device for collecting embroidery data using an industrial area array camera provided by an embodiment of the present invention;

[0042] Figure 5 It is a schematic diagram of the needle structure of an embroidery device for collecting embroidery data using an industrial area array camera provided by an embodiment of the present invention;

[0043] Figure 6 It is a system structure diagram of an embroidery data acquisition system using an industrial area array camera provided by an embodiment of the present invention;

[0044] Figure 7 It is a flowchart of the steps of a deformation data acquisition module of an embroidery data acquisition system using an industrial area array camera provided by an embodiment of the present invention;

[0045] Figure 8 It is a flowchart of the steps of a deformation error feedback module of an embroidery data acquisition system using an industrial area array camera provided by an embodiment of the present invention;

[0046] Fig. 9is a schematic diagram of the overall structure of an embroidery data acquisition device using an industrial area array camera provided by another embodiment;

[0047] Fig.10 is a left view of the entire embroidery data acquisition device using an industrial area array camera provided by another embodiment;

[0048] Fig.11 It is a top view of the entire embroidery data acquisition device using an industrial area array camera provided by another embodiment.

[0049] The accompanying drawings are marked as follows: 1. Base; 2. Workbench; 3. Embroidery machine needle assembly; 4. Embroidery machine bracket; 5. Embroidery frame; 6. Cloth; 7. Needle; 8. Camera bracket; 9. Second camera; 10. First camera; 11. Needle travel sensor; 12. Main bracket; 13. Camera gimbal; 14. Display. DETAILED DESCRIPTION

[0050] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation, structure, features and effects of an embroidery data acquisition device using an industrial array camera proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0051] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0052] The following is a detailed description of a specific solution of an embroidery data acquisition device using an industrial area array camera provided by the present invention in conjunction with the accompanying drawings. Example

[0053] like Figures 1 to 5As shown, this embodiment provides an embroidery device for collecting embroidery data using an industrial array camera. The device can not only be used for automated embroidery, but also can use the industrial array camera to collect embroidery data in the automated embroidery process. The device includes: an embroidery machine needle assembly 3, a needle 7, a first camera 10, a needle stroke sensor 11, and a second camera 9. The embroidery machine needle assembly 3 is the main equipment for embroidery. A plurality of needles 7 are installed below the embroidery machine needle assembly 3. The embroidery machine needle assembly 3 can independently control each needle 7 to move up and down reciprocatingly. During the up and down reciprocating movement, each needle 7 inserts the embroidery thread connected thereto into the cloth 6 from different positions and then passes it out, thereby forming an embroidery pattern on the cloth 6. Different needles 7 are respectively connected to embroidery threads of different colors or materials. The embroidery machine needle assembly 3 controls the plurality of needles 7 to form embroidery patterns of different colors and textures on the cloth 6.

[0054] It should be noted that the embroidery machine needle assembly 3 needs to input a pattern template when embroidering, and the pattern template contains the specific details and outline of the pattern to be embroidered. In this embodiment, the pattern template is an image file in jpg format. The embroidery machine needle assembly 3 embroiders according to the texture details shown in the pattern template. The embroidery machine needle assembly 3 is a prior art, and its specific structure and working principle are not described in detail in this embodiment.

[0055] The first camera 10 is an industrial area array camera, which is fixed on the embroidery machine needle assembly 3 and is used to capture images of the process in which the needle 7 embroiders the cloth 6. In this embodiment, the first camera 10 captures grayscale images. The first camera 10 is tilted downward toward the needle 7 to ensure that the first camera 10 can capture the process of the needle 7 penetrating into and out of the cloth 6.

[0056] In other embodiments, a plurality of first cameras 10 may be installed on the embroidery machine needle assembly 3 to ensure that the process of any needle 7 penetrating into and out of the cloth 6 can be captured by the first camera.

[0057] All embodiments of the present invention do not limit the installation method and the number of installations of the first camera 10 , and only need to be able to capture the process of the needle 7 penetrating into and out of the cloth 6 , and capture the local embroidery image on the cloth 6 under the needle 7 .

[0058] The needle stroke sensor 11 is used to obtain the stroke distance of the needle 7 when it reciprocates up and down. When the needle 7 is at the top, the stroke distance is 0, and when the needle 7 moves to the bottom, the stroke distance reaches the maximum value.

[0059] The embroidery machine needle assembly 3 is fixed on the embroidery machine bracket 4. Multiple embroidery machine needle assemblies 3 can be fixed on the embroidery machine bracket 4 at the same time, so that multiple patterns can be embroidered on the cloth 6 in parallel. The embroidery machine bracket 4 is fixed on the workbench 2, and the workbench 2 is fixed on the base 1. In addition, if Figure 1 , 3 As shown in FIG. 4 , a camera bracket 8 is fixed on the workbench 2. The camera bracket 8 is an inverted U-shaped structure. The camera bracket 8 can perform linear motion on the workbench 2 (i.e., Figure 4 The second camera 9 is installed on the crossbeam above the camera support 8. The second camera 9 has a vertical downward viewing angle, and the second camera 9 can move on the crossbeam above the camera support 8 (the second camera 9 is controlled by a screw rod structure in this embodiment, and a linear motor can be installed on the crossbeam above the camera support 8 in other embodiments to control the movement of the second camera 9).

[0060] The cloth 6 is fixed on the embroidery frame 5 and keeps the cloth 6 flat.

[0061] The embroidery frame 5 moves relative to the workbench 2; specifically, the workbench 2 is fixed, and the embroidery frame 5 moves along the horizontal plane on the workbench 2, and its movement trajectory is the trajectory of the image to be embroidered, so that the embroidery machine needle assembly 3 on the workbench 2 can embroider a pattern on the cloth 6. In this embodiment, the movement of the embroidery frame 5 is controlled by a mutually perpendicular screw rod structure, and in other embodiments, a linear motor can be used for control.

[0062] The working principle of the embroidery device is as follows: during the embroidery process, the first camera 10 collects images in real time, which are recorded as first images. The second camera 9 moves to different positions above the cloth 6 to collect image data of the embroidered pattern, which are recorded as second images.

[0063] In this embodiment, the movement process of the second camera 9 is manually remotely controlled, for example, the remote controller is used to control the camera support 8 to move on the workbench 2, and the remote controller is used to control the second camera 9 to move on the camera support 8, so that the second camera 9 is finally moved to different positions above the cloth 6 to collect image information of the pattern embroidered on the cloth 6. In other embodiments, the second camera 9 can move according to a pre-set trajectory, and then collect the patterns embroidered at different positions on the cloth 6 at different set time points and at different set positions.

[0064] For the first image and the second image obtained, the first image is used to monitor or record the movement of the needle 7 during the embroidery process, and the second image is used to sample the specific visual effect of the embroidered pattern. The computer (not marked in the drawings of the specification) on the embroidery device described in this embodiment stores the collected first image and the second image in a database (for example, a MySql database in the computer), and on the other hand, an embroidery data acquisition system using an industrial array camera is run on the computer to further collect embroidery data based on the first image and the second image. In addition, the collected second image is displayed on the display screen, for example, the second image is transmitted to the display screen using a network cable or a 5G network; the purpose is that the staff can judge whether there is an abnormality in the embroidery process by observing the embroidery pattern in the second image on the display screen. When there is an abnormality (for example, when the pattern is deformed), the second image stored in the database is further checked, and the specific problems existing in the embroidery process (for example, too dense stitches, too much tension of the embroidery thread, abnormal stitch trajectory, etc.) are traced back using the second image, and the embroidery machine needle assembly 3 is adjusted according to the specific problems to ensure the embroidery quality. Example

[0065] The problem considered in this embodiment is that although the first image and the second image directly collected by the embroidery device described in the first embodiment contain a large amount of information, such as the first image contains the specific embroidery action of the needle 7 and the texture details formed before and after embroidery, and the second image contains the texture information of the embroidery pattern, the pattern outline information and the image color distribution information. However, it is unable to intuitively reflect other important information, such as the stitch track information of the needle 7 and the texture information formed under different stitch tracks. In other words, such important information often requires the staff to spend energy to query, mark and identify from the first image and the second image, resulting in the data collected by the embroidery device (i.e., the first image and the second image) being unable to provide intuitive and important information for the embroidery process.

[0066] This embodiment provides an embroidery data acquisition system using an industrial area array camera. The system runs on the computer described in the first embodiment and is used to further acquire embroidery data based on the first image and the second image to provide intuitive and important information for the embroidery process.

[0067] like Figure 6 As shown, the system specifically includes: a stitch trajectory acquisition module, a stitch trajectory fusion module and a data visualization module.

[0068] The stitch track acquisition module is used to acquire the stitch track. The stitch track represents the embroidery track of the machine needle 7 during the embroidery process. The method for acquiring the stitch track is: acquiring the stitches of the adjacent embroidery process according to the stitch points in the first image acquired during the adjacent embroidery process, and the stitches of all adjacent embroidery processes constitute the stitch track.

[0069] The stitch track fusion module is used to display the stitch track in the second image.

[0070] The data visualization module is used to display the stitch trajectory on the second image in real time.

[0071] As an embodiment of the stitch track acquisition module, the module acquires stitches of adjacent embroidery processes according to stitch points in the first image acquired in adjacent embroidery processes, and the stitches of all adjacent embroidery processes constitute a stitch track, and the specific steps included are as follows:

[0072] (1) Acquire the first image.

[0073] When the needle 7 is at the top, the distance D between the needle tip of the needle 7 and the cloth 6 is measured, and the needle stroke sensor 11 is used to obtain the travel distance of the needle 7 in real time. When the travel distance of the needle 7 when moving downward is equal to D, the first camera collects image data once to obtain a first image. The first image obtained at this time captures the image information when the needle 7 pierces the cloth 6.

[0074] During the embroidery process, the needle 7 continuously penetrates into and out of the cloth 6, and the collected multiple first images describe the stitch information during the embroidery process.

[0075] (2) Get the stitch points in the first image.

[0076] The stitch point described in this embodiment refers to the insertion position of the needle 7 on the cloth 6.

[0077] As an example, the method of obtaining the pin point in the first image includes:

[0078] The ROI area of ​​the needle 7 is marked on the first image in advance, and the edge detection is performed on the ROI area. In this embodiment, the Canny edge detection algorithm is used to obtain the edge in the ROI area. The longest vertical edge in the ROI area is obtained, which represents the contour edge of the needle. The bottom end of the edge is marked as the stitch point.

[0079] The calculation method for this example is simple and fast.

[0080] As another example, obtaining the pin point in the first image includes the following method:

[0081] Although the first example mentioned above is simple and fast, when there are other edge interferences in the ROI area of ​​the needle 7 (for example, when there are long vertical edges on the embroidery pattern in the cloth 6), the stitch points may be obtained incorrectly.

[0082] In this example, a semantic segmentation neural network (such as DeepLabV3 neural network) is used to obtain the semantic area of ​​the sewing needle 7 in the first image, and the bottom pixel of the semantic area is used as the stitch point. If there are multiple bottom pixels in the semantic area, the center point of these pixels is used as the stitch point.

[0083] The pin points obtained in this example are relatively more accurate and reliable.

[0084] As another example, obtaining the pin point in the first image includes the following method:

[0085] The pixel points where the needle 7 contacts the cloth 6 are manually marked in the first image in advance and recorded as stitch points. However, it is necessary to ensure that the first camera 10 is stable and motionless (that is, it does not vibrate relative to the embroidery machine needle assembly 3) during the entire embroidery process, and the parameters of the first camera 10 (such as focal length, viewing angle, etc.) remain unchanged.

[0086] This example is the simplest and fastest, but requires high installation conditions for the first camera 10 .

[0087] (3) Map the first image into a top-down perspective to obtain a top view of the stitches.

[0088] Considering that in some embodiments, the viewing angle of the first camera 10 is obliquely downward, in this case, the present embodiment needs to transform the captured first image into a top-down viewing angle for the convenience of subsequent processing.

[0089] As an example, the first image is transformed into a top-down perspective, including the following method:

[0090] Before embroidery, the first camera 10 on the embroidery machine needle assembly 3 is used to capture an image of the chessboard, which is recorded as an oblique angle image. In addition, the second camera 9 is used to capture the image of the chessboard again at a top-down angle, which is recorded as a top-down angle image.

[0091] The homography matrix is ​​obtained according to the oblique view image and the overhead view image. The homography matrix refers to a transformation matrix that affine transforms the oblique view image into the overhead view image. This process is well known. For example, CN113487580B discloses a method and system for calculating the overlap of drone images based on polygon analysis, which includes the process of obtaining the homography matrix. This embodiment will not be described in detail.

[0092] For the first image collected during the embroidery process, the homography matrix is ​​used to perform an affine transformation on the first image, and the transformed result is recorded as a stitch top view.

[0093] If in some embodiments, the first camera 10 has a top-down perspective, the first image is directly used as a top view of the stitches.

[0094] It should also be noted that when multiple first cameras 10 are installed in the embroidery machine needle assembly 3, each first camera 10 corresponds to capturing the embroidery process of one or more needles 7. At this time, each needle 7 corresponds to a top view of the stitch.

[0095] (4) Obtain the stitch trajectory according to the stitch top view during the embroidery process and the stitch points on the stitch top view.

[0096] For the stitch top views of two adjacent first images collected during the embroidery process (abbreviated as adjacent stitch top views), the adjacent stitch top views are matched with corner points to obtain a number of corner point pairs, and the displacement of the needle 7 is obtained based on all the corner point pairs. The displacement of the needle 7 describes the displacement generated by the needle 7 when it embroiders twice adjacently during the embroidery process (that is, the position of the last insertion into the cloth 6 and the next insertion into the cloth 6) (the displacement represents a two-dimensional vector, also recorded as a displacement vector).

[0097] For the top views of adjacent stitches, the position of the stitch point in the top view of the latter stitch is recorded as d, and the displacement vector with d as the end point is recorded as the stitch of the adjacent embroidery process.

[0098] The stitches of all adjacent embroidery processes are spliced ​​end to end to form a stitch track. In other words, in adjacent embroidery processes, the end point of the displacement vector corresponding to the stitch of the previous embroidery process coincides with the starting point of the displacement vector corresponding to the stitch of the next embroidery process to obtain a stitch track.

[0099] The stitch trajectory represents the embroidery trajectory of the needle 7 during the embroidery process, that is, it includes the moving direction and moving distance of the needle in adjacent embroidery processes, and describes the action behavior of the needle 7 during the embroidery process.

[0100] As an example, the displacement of the needle 7 is obtained according to all the corner point pairs, including the following method:

[0101] The displacement vector formed by each corner point pair is obtained, and the average vector of the displacement vectors formed by all corner point pairs is used as the displacement of the needle 7.

[0102] It should be noted that the corner points contained in all the corner point pairs are obtained by using the SIFT corner point detection algorithm, and describe the position coordinates of adjacent stitches with the same texture features in the top view.

[0103] It should be further explained that, in this example, when performing corner point matching, what is mainly matched is the texture of the embroidery pattern in the stitch top view. In order to avoid interference from the texture of the needle 7 in the stitch top view, corner point matching is only performed on the corner points outside the ROI area where the needle 7 is located.

[0104] As an embodiment of the stitch track fusion module, the module displays the stitch track in the second image, including the steps of:

[0105] (1) Match the top view of the pin to the second image.

[0106] Whenever the second camera 9 captures a second image, the first image at that moment is acquired.

[0107] It should be noted that when the second camera 9 captures the second image but the first image has not been captured, the most recently captured first image is acquired. In some embodiments, there are multiple embroidery machine needle assemblies 3 embroidering on the cloth 6 in parallel. At this time, the embroidery pattern contained in the second image is embroidered by a certain embroidery machine needle assembly 3. At this time, the first image refers to the image captured when the needle 7 in the embroidery machine needle assembly 3 is embroidering.

[0108] For the stitch top view of the first image, the stitch top view is matched to the second image, including the following method: matching the stitch top view with the second image for corner points, and obtaining a plurality of corner point pairs. The stitch top view is translated so that the two corner points in the corner point pairs overlap.

[0109] It should be noted that, due to errors in corner point detection and corner point matching, when the two corner points in each corner point pair cannot be made to coincide, the stitch top view is translated until the Euclidean distance between the two corner points in each corner point pair is minimized. Specifically: the stitch top view is translated, and the Euclidean distances between the corner points in all corner point pairs are summed, and the simulated annealing algorithm is used to obtain the position to which the stitch top view is translated when the sum is maximized, and the stitch top view at the translated position is matched with the second image (or the stitch top view is matched to the second image).

[0110] In addition, corner point matching is a prior art in image processing technology, and will not be described in detail in this embodiment.

[0111] (2) Match the stitch trajectory to the second image.

[0112] The stitch top view described in (1) includes stitch points, which are located on the stitch trajectory; when the stitch top view is translated so that the stitch top view is matched to the second image, the stitch trajectory also follows the translation of the stitch top view, and finally when the stitch top view is matched to the second image, the stitch trajectory is also matched to the second image.

[0113] The purpose of matching the stitch track to the second image in this embodiment is that since the stitch track can only indicate the moving direction and moving distance of the needle 7 during the embroidery process, it is not possible to intuitively display the embroidery texture generated by the needle 7 at different moving directions and moving distances. When the stitch track is also matched to the second image, the moving process of the needle 7 can be matched with the embroidery texture generated at different moving directions and moving distances, so that after the data visualization module displays the second image and the stitch track at the same time, it is helpful for the staff to judge the problems with the stitches (such as too dense stitches, unreasonable stitch order, etc.) according to the deformation and stretching of the embroidery texture generated at different moving directions and moving distances, so as to make timely adjustments to the embroidery process. Example

[0114] The embroidery data acquisition system using an industrial area array camera provided in this embodiment also includes a deformation data acquisition module, such as Figure 6 In addition Figure 7 The steps included in the module are shown in the figure; the module takes into account that: in the second embodiment, by collecting the stitch tracks and integrating them into the second image, although it can intuitively display the deformation and stretching of the embroidery texture that may occur under different moving directions and moving distances, when the stitch tracks are seriously intertwined, it is difficult to directly or timely discover the deformation and stretching of the embroidery texture. This module is used to collect the texture deformation areas that may exist in the embroidery process and display them in the data visualization module described in the second embodiment, so that the staff can timely discover the deformation and stretching of the embroidery texture.

[0115] The method of collecting the texture deformation areas that may exist in the embroidery process includes:

[0116] All stitch points on the stitch track in the second image are clustered to obtain all categories, which are recorded as stitch categories. The stitch points in the same stitch category have similar distances, and the embroidery threads corresponding to these stitch points with similar distances (or concentrated stitch points) may cause stretching deformation to the local texture of the embroidery pattern. The main reasons for this stretching deformation are: the tension of the embroidery thread is relatively too large (for example, on the cloth 6 that is easy to elastically deform, the embroidery thread with a large tension causes the embroidery pattern to be uneven), or the densely distributed stitch points in the local range of the cloth 6 affect the elasticity of the cloth, resulting in the tension of the embroidery thread not matching the local elasticity of the cloth 6, and finally the local texture of the cloth 6 is deformed, such as abnormal bending of the edge texture of the embroidery image, too large gaps between the stitch points, unnatural color transition of the pattern, uneven and wrinkled pattern, and even local tearing or the risk of thread breakage on the cloth 6 of certain materials.

[0117] This embodiment further collects the stretching deformation that occurs during the embroidery process:

[0118] The convex hull area formed by all stitch points in each stitch category is obtained, and the difference between the texture on the second image and the texture of the pattern template in the convex hull area is obtained, and recorded as the local embroidery deformation degree of each stitch category. The local embroidery deformation degree describes the texture difference between the pattern embroidered by the embroidery machine needle assembly 3 in the local area and the standard reference pattern. The larger the value, the more serious the deformation of the pattern embroidered by the embroidery machine needle assembly 3.

[0119] It is further considered that the deformation of the pattern embroidered by the embroidery machine needle assembly 3 is not only caused by the mismatch between the tension of the embroidery thread and the local elasticity of the cloth 6, but may also be caused by other reasons, such as mechanical movement error of the embroidery frame 5, the cloth 6 being stretched too tight or too loose, and the needle 7 causing too much impact on the cloth 6. This embodiment further obtains the deformation caused by the mismatch between the tension of the embroidery thread and the local elasticity of the cloth 6 through the following methods (1) and (2):

[0120] (1) Any stitch category is recorded as the target category. Any stitch point in the target category is recorded as the target stitch point. In the stitch track of the second image, the previous adjacent stitch point of the target stitch point is obtained. The previous adjacent stitch points of all stitch points in the target category are clustered to obtain all first categories. The local embroidery deformation degree corresponding to each first category is obtained. The mean of the local embroidery deformation degrees corresponding to all first categories is recorded as the first deformation degree of the target category. Similarly, in the stitch track of the second image, the next adjacent stitch point of the target stitch point is obtained. The next adjacent stitch points of all stitch points in the target category are clustered to obtain all second categories. The local embroidery deformation degree corresponding to each second category is obtained. The mean of the local embroidery deformation degrees corresponding to all second categories is recorded as the second deformation degree of the target category.

[0121] It should be noted that if the target pin point does not have a previous adjacent pin point or a next adjacent pin point, then the pin point is removed from the target category.

[0122] The first category and the second category above represent clusters formed by stitch points adjacent to the stitch points in the target category, so both can also be called adjacent stitch categories of the target category, corresponding to a local embroidery deformation degree (ie, the first deformation degree and the second deformation degree) respectively.

[0123] (2) The tension deformation area and the non-tension deformation area are obtained by using the local embroidery deformation degree, the first deformation degree and the second deformation degree of each stitch category (that is, the tension deformation area and the non-tension deformation area are obtained by using the local embroidery deformation degree of each stitch category and the local embroidery deformation degree of the adjacent stitch category); wherein the tension deformation area refers to the local area deformed due to the mismatch between the embroidery thread tension and the elasticity of the cloth 6 in the local area; the non-tension deformation area refers to the local area deformed due to other reasons, such as the deformation area caused by the mechanical movement error of the embroidery frame 5 and the cloth 6 being stretched too tight or too loose.

[0124] As an example, the tension deformation area and the non-tension deformation area are obtained by using the local embroidery deformation degree, the first deformation degree, and the second deformation degree of each stitch category, including the following method:

[0125] For all stitch categories, the average of the local embroidery deformation degree, the first deformation degree, and the second deformation degree of each stitch category is taken as the comprehensive embroidery deformation degree of each stitch category. For stitch categories with a comprehensive embroidery deformation degree greater than or equal to th1 and less than or equal to th2, the convex hull area where all stitch points in each stitch category are located is recorded as the non-tension deformation area; for stitch categories with a comprehensive embroidery deformation degree greater than or equal to th2, the convex hull area where all stitch points in each stitch category are located is recorded as the tension deformation area.

[0126] This embodiment is described by taking th1=0.2 and th2=0.5 as an example. In other embodiments, th1 and th2 may be set to other values, which are not specifically limited in this embodiment.

[0127] In this example, considering that the embroidery thread tension is larger than the elasticity of the cloth 6, it will cause multiple consecutive embroidery processes (or multiple consecutive stitch points will be deformed). When the comprehensive embroidery deformation degree is greater than or equal to th2, the stitch category indicates that there are large deformations in the adjacent embroidery processes, which further indicates that the deformation is more likely to be caused by the large embroidery thread tension. When the comprehensive embroidery deformation degree is greater than or equal to th1 and less than or equal to th2, it indicates that there is no large deformation in the adjacent embroidery process, and it may be an accidental deformation caused by factors such as mechanical motion errors.

[0128] As a preferred example, the tension deformation area and the non-tension deformation area are obtained by using the local embroidery deformation degree, the first deformation degree, and the second deformation degree of each stitch category, including the following method:

[0129] For all stitch categories, the average of the local embroidery deformation degree, the first deformation degree and the second deformation degree of each stitch category is taken as the comprehensive embroidery deformation degree of each stitch category, and the stitch categories with a comprehensive embroidery deformation degree less than a preset threshold th1 are deleted.

[0130] For the remaining stitch categories, the local embroidery deformation degree, the first deformation degree, and the second deformation degree of each stitch category are used. The difference between any two of the three deformation degrees is calculated and the absolute value is removed. The result is recorded as the first difference. The average of all first differences obtained by the three deformation degrees is recorded as the deformation difference of the local area in adjacent embroidery processes, which is simply recorded as the deformation difference of each stitch category. The Otsu threshold segmentation algorithm is used to divide the deformation differences of all stitch categories into two parts. For the stitch category corresponding to the part with the smallest average deformation difference, the convex hull area where the stitch points in the stitch category are located is recorded as the tension deformation area; for the stitch category corresponding to the part with the largest average deformation difference, the convex hull area where the stitch points in the stitch category are located is recorded as the non-tension deformation area.

[0131] In this example, the deformation difference between adjacent embroidery processes is used to distinguish between tension deformation areas and non-tension deformation areas. Although the amount of calculation is increased, it does not depend on the specific threshold settings (that is, the settings of th1 and th2).

[0132] The above-mentioned tension deformation area and non-tension deformation area are the data collected by the deformation data collection module of this embodiment, and the tension deformation area and non-tension deformation area are displayed in the data visualization module described in the second embodiment, for example, the tension deformation area and non-tension deformation area are displayed with different color blocks. According to the tension deformation area and non-tension deformation area, the staff can quickly locate the position of unqualified embroidery (for example, the position where the embroidery thread is too tight or the stitches are too dense), and it can also help the embroidery monitoring personnel to further determine the problems existing in the embroidery process (for example, the problem of large mechanical motion error caused by equipment aging or failure). The staff can adjust the embroidery device according to the collected data, such as adjusting the tension of the embroidery thread, the density of the stitches, and the movement frequency of the needle 7. The specific adjustment method is well known and will not be described in detail in this embodiment.

[0133] As an optional example, the difference between the texture on the second image and in the convex hull area and the texture of the pattern template is obtained and recorded as the local embroidery deformation degree of each stitch category, including the following method:

[0134] The second image is grayed to obtain a second grayscale image. The pixels in the convex hull area of ​​the second grayscale image constitute the first area. The pattern template is converted into a grayscale image, which is recorded as a third grayscale image. The pixels in the convex hull area of ​​the third grayscale image constitute the second area. The absolute value of the difference between the grayscale values ​​of the pixels at the same position in the first area and the second area is recorded as the grayscale difference of the pixels. The mean of the grayscale differences of all the pixels in the first area is recorded as Q. The ratio of Q to 255 is recorded as the local embroidery deformation degree of each category. 255 represents the maximum value of the grayscale value of the pixels in the second grayscale image.

[0135] As a preferred example, the difference between the texture on the second image and in the convex hull area and the texture of the pattern template is obtained and recorded as the local embroidery deformation degree of each stitch category, including the method of:

[0136] The Soble operator is used to obtain the gradient of each pixel point in the first area (the gradient refers to a two-dimensional vector composed of the gradient direction and the gradient amplitude). Similarly, the Soble operator is used to obtain the gradient of each pixel point in the second area. The cosine distance of the gradient of the pixel points at the same position in the first area and the second area is recorded as the texture difference of the pixel points. The mean of the texture differences of all the pixels in the first area is recorded as Q, and Q is recorded as the local embroidery deformation degree of each stitch category.

[0137] Compared with the optional example, this preferred example can avoid the influence of light interference on the calculation result of the local embroidery deformation degree.

[0138] The calculation method of the local embroidery deformation degree corresponding to the first category and the local embroidery deformation degree corresponding to the second category are similar.

[0139] As an example, the clustering method used when clustering the stitch points is the mean shift clustering algorithm. Example

[0140] This embodiment takes into account that the tension deformation area and the non-tension deformation area in the third embodiment are obtained based on the deformation of the texture corresponding to the stitch track on the second image. When the stitch track and the texture on the second image do not correspond, for example, due to matching the stitch track to the second image in step (2) of the second embodiment, when there is an error in the process, it results in that the stitch track cannot be accurately matched to the second image. The reason for the existence of this error is, on the one hand, the algorithm error in the corner point matching result in Example 2; on the other hand, in Example 2, the corner point matching is performed using the stitch top view of the first image and the second image, wherein the first image and the second image are respectively captured by different cameras (respectively, the second camera 9 and the first camera 10), and the two cameras may have different parameters (such as resolution, lens distortion, shutter frequency, etc.). Even if these parameters are set the same in some embodiments, the second camera 9 captures a large-area embroidery pattern with a larger field of view (so as to collect the overall pattern outline and pattern color information of the embroidery), and the first camera 10 captures the detailed texture information of the local smaller field of view of the needle 7 during the embroidery process. Therefore, the focal lengths of the second camera 9 and the first camera 10 are different, which results in the first image and the second image having texture information of different scales, resulting in the inability to accurately match the stitch trajectory to the second image in some embodiments, and ultimately resulting in possible errors in the tension deformation area and the non-tension deformation area in Example 3.

[0141] This embodiment provides a deformation error feedback module based on the third embodiment. Figure 6 As shown, the module is used to re-obtain the stitch track on the second image according to the distribution of the tension deformation area along the stitch track obtained in the third embodiment. Figure 8 The following shows the steps involved in this module.

[0142] As an example, according to the distribution of the tension deformation area along the stitch track obtained in the third embodiment, the stitch track on the second image is re-obtained, including the following method:

[0143] First of all, it should be noted that the stitch tracks described in the second and third embodiments include several independent stitch tracks (referred to as independent tracks). Specifically, in the actual embroidery process, embroidery is not performed using one needle 7, but using multiple needles on the embroidery machine needle assembly 3 to embroider alternately. For example, after one needle 7 embroiders a part of the texture, another needle 7 is used to embroider other textures. The independent track refers to: the stitch track formed during the continuous embroidery process of the same needle 7.

[0144] (1) For any independent trajectory, if any stitch point in the independent trajectory is within the tension deformation region, then the tension deformation region is on the independent estimated trajectory. Then, all tension deformation regions on the same independent trajectory are obtained, and the number of all tension deformation regions on the same independent trajectory is recorded as the distribution of tension deformation regions along each independent trajectory, which is simply recorded as the tension deformation distribution feature of each independent trajectory.

[0145] (2) For all independent trajectories whose tension-deformation distribution characteristics are not equal to 0, the estimated error of the stitch trajectory is obtained according to the distribution of the tension-deformation distribution characteristics of all independent trajectories.

[0146] When more independent trajectories have smaller tension deformation distribution characteristics, it means that most of the independent trajectories have tension deformation, but there are fewer local areas with tension deformation. Considering that excessive tension of the same embroidery thread will cause deformation in more local areas, there are generally multiple tension deformation areas. Therefore, when more independent trajectories have smaller tension deformation distribution characteristics, it means that the stitch trajectory is more likely to have a larger error.

[0147] (3) Re-matching the stitch trajectory to the second image so that the estimation error of the stitch trajectory is minimized, and the stitch trajectory with the minimum estimation error is used as the stitch trajectory on the re-acquired second image.

[0148] (4) Based on the stitch track on the retrieved second image, the tension deformation area and the non-tension deformation area are retrieved using the method in the third embodiment, and the stitch track, the tension deformation area and the non-tension deformation area on the retrieved second image are displayed in the data visualization module described in the second embodiment.

[0149] As an example, obtaining the estimated error of the stitch track according to the distribution of the tension deformation distribution characteristics of all independent tracks includes the following steps:

[0150] Obtain independent tracks whose tension deformation distribution characteristics are less than or equal to th3, and record the number of these independent tracks as H. The larger H is, the more independent tracks have smaller tension deformation distribution characteristics, and H is further used as the estimation error. In other embodiments, the ratio of H to the number of all independent tracks can also be used as the estimation error.

[0151] This embodiment is described by taking th3=3 as an example. In other embodiments, it can be set to other values, which are not specifically limited in this embodiment.

[0152] As another example, obtaining the estimated error of the stitch track according to the distribution of the tension deformation distribution characteristics of all independent tracks includes the following steps:

[0153] The tension deformation distribution characteristics of all independent trajectories are obtained, and these tension deformation distribution characteristics are divided into two parts using the Otsu threshold segmentation algorithm. The part with the smallest average value of the tension deformation distribution characteristics is recorded as the first part, and the part with the smallest average value of the tension deformation distribution characteristics is recorded as the second part. The number of independent trajectories in the first part is recorded as H, and the ratio of H to the average value of the tension deformation distribution characteristics in the first part is obtained. The larger the ratio, the more independent trajectories have smaller tension deformation distribution characteristics. The ratio is used as the estimation error.

[0154] The calculation result of this example is relatively accurate, and there is no need to set the threshold th3, but a large amount of calculation is required.

[0155] This embodiment provides another method for calculating the tension deformation distribution characteristics of each independent track described in this embodiment (1), which specifically includes:

[0156] It can be seen from Example 3 (2) that each tension deformation area corresponds to a stitch category, and the stitch category corresponds to a comprehensive embroidery deformation degree. The comprehensive embroidery deformation degrees corresponding to all tension deformation areas on each independent track are summed up, and the sum result is used as the tension deformation distribution feature of each independent track.

[0157] In this method, not only the number of tension deformation areas on each independent trajectory is combined, but also the specific deformation conditions of each tension deformation area are considered, making the tension deformation distribution characteristics more reliable.

[0158] As an example, re-matching the stitch track to the second image so that the estimation error of the stitch track is minimized includes the following method:

[0159] For the stitch track fusion module proposed in the second embodiment, the stitch top view described in step (1) of the embodiment corresponding to the module, when the stitch top view is matched with the second image for corner point, a plurality of corner point pairs are obtained, and only one corner point pair is retained among the plurality of corner point pairs, and then the stitch track of the second image is re-obtained according to the implementation method of the stitch track fusion module in the second embodiment, and the estimated error of the stitch track is obtained by the above method of the present embodiment. Then only another corner point pair is retained among the plurality of corner point pairs, and then the stitch track of the second image is re-obtained according to the implementation method of the stitch track fusion module in the second embodiment, and the estimated error of the stitch track is obtained by the above method of the present embodiment, and so on, each corner point pair among the plurality of corner point pairs corresponds to an estimated error, and the N1 corner point pairs with the smallest estimated error are retained, and the stitch track of the second image is re-obtained according to the implementation method of the stitch track fusion module in the second embodiment using these N1 corner point pairs as the stitch track with the smallest estimated error of the stitch track.

[0160] In this embodiment, N1 is equal to half of the number of all corner point pairs (rounded down). In other embodiments, N1 may be set to other values, which are not specifically limited in this embodiment. DETAILED DESCRIPTION

[0162] like Fig. 9 , Fig.10 as well as Fig.11 As shown, this embodiment provides another embroidery data acquisition device using an industrial area array camera. The differences between this device and the device described in Example 1 include: the acquisition device of this embodiment is used to obtain the pattern template described in Example 1; the embroidery device in Example 1 is used to collect the stitch tracks, tension deformation areas and non-tension deformation areas described in Example 2 and Example 4.

[0163] The similarities between this device and the device described in Example 1 include: both use multiple (at least two) industrial area array cameras; the embroidery data acquisition device of this embodiment and the embroidery device provided in Example 1 both include: an embroidery frame 5, a cloth 6, a second camera 9, and a first camera 10; in addition, the embroidery data acquisition device of this embodiment also includes: a main body bracket 12, a camera gimbal 13 and a display 14.

[0164] In this embodiment, the cloth 6 is fixed on the embroidery frame 5 (same as in the first embodiment), and the embroidery frame 5 is fixed on the main frame 12. In addition, the second camera 9 is fixed on the main frame 12, the first camera 10 is fixed on the camera platform 13, and the camera platform 13 is fixed on the main frame 12. Fig. 9 , Fig.10 As shown, the second camera 9 is above the cloth 6, and the first camera 10 is below the cloth 6; the second camera 9 and the first camera 10 are industrial array cameras, respectively facing the cloth 6, the second camera 9 is vertically facing the cloth 6, and the first camera 10 is facing different positions on the cloth 6 under the control of the camera gimbal 13. The display 14 is used to display the second image and the first image output by the second camera 9 and the first camera 10 respectively.

[0165] In some other embodiments, a reflector (not marked in the drawings in the specification) is installed on the main support 12. For example, two reflectors are installed respectively facing the upper and lower surfaces of the cloth 6 to provide fill light to the upper and lower surfaces of the cloth 6. Since the reflector device is a well-known technology, it is used here as an optional embodiment and its specific structure is not described in detail in this embodiment.

[0166] The similarities between this embodiment and the first embodiment further include: the first camera 10 is used to capture the process of the embroidery needle (equivalent to the machine needle 7 in the first embodiment) entering and passing through the cloth 6, and capturing the local embroidery image on the cloth 6 at the embroidery position of the embroidery needle, and the second camera 9 is used to capture the embroidery pattern that has been embroidered on the cloth 6.

[0167] The working process of the embroidery data acquisition device of this embodiment is as follows: an embroidery worker (e.g., an embroiderer) sits in front of the device, facing the display 14, and manually embroiders on the cloth 6. During the embroidery process, the first camera 10 captures and tracks the image of the insertion and exit position of the embroidery needle in real time under the control of the camera gimbal 13 (i.e., the first image). In this embodiment, the first camera 10 captures the first image with a larger focal length so that the embroidery needle in the first image can be distinguished; when the embroidery is completed, the second camera 9 captures the overall image of the embroidery (i.e., the second image), and the second image needs to include all the contours and textures of the embroidery image (i.e., the focal length of the second camera 9 is smaller than that of the first camera 10). Finally, the computer (not marked in the figure in the specification) on the embroidery data acquisition device of this embodiment runs a pattern template data acquisition system, which obtains and stores the pattern template based on the first image captured in real time and the second image captured at the end. Example

[0168] This embodiment provides a pattern template data acquisition system, which runs on the computer described in Example 5. When the system is running, it is used to implement: obtaining and storing the pattern template according to the first image collected in real time and the second image collected finally as described in Example 5. The pattern template data acquisition system also includes: a stitch trajectory acquisition module, a stitch trajectory fusion module, and a data storage module.

[0169] The stitch track collection module of this embodiment is similar to the stitch track collection module in the second embodiment, and includes:

[0170] (5-1) Capture the first image.

[0171] (5-2) Obtain the stitch points in the first image.

[0172] (5-3) Map the first image into a top-down perspective to obtain a top view of the pin.

[0173] (5-4) The stitch trajectory is obtained according to the stitch top view during the embroidery process and the stitch points on the stitch top view.

[0174] Specifically, collecting the first image includes the following specific methods:

[0175] In this embodiment, the first image is collected every 0.2 seconds. In other embodiments, other sampling intervals may be set, which are not specifically limited in this embodiment.

[0176] The specific method of obtaining the pin point in the first image is as follows:

[0177] The neural network technology is used to obtain the position where the embroidery needle enters and exits the cloth 6 in the first image, and the position is used as the stitch point.

[0178] In some other embodiments, obtaining the stitch point in the first image includes the following specific methods:

[0179] After the embroidery worker uses the embroidery needle to perform an action of inserting into or piercing out of the cloth 6, the embroidery worker uses the mouse to mark the position where the embroidery needle inserts into and pierces out of the cloth 6 in the first image displayed on the display 14, and the position is used as the stitch point.

[0180] It should be noted that the neural network technology refers to inputting the first image into a key point detection neural network, such as the Stacked HourGlass Network (the neural network is a well-known technology), and the neural network outputs the positions where the embroidery needle enters and exits the cloth 6.

[0181] The method of the data set for training the neural network is as follows: collecting image data of the embroidery worker's embroidery process, which includes the information of the embroidery needle insertion and withdrawal, marking the positions of the embroidery needle insertion and withdrawal, and the marks are used as labels for the image data. All the image data and their corresponding labels constitute a data set.

[0182] The neural network is trained using the data set, the loss function during training is the cross entropy loss function, and the method for updating the neural network parameters during training is the stochastic gradient descent algorithm. Since the specific training process of the neural network is well known, this embodiment will not be described in detail.

[0183] In this embodiment, the first images in which no stitch points are detected are deleted, and these first images do not participate in the subsequent analysis and processing process.

[0184] In other embodiments, after the stitch point is obtained in the first image, the camera platform 13 controls the viewing angle of the first camera 10 so that the stitch point is at the center of the field of view of the first camera 10, and then increases the focal length (for example, the focal length is automatically increased by 5%) to capture the first image again, in order to ensure that the embroidery texture around the stitch point in the first image is clear. In this embodiment, the first images in which no stitch point is detected or the stitch point is not at the center of the field of view are deleted, and these first images do not participate in the subsequent analysis and processing process.

[0185] It should be noted that, in the process of obtaining the stitch points, the camera platform 13 needs to control the first camera 10 to track the embroidery worker's gestures in real time (so that the embroidery worker's action of inserting and removing the cloth 6 can be captured). One method is that the embroidery worker manually controls the rotation of the camera platform 13 so that the first camera 10 faces the embroidery worker's hand, thereby ensuring that the embroidery worker's action of inserting and removing the cloth 6 can be captured.

[0186] Another method is: considering that the second camera 9 needs to capture the entire embroidery image with a smaller focal length, the field of view of the second camera 9 is larger than that of the first camera 10, that is, the second camera 9 can detect the position of the hand in real time with a larger field of view, and then use a semantic segmentation neural network (such as DeepLabV3) to segment the semantic area of ​​the hand from the image captured in real time by the second camera 9. The center of the semantic area is used as the position of the hand, and the position of the hand is sent to the camera gimbal 13 via Bluetooth or a data cable, so that the first camera 10 is facing the embroiderer's hand, thereby ensuring that the action of the embroiderer inserting and removing the cloth 6 can be captured.

[0187] The methods described in (5-3) and (5-4) are substantially the same as those in Example 2 and will not be described in detail in this embodiment.

[0188] The stitch track fusion module of this embodiment is the same as the stitch track fusion module described in the second embodiment, and is used to obtain the stitch track, which will not be described in detail in this embodiment.

[0189] A data storage module, the pattern template data to be stored in this module includes the pattern template (i.e., the second image) and the stitch trajectory for generating the pattern template. This module packages the pattern template data into a DST file and saves it in a data storage device (e.g., the mechanical hard disk in the computer described in Example 5). The method of packaging into a DST file is well known and will not be described in detail in this embodiment.

[0190] The pattern template included in the DST file can be used to implement all the methods included in the second embodiment. The embroidery machine needle assembly 3 in the first embodiment performs embroidery according to the stitch tracks included in the DST file.

[0191] It should be noted that the stitch track described in the second embodiment can be directly read from the DST file, but this approach will have the following problems: due to mechanical motion errors, motion control errors, or mechanical vibrations and mechanical failures of the embroidery machine needle assembly, the stitch track actually generated by the embroidery machine needle assembly 3 during the embroidery process is not completely the same as the stitch track in the DST file, and there may even be large differences in the local position of the stitch track. Therefore, in the above second embodiment of the present invention, the first camera 10 and the second camera 9 are reused to obtain the stitch track, which is steps (1) and (2) of the stitch track acquisition module in the second embodiment.

[0192] Finally, it should be noted that the devices provided in the first and fifth embodiments of the present invention both include a cloth 6, a first camera 10, and a second camera 9, wherein the first image captured by the first camera 10 is used to capture the embroidery process on the cloth 9 in real time; the second camera 9 is installed above the cloth 6, and the second image captured by the second camera 9 contains the embroidery pattern embroidered on the cloth 6. When the cloth 6, the first camera 10, and the second camera 9 are installed on the embroidery machine, the device structure described in the first embodiment is obtained, which is used to collect embroidery data during automatic embroidery of the embroidery machine. When the cloth 6, the first camera 10, and the second camera 9 are installed on the main body bracket 12, the device described in the fifth embodiment is obtained, which is used to collect embroidery data during manual embroidery. The cloth 6, the first camera 10, and the second camera 9 included in the first and fifth embodiments are the most basic embroidery data acquisition devices using industrial array cameras provided by the present invention. In other embodiments of the present invention, reasonable modifications or equivalent replacements can also be made based on the most basic embroidery data acquisition device using industrial array cameras, and the present invention does not specifically limit them.

[0193] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An embroidery data acquisition device using an industrial area array camera, the device being installable on an embroidery machine or on a main body bracket (12); when installed on the main body bracket (12), the device is used to collect embroidery data during manual embroidery; When installed on an embroidery machine, the device is used to collect embroidery data during automatic embroidery of the embroidery machine. The device comprises: A piece of cloth (6), a first camera (10) and a second camera (9), wherein a first image captured by the first camera (10) is used to capture the embroidery process on the piece of cloth (6) in real time; the second camera (9) is installed above the piece of cloth (6), and a second image captured by the second camera (9) contains an embroidery pattern embroidered on the piece of cloth (6), characterized in that the acquisition device also includes an acquisition system, and the system includes: A stitch track acquisition module, used for forming a stitch track from all stitch points in the first image, wherein the stitch track is composed of a plurality of independent tracks; A stitch track fusion module, for matching the stitch track to the second image; A deformation data collection module is used to collect tension deformation areas and non-tension deformation areas in the embroidery process; The method for obtaining the tension deformation area and the non-tension deformation area comprises: clustering all stitch points in the stitch track into a plurality of stitch categories, recording the difference between the texture in the second image and the texture of the pattern template in the local area where the stitch category is located as the local embroidery deformation degree of the stitch category; for adjacent stitch points of stitch points in the stitch category on the stitch track, clustering the adjacent stitch points, and recording the obtained categories as adjacent stitch categories of the stitch category, and obtaining a local embroidery deformation degree corresponding to each adjacent stitch category, and using all stitch categories and the local embroidery deformation degrees of all adjacent stitch categories, dividing the local areas where all stitch categories are located into tension deformation areas and non-tension deformation areas; A deformation error feedback module, used for re-obtaining a stitch track on the second image according to the distribution of the tension deformation area along the stitch track, wherein the stitch track on the second image has a minimum estimation error, and the estimation error of the stitch track is obtained by the distribution number of the tension deformation area on each independent track in the stitch track; The tension deformation area and the non-tension deformation area existing in the embroidery process are re-collected using the stitch track on the re-acquired second image.

2. The embroidery data acquisition device using an industrial area array camera according to claim 1, characterized in that: The specific steps of forming a stitch track from all stitch points in the first image are as follows: transforming each first image into a top view of the pin from a top-down perspective; For the stitch top views of the first images acquired twice adjacently, the displacement vector formed by the stitch points in the stitch top view is obtained, and the displacement vectors obtained in all adjacent first images are spliced ​​end to end to form a stitch trajectory.

3. The embroidery data acquisition device using an industrial area array camera according to claim 2, characterized in that: The specific steps of matching the stitch track to the second image are as follows: For a first image captured when a second image is captured, the stitch points contained in the first image are within a stitch trajectory; Corner point matching is performed on the stitch top view of the first image and the second image to obtain a number of corner point pairs; the stitch top view and the stitch trajectory are translated until the Euclidean distance between the two corner points in each corner point pair is minimized, so that the stitch trajectory is matched to the second image.

4. The embroidery data acquisition device using an industrial area array camera according to claim 1, characterized in that: For the adjacent stitch points of stitch points in the stitch category on the stitch track, the adjacent stitch points are clustered, and the obtained categories are recorded as the adjacent stitch categories of the stitch category, and the specific steps included are as follows: Any stitch category is recorded as a target category, and any stitch point in the target category is recorded as a target stitch point; the adjacent stitch points include the previous adjacent stitch point and the next adjacent stitch point of the target stitch point in the stitch track; The adjacent stitch categories of the target category include a first category and a second category, wherein the previous adjacent stitch points of all stitch points in the target category are clustered to obtain the first category; The next adjacent stitch points of all stitch points in the target category are clustered to obtain a second category.

5. The embroidery data acquisition device using an industrial area array camera according to claim 1, characterized in that: The specific steps for obtaining the local embroidery deformation degree are as follows: For the convex hull area where all stitch points in a stitch category or all stitch points in adjacent stitch categories are located, the second image is grayed to obtain a second grayscale image, and the pixels in the convex hull area on the second grayscale image constitute the first area. The pattern template is converted into a grayscale image, which is recorded as a third grayscale image. The pixels in the convex hull area on the third grayscale image constitute the second area, and the texture difference between the pixels in the first area and the second area is recorded as the degree of local embroidery deformation.

6. The embroidery data acquisition device using an industrial area array camera according to claim 1, characterized in that: The method utilizes the local embroidery deformation degrees of all stitch categories and all adjacent stitch categories to divide the local areas where all stitch categories are located into tension deformation areas and non-tension deformation areas, and includes the following specific steps: The average of the local embroidery deformation degrees corresponding to all adjacent stitch categories of each stitch category is recorded as the adjacent deformation degree of each stitch category; the difference between the local embroidery deformation degree and the adjacent deformation degree of each stitch category is recorded as the deformation difference of each stitch category; Using the Otsu threshold segmentation algorithm, the deformation differences of all stitch categories are divided into two parts, and the convex hull area of ​​the stitch category corresponding to the part with the smallest average deformation difference is recorded as the tension deformation area; The convex hull area where the stitch category corresponding to the part with the largest average deformation difference is located is recorded as the non-tension deformation area.

7. The embroidery data acquisition device using an industrial area array camera according to claim 1, characterized in that: The specific steps of obtaining the estimated error are as follows: The number of all tension-deformation regions on the same independent trajectory is recorded as the tension-deformation distribution characteristic of each independent trajectory; The Otsu threshold segmentation algorithm is used to divide the tension deformation distribution characteristics of all independent trajectories into two parts. The part with the smallest average value of the tension deformation distribution characteristics is recorded as the first part. The number of independent trajectories in the first part is recorded as H. The ratio of H to the average value of the tension deformation distribution characteristics in the first part is obtained, and the ratio is used as the estimation error of the stitch trajectory.

8. The embroidery data acquisition device using an industrial area array camera according to claim 3, characterized in that: The method of re-obtaining the stitch track on the second image according to the distribution of the tension deformation area along the stitch track includes the following specific steps: For the several corner point pairs obtained by corner point matching of the stitch top view of the first image and the second image, when one corner point pair is retained, the stitch trajectory is re-matched to the second image, and the estimated error of the stitch trajectory is regained, which is recorded as the estimated error of each corner point pair. The N1 corner point pairs with the smallest estimated errors are retained, and the stitch trajectory is re-matched to the second image using the N1 corner point pairs to obtain the stitch trajectory with the smallest estimated error; N1 is a preset integer.

9. The embroidery data acquisition device using an industrial area array camera according to claim 1, characterized in that: The stitch point indicates the position where the embroidery needle contacts the cloth (6) in the first image.

Images