Image processing apparatus, sewing machine, image processing method, and sewing correction method

Abstract

An image processing device (100) is provided with: an object information acquisition unit (102) that acquires an object image (SM) indicating an image related to a sewing object (S) after sewing processing, and correction point beforehand information before sewing processing of a correction point (CP) for correcting a target stitch line (RL) specified in a stitch region (SA), which is a region in which a stitch (CH) is formed; a feature point calculation unit (104) that calculates a feature point after the sewing process on the basis of the object image (SM) after the sewing process and position data before the sewing process when the correction point beforehand information includes position data of the feature point; and an interpolation point calculation unit (107) that, if the correction point beforehand information includes position data and sewing direction data of an interpolation point (IP), calculates the interpolation point (IP) after the sewing process on the basis of the object image after the sewing process, the position data before the sewing process, and the sewing direction data.

Description

TECHNICAL FIELD

[0001] The present application relates to an image processing apparatus, a sewing machine, an image processing method, and a sewing correction method. BACKGROUND

[0002] Sometimes, a stitch is formed in a sewing target object in order to improve the appearance design property of the sewing target object. A technique of forming a stitch in a skin material used for a vehicle seat is disclosed in Patent Literature 1. In addition, if a stitch is formed in a sewing target object that has a thickness and elasticity like a skin material used for a vehicle seat, it is possible that the sewing target object shrinks and the surface of the sewing target object is displaced. In correspondence therewith, a technique of appropriately identifying the displacement of the surface of the sewing target object and forming a stitch based on a target stitch line is disclosed in Patent Literature 2.

[0003] Patent Literature 1: Japanese Patent Application Publication No. 2013-162957

[0004] Patent Literature 2: Japanese Patent Application Publication No. 2021-074074 SUMMARY

[0005] However, in Patent Literature 2, the positional deviation is detected with a perforation pattern as a reference, and thus the pattern serving as a reference needs to have a unique shape feature two-dimensionally. Therefore, for example, there is a problem that it is difficult to apply the stitch formation technique disclosed in Patent Literature 2 to a simple design pattern such as a band curve.

[0006] An object of the present application is to appropriately identify the displacement of the surface of a sewing target object and form a stitch based on a target stitch line.

[0007] According to the present application, there is provided an image processing apparatus including: a target object information acquisition unit that acquires target object information including a target object image and correction point pre-information, the target object image representing an image related to a sewing target object after a sewing process, the correction point pre-information representing information of a correction point before the sewing process for correcting a target stitch line defined in a stitch region in which a stitch is formed; a feature point calculation unit that, in a case where the correction point pre-information includes position data of a feature point having a two-dimensional positioning element, calculates the feature point after the sewing process based on the target object image after the sewing process and the position data before the sewing process; and an interpolation point calculation unit that, in a case where the correction point pre-information includes position data of an interpolation point having a one-dimensional positioning element and sewing direction data representing a sewing direction, calculates the interpolation point after the sewing process based on the target object image after the sewing process and the position data and the sewing direction data before the sewing process.

[0008] Effects of the Invention

[0009] According to the present application, the displacement of the surface of the sewing object can be appropriately recognized, and the stitches are formed based on the target stitch line. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 is a perspective view showing a sewing machine according to the present embodiment.

[0011] Figure 2 is a perspective view showing a part of a sewing machine according to the present embodiment.

[0012] Figure 3 is a sectional view showing a part of a sewing object according to the present embodiment.

[0013] Figure 4 is a plan view showing a part of a sewing object according to the present embodiment.

[0014] Figure 5 is a sectional view showing a part of a sewing object according to the present embodiment.

[0015] Figure 6 is a plan view showing a part of a sewing object according to the present embodiment.

[0016] Figure 7 is a functional block diagram showing a sewing machine according to the present embodiment.

[0017] Figure 8 is an explanatory view of a correction point according to the present embodiment.

[0018] Figure 9 is an explanatory view of an interpolation point according to the present embodiment.

[0019] Figure 10 is an explanatory view of a correction method of a target stitch line according to the present embodiment.

[0020] Figure 11 is an explanatory view of a learning process according to the present embodiment.

[0021] Figure 12 is an explanatory view of a reference point according to the present embodiment.

[0022] Figure 13 is an explanatory view of a calculation method of a boundary point according to the present embodiment.

[0023] Figure 14 is an explanatory view of a calculation method of an intermediate point according to the present embodiment.

[0024] Figure 15is a diagram for explaining one example of a correction point calculation method according to the present embodiment.

[0025] Figure 16 is a flowchart showing a sewing method according to the present embodiment.

[0026] Figure 17 is a flowchart showing a correction point calculation method according to the present embodiment.

[0027] Figure 18 is a block diagram showing a computer system according to the present embodiment. DETAILED DESCRIPTION

[0028] Hereinafter, an embodiment according to the present application will be described with reference to the drawings, but the present application is not limited thereto. The structural elements of the embodiment described below can be appropriately combined. In addition, a part of the structural elements can not be used.

[0029] In the present embodiment, a local coordinate system is defined with respect to the sewing machine 1. In the following description, the local coordinate system defined with respect to the sewing machine 1 is appropriately referred to as a sewing machine coordinate system. The sewing machine coordinate system is defined by an XYZ orthogonal coordinate system. In the present embodiment, the positional relationship of each part is described based on the sewing machine coordinate system. The direction parallel to the X axis in the defined plane is set as the X axis direction. The direction parallel to the Y axis orthogonal to the X axis in the defined plane is set as the Y axis direction. The direction parallel to the Z axis orthogonal to the defined plane is set as the Z axis direction. In addition, the direction of rotation or inclination centered on the X axis is set as the θX direction. The direction of rotation or inclination centered on the Y axis is set as the θY direction. The direction of rotation or inclination centered on the Z axis is set as the θZ direction. In addition, in the present embodiment, the plane including the X axis and the Y axis is appropriately referred to as the XY plane. The plane including the X axis and the Z axis is appropriately referred to as the XZ plane. The plane including the Y axis and the Z axis is appropriately referred to as the YZ plane. The XY plane is parallel to the defined plane. The XY plane, the XZ plane, and the YZ plane are orthogonal to each other. In addition, in the present embodiment, the XY plane and the horizontal plane are parallel. The Z axis direction is the up-down direction. The +Z direction is the upward direction. The -Z direction is the downward direction. Furthermore, the XY plane can also be inclined with respect to the horizontal plane.

[0030] [Physical structure of sewing machine]

[0031] Figure 1 is a perspective view showing the sewing machine 1 according to the present embodiment. Figure 2This is a perspective view showing a portion of the sewing machine 1 according to this embodiment. In this embodiment, the sewing machine 1 is an electronic circulating sewing machine. The sewing machine 1 includes a sewing machine body 10, an operating device 20, and a photographing device 30. The operating device 20 is operated by the operator. The photographing device 30 is capable of photographing the object S being sewn.

[0032] The sewing machine body 10 is mounted on the upper surface of the worktable 2. The sewing machine body 10 includes a sewing machine frame 11, a needle bar 12, a needle plate 13, a holding member 15, an actuator 16, an actuator 17, and an actuator 18. The needle bar 12 and the needle plate 13 are respectively supported on the sewing machine frame 11. The holding member 15 is supported on the sewing machine frame 11 via a support member 14. The actuator 16 generates a force to move the needle bar 12. The actuator 17 generates a force to move the holding member 15. The actuator 18 generates a force to move at least a portion of the holding member 15.

[0033] The sewing machine frame 11 has a horizontal arm 11A, a base 11B, a vertical arm 11C, and a head 11D. The horizontal arm 11A extends in the Y-axis direction. The base 11B is positioned below the horizontal arm 11A. The vertical arm 11C is configured to connect the +Y side end of the horizontal arm 11A to the base 11B. The head 11D is positioned on the -Y side of the horizontal arm 11A.

[0034] The needle bar 12 holds the sewing machine needle 3. The needle bar 12 holds the sewing machine needle 3 in a manner that makes the sewing machine needle 3 parallel to the Z-axis. The needle bar 12 is movably supported on the head 11D in the Z-axis direction.

[0035] The needle plate 13 supports the object S being sewn. The needle plate 13 also supports the holding member 15. The needle plate 13 is supported on the base 11B. The needle plate 13 is positioned below the holding member 15.

[0036] The holding member 15 holds the object being sewn, S. The holding member 15 can move within the XY plane, including the sewing position Ps directly below the sewing machine needle 3, while holding the object being sewn, S. The holding member 15 can also move within the XY plane, including the shooting position Pf directly below the shooting device 30, while holding the object being sewn, Pf. While holding the object being sewn, the holding member 15 moves within the XY plane, including the sewing position Ps, based on sewing data, thereby forming a stitch CH on the object being sewn, S. The holding member 15 is supported on the horizontal arm 11A via the support member 14.

[0037] The holding member 15 has a presser foot member 15A and a lower plate 15B opposite to the presser foot member 15A. The presser foot member 15A is a frame-shaped member. The presser foot member 15A is movable in the Z-axis direction. The lower plate 15B is disposed below the presser foot member 15A. The holding member 15 holds the sewing object S by clamping it with the presser foot member 15A and the lower plate 15B.

[0038] If the presser foot component 15A moves in the +Z direction, the presser foot component 15A and the lower plate 15B separate. This allows the operator to place the workpiece S between the presser foot component 15A and the lower plate 15B. If the presser foot component 15A moves in the -Z direction with the workpiece S placed between the presser foot component 15A and the lower plate 15B, the workpiece S is clamped between the presser foot component 15A and the lower plate 15B. Thus, the workpiece S is held by the holding member 15. Furthermore, if the presser foot component 15A moves in the +Z direction, the holding of the workpiece S by the holding member 15 is released. This allows the operator to remove the workpiece S from between the presser foot component 15A and the lower plate 15B.

[0039] Actuator 16 generates power to move needle bar 12 in the Z-axis direction. Actuator 16 includes a pulse motor. Actuator 16 is disposed on horizontal arm 11A.

[0040] A horizontal arm shaft extending in the Y-axis direction is configured inside the horizontal arm 11A. An actuator 16 is connected to the +Y side end of the horizontal arm shaft. The -Y side end of the horizontal arm shaft is connected to the needle bar 12 via a first power transmission mechanism. The first power transmission mechanism is located inside the head 11D. The horizontal arm shaft rotates due to the operation of the actuator 16. The power generated by the actuator 16 is transmitted to the needle bar 12 via the horizontal arm shaft and the first power transmission mechanism. Consequently, the sewing machine needle 3 held by the needle bar 12 reciprocates in the Z-axis direction.

[0041] A timing belt extending in the Z-axis direction is arranged inside the vertical arm 11C. Additionally, a base shaft extending in the Y-axis direction is arranged inside the base 11B. Pulleys are respectively arranged on the horizontal arm shaft and the base shaft. The timing belt is mounted on the pulleys arranged on the horizontal arm shaft and the base shaft. The horizontal arm shaft and the base shaft are connected via a first power transmission mechanism including the timing belt.

[0042] A vessel is disposed inside the base 11B. The vessel houses the spool that has been placed into the spool housing. The horizontal arm shaft and the base shaft rotate independently due to the operation of the actuator 16. The power generated by the actuator 16 is transmitted to the vessel via the horizontal arm shaft, the timing belt, and the base shaft. Thus, the vessel and the needle bar 12 rotate synchronously in the Z-axis reciprocating motion.

[0043] Actuator 17 generates power to move the holding member 15 in the XY plane. Actuator 17 includes a pulse motor. Actuator 17 includes an X-axis motor 17X and a Y-axis motor 17Y. X-axis motor 17X generates power to move the holding member 15 in the X-axis direction. Y-axis motor 17Y generates power to move the holding member 15 in the Y-axis direction. Actuator 17 is disposed inside the base 11B.

[0044] The power generated by the actuator 17 is transmitted to the holding member 15 via the support member 14. Thus, the holding member 15 can move between the sewing machine needle 3 and the needle plate 13 in the X-axis and Y-axis directions, respectively. Through the operation of the actuator 17, the holding member 15 can move within the XY plane, including the sewing position Ps directly below the sewing machine needle 3, while holding the object S to be sewn.

[0045] Actuator 18 generates power to move presser foot member 15A of holding member 15 in the Z-axis direction. Actuator 18 includes a pulse motor. Presser foot member 15A moves in the +Z direction, thereby separating presser foot member 15A from lower plate 15B. Presser foot member 15A moves in the -Z direction, thereby clamping the sewing object S by presser foot member 15A and lower plate 15B.

[0046] like Figure 2 As shown, the sewing machine body 10 has a presser foot member 19 arranged around the sewing machine needle 3. The presser foot member 19 presses down on the workpiece S to be sewn around the sewing machine needle 3. The presser foot member 19 is movably supported on the head 11D in the Z-axis direction. The presser foot member 19 is connected to a horizontal arm shaft arranged inside the horizontal arm 11A via a second power transmission mechanism. The second power transmission mechanism is a different power transmission mechanism from the first power transmission mechanism that transmits power to the needle bar 12. The second power transmission mechanism that transmits power to the presser foot member 19 is arranged inside the head 11D. By operating the actuator 16, the presser foot member 19 moves in the Z-axis direction in conjunction with the needle bar 12. The presser foot member 19 suppresses the lifting of the workpiece S caused by the movement of the sewing machine needle 3.

[0047] The operating device 20 is operated by the operator. The sewing machine 1 operates by operating the operating device 20. In this embodiment, the operating device 20 includes an operating panel 21 and an operating pedal 22. The operating panel 21 is mounted on the upper surface of the worktable 2. The operating pedal 22 is located below the worktable 2. The operator operates the operating pedal 22 using their foot. The sewing machine 1 operates by the operator operating at least one of the operating panel 21 and the operating pedal 22.

[0048] The imaging device 30 captures an image of the sewn object S held by the holding member 15. The imaging device 30 has an optical system and an image sensor that receives light incident through the optical system. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0049] The imaging device 30 is positioned above the needle plate 13 and the holding member 15. The imaging position Pf includes the position of the optical axis AX of the optical system of the imaging device 30. An imaging area FA is defined for the imaging device 30. The imaging area FA includes the field of view of the optical system of the imaging device 30. The imaging area FA includes the imaging position Pf. The imaging device 30 acquires an image of at least a portion of the sewn object S disposed in the imaging area FA. The imaging device 30 captures an image from above of at least a portion of the sewn object S disposed inside the presser foot member 15A.

[0050] The position of the imaging device 30 is fixed. The relative position of the imaging device 30 and the sewing machine frame 11 is fixed. The relative position of the optical axis AX of the optical system of the imaging device 30 and the sewing machine needle 3 in the XY plane is fixed. The relative position data, which represents the relative position between the optical axis AX of the optical system of the imaging device 30 and the sewing machine needle 3 in the XY plane, is known data that can be derived from the design data of the sewing machine 1.

[0051] The position of the image acquired by the imaging device 30 is defined in the camera coordinate system. Using a defined transformation formula or transformation matrix, the position of the image defined in the camera coordinate system is transformed into the position of the image defined in the sewing machine coordinate system.

[0052] Furthermore, in cases where the actual position of the imaging device 30 differs from its position in the design data due to installation errors, the precise relative position between the optical axis AX of the imaging device 30's optical system and the sewing machine needle 3 can be calculated based on the difference between the actual position of the imaging device 30 and the position of the moved sewing machine needle 3 in the XY plane. After the imaging device 30 is installed, the position of the sewing machine needle 3 in the XY plane is measured, and the measured position of the sewing machine needle 3 is moved toward the imaging device 30 by a known amount. The difference between the actual position of the imaging device 30 and the position of the moved sewing machine needle 3 in the XY plane is then calculated.

[0053] The imaging device 30 captures an image of the sewn object S and outputs an object image SM representing the image of the sewn object S to the control device 40. In this embodiment, the object image SM includes at least an image of the sewn object S after sewing. Alternatively, the object image SM may also include an image of the sewn object S before sewing. In this embodiment, the imaging device 30 captures an image of the sewn object S before sewing. Additionally, the imaging device 30 captures an image of the sewn object S after sewing.

[0054] As described below Figure 8 As shown, the imaging device 30 captures images of the imaging area FA, which includes the correction points CP set in the target stitch line RL. The imaging device 30 sequentially captures images of multiple imaging areas FA corresponding to multiple correction points CP set in the target stitch line RL to be sewn, and outputs the object image SM to the control device 40.

[0055] [Object to be sewn]

[0056] Figure 3 This is a cross-sectional view showing a portion of the sewing object S involved in this embodiment. Figure 4 This is a top view showing a portion of the sewing object S involved in this embodiment. Figure 5 This is a cross-sectional view showing a portion of the sewing object S involved in this embodiment. Figure 6 This is a top view showing a portion of the sewing object S involved in this embodiment. Figure 3 and Figure 4 The object S to be sewn is shown before the sewing process. Figure 5 and Figure 6 The sewn object S after sewing is shown. In this embodiment, the sewn object S is the cover material used for a vehicle seat.

[0057] like Figure 3 As shown, the object S to be sewn has a surface material 4, a padding material 5, and a backing material 6. A hole 7 is provided in the surface material 4. The hole 7 can also be provided so that the padding material 5, the backing material 6, and the surface material 4 are all connected together.

[0058] The surface material 4 is the seating surface that comes into contact with the passenger when the passenger is seated in the vehicle seat. The surface material 4 includes at least one of woven fabric, nonwoven fabric, and leather. The padding material 5 is elastic. The padding material 5 includes, for example, polyurethane resin. The backing material 6 includes at least one of woven fabric, nonwoven fabric, and leather.

[0059] like Figure 4As shown, multiple holes 7 are provided on the surface material 4. The surface material 4 includes: a textured region TA, which is a region on the surface where the holes 7 are formed at equal intervals; and a stitch region SA, which is a region on the surface where no holes 7 are formed. The target stitch line RL to be formed on the sewn object S is defined in the stitch region SA. The target stitch line RL is stored in the sewing data storage unit 51 of the storage device 50, which will be described later.

[0060] [Displacement of the surface of the object being sewn]

[0061] The object being sewn, S, has thickness and elasticity. By forming a stitch CH on the object S with thickness and elasticity, thus achieving... Figure 5 As shown, the object S being sewn at the location where the stitch CH is formed is flattened and shrinks in the Z-axis direction. If the object S shrinks, its surface will be stretched and displaced. Figure 6 As shown, if a stitch CH is formed on the sewing object S, the surface of the sewing object S will be displaced in the XY plane relative to the target stitch line RL.

[0062] If the surface of the object being sewn, S, is displaced in the XY plane relative to the target stitch line, RL, and the holding member 15 is moved along the target stitch line, RL, it will be difficult to form the stitch CH at the desired position on the surface of the object being sewn, S.

[0063] In this embodiment, when the surface of the object being sewn, S, is displaced due to the shrinkage caused by the formation of the stitch CH, the position of the target stitch RL is corrected based on the amount of displacement of the surface of the object being sewn, and the holding member 15 moves based on the corrected target stitch RL.

[0064] [Control structure of a sewing machine]

[0065] Figure 7 This is a functional block diagram illustrating the sewing machine 1 according to this embodiment. The sewing machine 1 includes a control device 40, a storage device 50, an input device 60, an output device 70, and an image processing device 100.

[0066] The control device 40 includes a computer system. For example... Figure 7As shown, the control device 40 is connected to actuators 16, 17, and 18, operating device 20, imaging device 30, storage device 50, input device 60, output device 70, and image processing device 100. Actuator 16 moves the sewing machine needle 3 in the Z-axis direction. Actuator 17 moves the holding member 15 in the XY plane. Actuator 18 moves the presser foot member 15A of the holding member 15 in the Z-axis direction. Furthermore, the control device 40 is connected to drive quantity sensors 31 and 32. Drive quantity sensor 31 detects the drive quantity of actuator 16. Drive quantity sensor 32 detects the drive quantity of actuator 17.

[0067] Based on the processing results of the image processing device 100, the control device 40 outputs control commands to control the actuator 17 that moves the holding member 15. The drive quantity sensor 31 includes an encoder that detects the rotation amount of the pulse motor that acts as the actuator 16. The detection data from the drive quantity sensor 31 is output to the control device 40.

[0068] The drive quantity sensor 32 includes: an X-axis sensor 32X, which detects the rotation of the X-axis motor 17X; and a Y-axis sensor 32Y, which detects the rotation of the Y-axis motor 17Y. The X-axis sensor 32X includes an encoder for detecting the rotation of the X-axis motor 17X. The Y-axis sensor 32Y includes an encoder for detecting the rotation of the Y-axis motor 17Y. The detection data from the drive quantity sensor 32 is output to the control device 40.

[0069] The drive quantity sensor 32 functions as a position sensor to detect the position of the holding member 15 in the XY plane. The drive quantity of the actuator 17 and the movement quantity of the holding member 15 are in a one-to-one correspondence.

[0070] X-axis sensor 32X detects the rotation of X-axis motor 17X, thereby detecting the movement of holding member 15 in the X-axis direction from the origin of the sewing machine coordinate system. Y-axis sensor 32Y detects the rotation of Y-axis motor 17Y, thereby detecting the movement of holding member 15 in the Y-axis direction from the origin of the sewing machine coordinate system.

[0071] The control device 40 controls the actuator 16 based on the detection data from the drive quantity sensor 31. The control device 40, for example, determines the operating timing of the actuator 16 based on the detection data from the drive quantity sensor 31.

[0072] The control device 40 controls the actuator 17 based on the detection data of the drive quantity sensor 32. The control device 40 performs feedback control on the actuator 17 based on the detection data of the drive quantity sensor 32 so that the holding member 15 moves to the desired position.

[0073] The control device 40 calculates the position of the holding member 15 in the XY plane based on the detection data from the drive quantity sensor 32. Based on the detection data from the drive quantity sensor 32, the control device 40 detects the amount of movement of the holding member 15 from the origin in the XY plane. The control device 40 then calculates the position of the holding member 15 in the XY plane based on the detected amount of movement.

[0074] The control device 40 also includes an object image acquisition unit 41, a correction point setting unit 42, an object information output unit 43, a correction point information acquisition unit 44, a target line trace correction unit 45, and a needle drop point correction unit 46.

[0075] The object image acquisition unit 41 acquires an object image SM representing an image related to the sewing object S. The object image acquisition unit 41 acquires the object image SM from the imaging device 30.

[0076] In this embodiment, the object image acquisition unit 41 acquires the object image SM captured by the imaging device 30 before the sewing process. Furthermore, if the object image SM captured by the imaging device 30 before the sewing process is the same as the design data of the sewn object S in its initial state stored in the design data storage unit 52, the object image acquisition unit 41 can also acquire it from the design data storage unit 52 of the sewn object S in its initial state as the object image SM before the sewing process.

[0077] In order to correct the target stitch line RL defined in the stitch area SA, the correction point setting unit 42 sets a correction point CP at an arbitrary position. The correction point CP is set for the sewing object S before sewing. If the stitch CH is formed, the surface of the sewing object S is displaced relative to the correction point CP and the target stitch line RL. Therefore, after the stitch CH is formed on one target stitch line RL, the position of the target stitch line RL to be sewn is corrected based on the position of the correction point CP set in the target stitch line RL to be sewn.

[0078] Figure 8 This is an explanatory diagram of the calibration point CP involved in this embodiment. Figure 9 This is an explanatory diagram of the interpolation point IP involved in this embodiment. For example... Figure 8 As shown, a correction point CP is set in the sewing object S. The correction point CP is used in the correction of the target stitch RL. The correction point CP is set to overlap with the target stitch RL of the object being corrected within the stitch region SA. The position of the correction point CP is defined in the sewing machine coordinate system. The correction point CP includes a feature point FP and an interpolation point IP.

[0079] Feature point FP represents the correction point CP, which is located, for example, at the intersection of the target stitch line RL. The stitch area SA has a characteristic shape, and the correction point CP has two-dimensional positioning elements. The characteristic shape of the stitch area SA is, for example, an X-shape. In addition, feature points FP are set at both ends of the target stitch line RL. In the correction point setting unit 42, the position of feature point FP in the sewing object S before sewing processing, along with the information that it is a feature point FP, is set.

[0080] Interpolation point IP represents the following correction point CP, which has no positioning element in the direction along the target stitch line RL, i.e., the sewing direction, and in the intersecting direction (e.g., referring to...) Figure 13 The intersection direction (CD) has positioning elements, that is, it only has one-dimensional positioning elements. The direction along the target line trace RL is the sewing direction, such as a straight line or a gently curving section, etc. (see...) Figure 9 The sewing direction SD). In this embodiment, the positioning element includes a hole pattern composed of multiple holes 7. Specifically, in the intersecting direction that intersects the sewing direction, there are hole patterns on both sides of the stitch area SA where no holes 7 are formed, and the pattern is uneven. Because the pattern is uneven, positioning can be performed in the intersecting direction that intersects the sewing direction. In contrast, in the sewing direction, similar hole patterns are continuous, making positioning difficult. The correction point setting unit 42 sets the position of the interpolation point IP in the sewing object S before sewing processing together with the data of the sewing direction SD for the interpolation point IP and the information that it is the interpolation point IP. The sewing direction SD is the tangential direction relative to the interpolation point IP of the target stitch line RL.

[0081] The correction point CP is set at any position by the operator. The operator can operate the input device 60 to set the correction point CP at any position on the sewing object S. The correction point information acquisition unit 44 sets the correction point CP based on the input data generated by the operation of the input device 60. The correction point CP can be set automatically based on design data. Alternatively, after the operator sets the feature point FP at any position, the interpolation point IP can be set based on the design data.

[0082] The correction points CP, set by the operator or automatically, are stored as correction point data in the correction point data storage unit 53 of the storage device 50, which will be described later. In the sewing process described later, in order to calculate the position of the correction point CP after displacement of the target stitch line RL to be sewn next, which has been displaced due to sewing, the imaging device 30 acquires an image SM of the object, including the correction point CP set by the correction point setting unit 42, in the imaging area FA.

[0083] Before the sewing process, the object image acquisition unit 41 acquires an object image SM that includes the shooting area FA containing the correction points CP set in the correction point setting unit 42. More specifically, the object image acquisition unit 41 sequentially acquires object images SM of multiple shooting areas FA corresponding to multiple correction points CP set in the target stitch line RL where the next sewing process is to be performed. The object image acquisition unit 41 transforms the position of the object image SM defined in the camera coordinate system to the position of the object image SM defined in the sewing machine coordinate system using a predetermined transformation formula or transformation matrix.

[0084] The object information output unit 43 outputs object information, including the object image SM acquired by the object image acquisition unit 41, to the image processing apparatus 100. The object information includes coordinate data of the shooting area FA. The object information also includes prior correction point information, which represents information about correction points CP included in the object image SM. The prior correction point information includes identifiers for identifying each of the multiple correction points CP.

[0085] The prior information for the correction point includes whether the correction point CP is a feature point FP or an interpolation point IP. If the correction point CP is a feature point FP, the prior information includes the location data of the feature point FP. If the correction point CP is an interpolation point IP, the prior information includes the location data of the interpolation point IP and the sewing direction SD.

[0086] Here, the positions of feature point FP and interpolation point IP represent their positions before displacement occurs through the sewing process. Therefore, for example, if the target stitch line RL is set at the center of the width of the stitch area SA, the position of correction point CP in the object image SM output by object information output unit 43 is offset from the center of the width of the stitch area SA.

[0087] Additionally, the correction point data may also include configuration data for the holes 7 in the texture region TA. For example, the correction point data may include data on the relative position of the hole 7A closest to the interpolation point IP with respect to the interpolation point IP, and the hole-to-hole distance DH between the hole 7B closest to the hole 7A and the hole 7A. The position of the hole 7A and the hole-to-hole distance DH may be information set by the operator, or they may be extracted from the object image SM by the image processing device 100 if the object image SM is obtained before the sewing process.

[0088] The correction point information acquisition unit 44 acquires correction point information output from the image processing device 100. The correction point information acquisition unit 44 acquires correction point information corresponding to object information, including the object image SM output by the object information output unit 43. The correction point information includes identifiers for each of the multiple correction points CP and displacement data of the correction points CP after displacement due to the sewing process. The correction point information acquisition unit 44 acquires displacement data of the correction points CP, including all feature points FP and interpolation points IP corresponding to the target stitch line RL to be sewn next.

[0089] The target stitch correction unit 45 corrects the position of the target stitch RL based on the correction point information CP obtained by the correction point information acquisition unit 44. Due to the displacement of the surface of the sewn object S caused by the sewing process, the target stitch RL, defined by the set correction point CP and the sewing data, will be displaced. The image processing device 100, described later, calculates the displacement of the correction point CP. The target stitch correction unit 45 corrects the target stitch RL based on the calculated displacement of the correction point CP.

[0090] Figure 10 This is an explanatory diagram of the correction method for the target line trace RL according to this embodiment. The target line trace correction unit 45, for example, displaces the target line trace RL by a displacement amount equal to the displacement amount of the correction point CP.

[0091] The target line trace correction unit 45 uses two adjacent correction points CP as references, and based on the displacement, deformation, and rotation of the vector connecting the correction points CP, causes the target line trace RL to undergo displacement, deformation, and rotation. For example, as Figure 10 As shown, the pre-sewing vector SV11, connecting correction points CP1 and CP2, changes to the post-sewing vector SV12 due to the displacement of the surface of the object S being sewn. The target line trace RL between correction points CP1 and CP2 deforms by the same amount of deformation as the pre-sewing vector SV11 towards the post-sewing vector SV12. Furthermore, the target line trace RL between correction points CP1 and CP2 rotates by the same amount of rotation as the pre-sewing vector SV11 towards the post-sewing vector SV12.

[0092] Furthermore, the pre-sewing vector SV21 connecting correction points CP2 and CP3 changes to the post-sewing vector SV22 due to the displacement of the surface of the sewn object S. The target stitch RL between correction points CP2 and CP3 deforms with the same displacement and deformation as the pre-sewing vector SV21 to the post-sewing vector SV22. Additionally, the target stitch RL between correction points CP2 and CP3 rotates with the same rotation as the pre-sewing vector SV21 to the post-sewing vector SV22. Similarly, the target stitch RL between correction points CP3 and CP4 also undergoes displacement, deformation, and rotation. Furthermore, the target stitch RL between correction points CP4 and CP5 also undergoes displacement, deformation, and rotation in the same way. Thus, the corrected target stitch RL is obtained.

[0093] The needle drop point correction unit 46 corrects the position where the sewing machine needle 3 descends when the stitch CH is formed on the target stitch line RL. If the target stitch line correction unit 45 corrects the target stitch line RL, the length of the target stitch line RL between the correction points CP will change. Furthermore, the rate of change of the length of the target stitch line RL between different correction points CP may deviate. Therefore, the interval between the positions where the sewing machine needle 3 descends when the stitch CH is formed on the target stitch line RL, i.e., the needle drop points, will deviate.

[0094] The needle point correction unit 46 recalculates the needle point relative to the target line trace RL corrected by the target line trace correction unit 45, by setting the interval of the needle point to be fixed and the number of needle points to vary.

[0095] Storage device 50 includes non-volatile memory and volatile memory. Non-volatile memory is, for example, ROM (Read Only Memory) or other memory. Volatile memory is, for example, RAM (Random Access Memory). Figure 7 As shown, the storage device 50 is connected to the control device 40. The storage device 50 includes a sewing data storage unit 51, a design data storage unit 52, a correction point data storage unit 53, and a program storage unit 54.

[0096] The sewing data storage unit 51 stores the sewing data referenced during the sewing process. The sewing process refers to forming the stitch CH (reference stitch) on the sewing object S. Figure 6 The processing of ) . Sewing data includes the target stitch line RL (refer to) formed on the stitch CH of the sewing object S. Figure 4 and Figure 6 The target stitch line RL defines the target shape of the stitch CH formed on the sewing object S and the target position of the stitch CH in the sewing machine coordinate system.

[0097] For the object S to be sewn, multiple target stitch lines RL for forming multiple stitches CH are defined. Furthermore, the sewing process includes multiple sewing processes corresponding to each of the multiple stitches CH. Figure 4 A portion of the sewing object S shown includes a first target stitch RL1, a second target stitch RL2, a third target stitch RL3, and a fourth target stitch RL4. Figure 6 In the state shown, a line trace CH is formed on the first target line trace RL1.

[0098] The movement conditions of the holding component 15 include the movement trajectory of the holding component 15 as defined in the sewing machine coordinate system. The movement trajectory of the holding component 15 includes the movement trajectory of the holding component 15 in the XY plane. The movement conditions of the holding component 15 are determined based on the target stitch line RL.

[0099] For example, the first sewing process includes forming a first stitch CH1 on the sewing object S based on a first target stitch line RL1. The first sewing process is initially performed after the sewing object S is held in the holding member 15. Following the first sewing process, a second sewing process is performed, which forms a second stitch on the sewing object S based on a second target stitch line RL2. Similarly, the third sewing process, the fourth sewing process, and the sewing process forming stitch CH for the corresponding target stitch line RL are performed sequentially.

[0100] The design data storage unit 52 stores the design data of the sewn object S. The design data of the sewn object S includes the location and extent of the texture area TA and the location and extent of the stitch area SA on the surface of the sewn object S. When the sewn object S is designed using CAD (Computer Aided Design), the design data of the sewn object S includes the CAD data.

[0101] The design data for the sewing object S is the design data for the sewing object S in its initial state. The initial state of the sewing object S refers to its state before the first sewing process. That is, the initial state of the sewing object S refers to the state before the stitch CH is formed.

[0102] The correction point data storage unit 53 stores correction point data, which serves as data for correction points CP, in order to correct the target line trace RL defined in the line trace region SA. The correction point data includes data on the position of feature point FP, the position of interpolation point IP, and data on the sewing direction SD. Additionally, the correction point data may also include configuration data of holes 7 in the texture region TA. For example, the correction point data may include data on the relative position of hole 7A, which is closest to interpolation point IP, relative to interpolation point IP, and the hole distance DH between hole 7B, which is closest to hole 7A, and hole 7A.

[0103] The calibration point data storage unit 53 stores the calibration point CP data input from the input device 60 through the operator's operation. Additionally, the calibration point data storage unit 53 updates and stores the calibration point CP data acquired by the calibration point information acquisition unit 44 of the control device 40. The calibration point CP data acquired by the calibration point information acquisition unit 44 is the data of the calibration point CP after it has moved within the sewing machine coordinate system due to the deformation of the sewing object S caused by the formation of stitch CH on the previous target stitch RL where the calibration point CP is set.

[0104] The program storage unit 54 stores the computer program used to control the sewing machine 1. The computer program is read into the control device 40. The control device 40 controls the sewing machine 1 according to the computer program stored in the program storage unit 54.

[0105] Input device 60 accepts various input operations performed by the operator. Input data is generated by operating input device 60. Examples of input devices 60 include computer keyboards, mice, operation keys, microphones for accepting sound input, and touch panels.

[0106] Output device 70 outputs output data. Examples of output devices 70 include a display device and a printing device. The display device outputs display data as output data. The printing device outputs printing data as output data. Examples of display devices include flat panel displays such as liquid crystal displays (LCDs) or organic electroluminescence displays (OELs). Examples of printing devices include inkjet printers.

[0107] The image processing apparatus 100 includes a computer system. For example... Figure 7As shown, the image processing device 100 is connected to the control device 40. The image processing device 100 processes images related to the sewing object S. The image processing device 100 includes a learning model storage unit 101, an object information acquisition unit 102, a boundary line extraction unit 103, a feature point calculation unit 104, a boundary point calculation unit 105, an intermediate point calculation unit 106, an interpolation point calculation unit 107, and a correction point information output unit 108.

[0108] The learning model storage unit 101 stores a comparison model captured by using the hole pattern on the surface of the sewing object S before sewing as two-dimensional data such as image data, or a learning model generated by machine learning.

[0109] The comparison model can be generated from more than one image without requiring a recognition device. The comparison model can use the image data itself as comparison data, or it can be used beforehand by extracting two or more shape attributes that are features and their representative coordinates from the image data. Examples of the two or more shape attributes that are features include perforations. Representative coordinates of the shape attributes include, for example, the center or centroid.

[0110] In this embodiment, the case of using learning data generated through machine learning will be described. The learning model generated through machine learning includes a recognizer for recognizing texture regions TA and line streaks SA.

[0111] Figure 11 This is an explanatory diagram of the learning process involved in this embodiment. Many learning images SW are collected as learning data. The learning images SW represent the learning data related to the object image SM after the sewing process. Each learning image SW is assigned boundary line data BW representing the boundary line BL between the texture region TA and the stitch region SA.

[0112] A learning model is generated by performing machine learning on the learning image SW, which is assigned as the teacher data, using a specified machine learning algorithm to perform machine learning on the boundary data BW. Examples of machine learning algorithms include at least one decision tree, random forest, and neural network.

[0113] The object information acquisition unit 102 acquires object information, which includes an object image SM output by the object information output unit 43 of the control device 40. The object information is associated with prior correction point information, which indicates information about correction points CP included in the object image SM. The object image SM acquired by the object information acquisition unit 102 is an image obtained after the surface of the sewn object S has been displaced through a sewing process. The prior correction point information acquired by the object information acquisition unit 102 is information about the correction points CP before the surface of the sewn object S has been displaced through the sewing process. Therefore, there is a discrepancy between the position of the correction points CP after the sewing process and the acquired position data of the correction points CP.

[0114] Figure 12 This is an explanatory diagram of the reference point RP involved in this embodiment. Figure 9 and Figure 12 It is the image SM of the object in the same shooting area FA. Figure 9 It is the image SM of the object before sewing. Figure 12 The image SM is the object image after the surface of the sewn object S has been displaced through the sewing process. The reference point RP is a point representing the position of the midpoint HP before the surface of the sewn object S was displaced through the sewing process. The position data of the correction point CP acquired by the object information acquisition unit 102 is processed as the position data of the reference point RP. The image processing device 100 calculates the displacement of the correction point CP and outputs the displacement from the reference point RP to the correction point CP.

[0115] The boundary line extraction unit 103, based on the sewn object image SM obtained by the object information acquisition unit 102 and the learning model stored in the learning model storage unit 101, divides the surface of the sewn object S into a texture region TA and a stitch region SA, and extracts the boundary line BL between the texture region TA and the stitch region SA.

[0116] The boundary line extraction unit 103 does not need to capture all the contour edges of the boundary line BL; it only needs to extract the centroids of the holes 7 on or near the boundary line BL. Using the reference point RP as the center, image data including the texture region TA and the line stitch region SA are used as templates for matching, or a pattern of centroid coordinates (coordinate point sequence data) of the holes 7 is used as a template for matching. Through these matching processes, the positional relationship between the boundary line BL and the reference point RP can be determined, and the centroid sequence of the holes 7 near the boundary line BL can be extracted.

[0117] As described below Figure 13As shown, the boundary line BL is a line that passes through the edge of the texture region TA. When the texture region TA contains a hole 7, the boundary line BL is generated by either connecting the outer edge of the outermost hole 7 in the texture region TA or connecting the centroid of the hole 7. The boundary line extraction unit 103 extracts the boundary line from the edge of the texture region TA containing the hole 7. Figure 8 The image SM of the object, including the feature points FP shown, and containing Figure 8 The boundary line BL is extracted from any of the object images SM, including the interpolation point IP shown.

[0118] When the boundary line extraction unit 103 extracts the boundary line BL from the object image SM including the feature point FP, the feature point calculation unit 104 calculates the position of the feature point FP based on the boundary line BL. For example, as described in Patent Document 2, the feature point calculation unit 104 calculates the feature point FP based on the relative positions of multiple points on the boundary line BL, such as corner points, maxima, minima, inflection points, or the centroid of the hole 7.

[0119] When the boundary line extraction unit 103 extracts the boundary line BL from the object image SM including the interpolation point IP, the boundary point calculation unit 105 calculates the boundary points BP representing the two ends of the width direction of the stitch region SA, which passes through the reference point RP, based on the boundary line BL. The width direction of the stitch region SA is orthogonal to the sewing direction SD. The interpolation point IP is set on the straight or gently curved stitch region SA. Therefore, the boundary point calculation unit 105 can obtain and fix the width direction of the stitch region SA, and calculate the two boundary points BP on the boundary line BL, which are its two ends.

[0120] Figure 13 This is an explanatory diagram of the method for calculating the boundary point BP according to this embodiment. The correction point prior information acquired by the object information acquisition unit 102 is associated with the sewing direction SD. Due to the sewing process, the surface of the sewn object S is displaced approximately in the intersecting direction CD, which is orthogonal to the sewing direction SD. Therefore, the displacement of the correction point CP in the intersecting direction CD is first calculated. The intersecting direction CD is the width direction of the stitch area SA.

[0121] like Figure 13 As shown, the boundary point calculation unit 105 first calculates a straight line passing through the reference point RP and parallel to the intersection direction CD orthogonal to the sewing direction SD, based on the position of the correction point CP (i.e., the position of the reference point RP) and the sewing direction SD obtained by the object information acquisition unit 102. The boundary point calculation unit 105 then calculates the position where this straight line intersects the boundary line BL, and uses this position as the boundary point BP.

[0122] Furthermore, the significance of the boundary point calculation unit 105 calculating the boundary point BP lies in calculating the scale and angle changes of the object image SM. The method is not limited to calculating the boundary point BP; for example, when matching image data or a coordinate point library is used to determine the positional relationship between the boundary line BL and the reference point RP, by adding the scale and angle to the variation parameters, the scale and angle changes of the object image SM can be calculated.

[0123] The midpoint calculation unit 106 calculates the midpoint HP of the line trace area SA, which represents the midpoint in the width direction of the line trace area SA passing through the reference point RP.

[0124] Figure 14 This is an explanatory diagram illustrating the method for calculating the intermediate point HP according to this embodiment. The width direction of the line trace region SA passing through the reference point RP is the intersection direction CD. Furthermore, the two ends of the width direction of the line trace region SA are two boundary points BP calculated by the boundary point calculation unit 105. Therefore, the intermediate point HP represents the midpoint of the two boundary points BP calculated by the boundary point calculation unit 105. The intermediate point calculation unit 106 calculates the intermediate point HP based on the two boundary points BP calculated by the boundary point calculation unit 105. Alternatively, if the positional relationship between the boundary line BL and the reference point RP is calculated through matching processing, the intermediate point calculation unit 106 calculates the intermediate point HP based on the scale and angle changes of the reference point RP and the object image SM.

[0125] Since the target stitch line RL passes through the center of the width direction of the stitch region SA, in this invention, the midpoint HP can be regarded as the interpolation point IP after sewing. That is, the interpolation point calculation unit 107 can calculate the interpolation point IP only by correcting the cross direction CD. For example, when the interpolation points IP are densely set in the curved part of the target stitch line RL, the correction of the target stitch line RL can be fully performed by using the feature point FP and the midpoint HP. However, in order to achieve more precise correction, the correction of the sewing direction SD is also performed.

[0126] The interpolation point calculation unit 107 calculates the interpolation point IP after the sewing process. The interpolation point calculation unit 107 calculates the displacement in the sewing direction SD caused by the sewing process, based on the intermediate point HP calculated by the intermediate point calculation unit 106. The interpolation point calculation unit 107 calculates the displacement in the sewing direction SD based on the displacement of the perforation pattern, i.e., the hole 7, in the texture region TA.

[0127] Figure 15 This is an explanatory diagram illustrating an example of the interpolation point IP calculation method according to this embodiment. The interpolation point calculation unit 107 first calculates the interpolation point IP based on hole 7A (refer to...). Figure 9The position of the reference hole 7R relative to the interpolation point IP obtained by the object information acquisition unit 102 is calculated. The relative position of the reference hole 7R relative to the intermediate point HP is the same as the relative position of the hole 7A relative to the correction point CP.

[0128] The interpolation point calculation unit 107 then searches for hole 7A in the intersecting direction CD for reference hole 7R. The method for searching hole 7A can be a known method such as pattern matching. Alternatively, the search range can be based on the inter-hole distance DH between the nearest hole 7B and hole 7A (see reference). Figure 9 The settings are as follows. Specifically, for example, with reference hole 7R as the center, the search range of the cross direction CD is set to 1 / 2 of the distance DH between holes.

[0129] The interpolation point calculation unit 107 calculates the distance up to the hole 7A detected relative to the reference hole 7R. The distance up to the hole 7A detected relative to the reference hole 7R is the displacement of the interpolation point IP in the crossing direction CD. The interpolation point calculation unit 107 calculates the interpolation point IP based on the intermediate point HP and the displacement of the interpolation point IP in the crossing direction CD.

[0130] Furthermore, when the boundary line BL and boundary point BP are calculated by matching image data or coordinate point data, the variation area of ​​the interpolation point IP is predicted based on the intersection direction CD, the distance between holes DH, the maximum scale, and the angle variation, and the search area is set. This can replace the processing performed by the boundary line extraction unit 103, the feature point calculation unit 104, the boundary point calculation unit 105, the intermediate point calculation unit 106, and the interpolation point calculation unit 107, and the interpolation point IP can be calculated by matching only.

[0131] The correction point information output unit 108 outputs correction point information, representing a new correction point CP, to the control device 40. This new correction point CP includes a feature point FP calculated by the feature point calculation unit 104 and an interpolation point IP calculated by the interpolation point calculation unit 107. The correction point information includes identifiers for each of the multiple correction points CP. The correction point information also includes displacement data for the newly calculated correction point CP obtained by the object information acquisition unit 102.

[0132] [Control device processing]

[0133] Figure 16This is a flowchart illustrating the sewing method according to this embodiment. In this embodiment, the sewing method includes alignment processing S10, object image acquisition processing S11, object information output processing S12, correction point information acquisition processing S13, target stitch correction processing S14, needle drop point correction processing S15, sewing processing S16, and end determination processing S17. The control device 40 executes this process. Figure 16 The processing is shown.

[0134] The alignment process S10 involves associating the texture region TA and stitch region SA of the sewing object S held by the holding member 15 with the sewing machine coordinate system. After holding the sewing object S in the holding member 15 before the sewing process, the imaging device 30 captures an image of the sewing object S. The imaging device 30 captures, for example, the correction point CP, which includes multiple feature points FP and interpolation points IP of the sewing object S. Furthermore, if alignment marks are provided in the sewing object S, the imaging device 30 captures the alignment marks. The position of the image of the sewing object S captured by the imaging device 30 is defined in the camera coordinate system. Using a defined transformation formula or transformation matrix, the position of the image defined in the camera coordinate system is transformed into the position of the image defined in the sewing machine coordinate system. Thus, the position of the texture region TA and the position of the stitch region SA of the sewing object S are defined in the sewing machine coordinate system.

[0135] The object image acquisition process S11 is the process of acquiring the object image SM. The object image acquisition unit 41 can acquire the object image SM in its initial state from the imaging device 30 before the first sewing process, or it can acquire the object image SM in its initial state from the design data storage unit 52.

[0136] After the first sewing process, an object image SM is acquired by the imaging device 30. The imaging device 30 acquires the object image SM after the first sewing process and before the second sewing process. Similarly, the imaging device 30 acquires the object image SM between each of the second sewing process and the final sewing process.

[0137] The object information output processing S12 is a process in which object information, including the object image SM obtained by the object image acquisition processing S11, is output to the image processing device 100. The object information output unit 43 outputs object information, including the object image SM obtained by the object image acquisition unit 41, to the image processing device 100. In the object information output processing S12, the object images SM corresponding to one target line trace RL can be summarized and output, or the object images SM can be output one by one.

[0138] The correction point information acquisition process S13 is a process of acquiring correction point information output from the image processing device 100. The correction point information acquisition unit 44 acquires correction point information corresponding to object information, including the object image SM output by the object information output unit 43. Furthermore, in the object information output process S12, after outputting object information corresponding to a target line trace RL, the control device 40 enters a waiting state for the correction point information acquisition process S13 until the correction point information corresponding to the target line trace RL is acquired.

[0139] The target stitch correction process S14 is a process that corrects the target stitch RL based on the correction point CP obtained through the correction point information acquisition process S13. Due to the displacement of the surface of the sewn object S caused by the sewing process, the target stitch RL, defined by the correction point CP set by the operator and the sewing data, is displaced. The displacement amount of the correction point CP is calculated by the image processing device 100 and obtained by the correction point information acquisition process S13. The target stitch correction unit 45 corrects the target stitch RL based on the displacement amount of the correction point CP obtained from the image processing device 100. For example, the target stitch correction unit 45 displaces the target stitch RL by the same amount as the displacement amount of the correction point CP.

[0140] Multiple correction points CP are set for the target stitch line RL. The target stitch line RL is corrected based on the displacement of each correction point CP. Using two adjacent correction points CP as a reference, the target stitch line RL is displaced, deformed, and rotated based on the displacement, deformation, and rotation of the vector connecting the correction points CP, thereby correcting the target stitch line RL. The position of the target stitch line RL in the sewing machine coordinate system is corrected.

[0141] The target stitch correction process S14 is not performed before the first sewing process. The first sewing process is performed based on the target stitch RL in a predetermined initial state obtained through sewing data. The target stitch correction process S14 is performed after the first sewing process and before the second sewing process. Similarly, the target stitch correction process S14 is performed between each of the second and final sewing processes.

[0142] The needle drop point correction process S15 is a process that corrects the position where the sewing machine needle 3 descends when the stitch CH is formed on the target stitch line RL. The target stitch line correction process S14 corrects each target stitch line RL between adjacent correction points CP. As a result, the interval between the needle drop points of the target stitch lines RL set between different correction points CP deviates. The needle drop point correction unit 46 recalculates the needle drop points for one target stitch line RL corrected by the target stitch line correction process S14, by setting the interval between needle drop points to a fixed value and varying the number of needle drop points.

[0143] Sewing process S16 is the process of forming stitch CH based on target stitch line RL. The first sewing process is performed based on the target stitch line RL in a predetermined initial state using sewing data. The second and final sewing processes are performed based on the target stitch line RL corrected by target stitch line correction process S14 and the needle drop point corrected by needle drop point correction process S15. Control device 40 outputs control commands to actuator 17 to form stitch CH according to the target stitch line RL.

[0144] The end determination process S17 is a process that determines whether the sewing process of the sewing object S has ended. The control device 40 determines whether the sewing process of the sewing object S has ended based on the sewing data. If the sewing process from the first sewing process to the last sewing process has ended, the control device 40 determines in the end determination process S17 that the sewing process has not ended. If the control device 40 determines that the sewing process has not ended, it jumps to the object image acquisition process S11. If the last sewing process has ended, the control device 40 determines in the end determination process S17 that the sewing process has ended.

[0145] [Processing by the image processing device]

[0146] Figure 17 This is a flowchart illustrating the correction point calculation method according to this embodiment. In this embodiment, the correction point calculation method includes object information acquisition processing S21, correction point determination processing S22, boundary line extraction processing S23, feature point calculation processing S24, boundary line extraction processing S25, interpolation point calculation processing S26, and correction point information output processing S27. The image processing apparatus 100 executes this process. Figure 17 The processing is shown.

[0147] The object information acquisition process S21 is the following process: the control device 40 acquires information containing... Figure 16 The object information output in the object information output processing S12 includes the object image SM. The image processing apparatus 100 begins processing by acquiring the object information. Figure 17The series of processes shown.

[0148] The correction point determination process S22 is as follows: it determines whether the correction point CP contained in the object image SM obtained in the object information acquisition process S21 is a feature point FP or an interpolation point IP. The object information contains information about whether the correction point CP is a feature point FP or an interpolation point IP. If object information, including the object image SM containing the feature point FP, is obtained, the control device 40 determines in the correction point determination process S22 that it is a feature point FP and proceeds to the boundary line extraction process S23. If object information, including the object image SM containing the interpolation point IP, is obtained, the control device 40 determines in the correction point determination process S22 that it is an interpolation point IP and proceeds to the boundary line extraction process S25.

[0149] Boundary line extraction process S23 is as follows: based on the object image SM containing feature points FP and the learning model, the surface of the sewn object S is segmented into a texture region TA and a stitch region SA, and the boundary line BL between the texture region TA and the stitch region SA is calculated. Boundary line extraction process S23 is performed sequentially on the object image SM containing feature points FP obtained in the object information acquisition process S21.

[0150] In the boundary line extraction process S23, the boundary line extraction unit 103 obtains the learning model from the learning model storage unit 101. Based on the object image SM containing feature points FP obtained in the object information acquisition process S21 and the obtained learning model, the boundary line extraction unit 103 divides the surface of the sewn object S into a texture region TA and a stitch region SA, and calculates the boundary line BL between the texture region TA and the stitch region SA.

[0151] Feature point calculation processing S24 is a process in which the position of feature point FP after sewing is calculated based on feature point FP set in the sewing object S before sewing and boundary line BL extracted in boundary line extraction processing S23. Feature point calculation processing S24 is performed, for example, in the order referred to in correction point calculation processing S3 described in Patent Document 2.

[0152] Boundary line extraction process S25 is as follows: based on the object image SM containing interpolation points IP and the learning model, the surface of the sewn object S is segmented into a texture region TA and a stitch region SA, and the boundary lines BL of the texture region TA and the stitch region SA are calculated. Boundary line extraction process S25 is performed sequentially on the object image SM containing interpolation points IP obtained in the object information acquisition process S21.

[0153] In the boundary line extraction process S25, the boundary line extraction unit 103 obtains the learning model from the learning model storage unit 101. Based on the object image SM including interpolation points IP obtained in the object information acquisition process S21 and the obtained learning model, the boundary line extraction unit 103 divides the surface of the sewn object S into a texture region TA and a stitch region SA, and calculates the boundary line BL between the texture region TA and the stitch region SA.

[0154] The interpolation point calculation process S26 is as follows: based on the interpolation point IP set in the sewing object S before sewing and the boundary line BL extracted in the boundary line extraction process S25, the position of the interpolation point IP after sewing is calculated. In the interpolation point calculation process S26 of this embodiment, the two boundary points BP and the center between them, i.e., the midpoint HP, are calculated based on the boundary line BL, and the interpolation point IP is calculated based on the deviation of the hole 7 near the interpolation point IP and the midpoint HP.

[0155] In the interpolation point calculation process S26, the boundary point calculation unit 105 calculates the intersection direction CD, which is orthogonal to the sewing direction SD and passes through the reference point RP. The intersection direction CD, which passes through the reference point RP, is the width direction of the stitch area SA. Based on the reference point RP and the intersection direction CD, the boundary point calculation unit 105 calculates two boundary points BP, which represent the two ends of the width direction of the stitch area SA.

[0156] The midpoint calculation unit 106 calculates the midpoint HP, which represents the center of the stitch area SA passing through the reference point RP, based on two boundary points BP. The midpoint HP calculated by the interpolation point calculation process S26 can be regarded as the interpolation point IP after the sewing process. That is, the interpolation point calculation unit 107 can calculate the interpolation point IP only by correcting the cross direction CD.

[0157] In the interpolation point calculation process S26 of this embodiment, the deviation of the hole 7 near the intermediate point HP, the interpolation point IP, and the interpolation point IP are calculated. That is, the correction of the sewing direction SD is also performed. The interpolation point calculation unit 107 calculates the position of the reference hole 7R relative to the intermediate point HP based on the relative position of the hole 7A with respect to the interpolation point IP obtained in the object information acquisition process S21. The interpolation point calculation unit 107 searches for the hole 7A in the cross direction CD for the reference hole 7R and calculates the distance up to the hole 7A detected for the reference hole 7R. The interpolation point calculation unit 107 calculates the interpolation point IP based on the intermediate point HP and the distance up to the hole 7A detected for the reference hole 7R, which represents the displacement of the interpolation point IP in the cross direction CD.

[0158] The correction point information output processing S27 involves outputting correction point information representing a new correction point CP to the control device 40. This new correction point CP includes the feature point FP calculated by the feature point calculation processing S24 and the interpolation point IP calculated by the interpolation point calculation processing S26. In the correction point information output processing S27, correction point information corresponding to the object information obtained by the object information acquisition processing S21 is output. The correction point information includes displacement data of the newly calculated correction point CP relative to the prior correction point information of the correction point CP obtained by the object information acquisition processing S21.

[0159] [Computer System]

[0160] Figure 18 This is a block diagram illustrating an example of a computer system 1000. The control device 40 and the image processing device 100 each include a computer system 1000. The computer system 1000 has a processor 1001, such as a CPU (Central Processing Unit), a main memory 1002 including non-volatile memory and volatile memory, a storage device 1003, and an interface 1004 including input / output circuitry. Non-volatile memory is, for example, ROM (Read Only Memory). Volatile memory is, for example, RAM (Random Access Memory). The functions of the control device 40 and the image processing device 100 are each stored as computer programs in the storage device 1003. The processor 1001 reads the computer program from the storage device 1003, expands it to the main memory 1002, and executes the aforementioned processing according to the computer program. Furthermore, the computer program can also be transmitted to the computer system 1000 via a network.

[0161] The computer program includes various application software for implementing the above functions. According to the above implementation, the computer program can perform the following steps: based on an object image SM representing an image involving a sewn object S, the surface of the sewn object S is divided into a texture region TA and a stitch region SA; the boundary line BL between the texture region TA and the stitch region SA is extracted; object information is obtained, which includes the object image SM after sewing and prior correction point information representing prior correction point CP used to correct the target stitch line RL defined in the stitch region SA; if the prior correction point information includes position data of feature points FP with two-dimensional positioning elements, the feature points FP after sewing are calculated based on the boundary line BL after sewing and the position data before sewing; and if the prior correction point information includes position data of interpolation points IP with one-dimensional positioning elements and sewing direction data representing the sewing direction SD, the interpolation points IP after sewing are calculated based on the boundary line BL after sewing, the position data before sewing, and the sewing direction data.

[0162] [Effect]

[0163] As described above, according to this embodiment, based on the object image SM of the sewing object S, the boundary line BL between the textured region TA with holes 7 and the stitch region SA without holes 7 is calculated on the surface of the sewing object S. Prior correction point information is obtained, which represents information prior to sewing processing for the correction point CP used to correct the target stitch line RL defined in the stitch region SA. The correction point CP includes a feature point FP and an interpolation point IP. The feature point FP has two-dimensional positioning features, while the interpolation point IP does not have two-dimensional positioning features but has one-dimensional positioning features.

[0164] When the correction point CP is a feature point FP, the prior information for the correction point includes the position data of the feature point FP. The feature point FP after sewing is calculated based on the boundary line BL after sewing and the position data of the feature point FP before sewing. When the correction point CP is an interpolation point IP, the prior information for the correction point includes the position data of the interpolation point IP and the sewing direction data representing the sewing direction SD. The interpolation point IP after sewing is calculated based on the boundary line BL after sewing, the position data of the interpolation point IP, and the sewing direction data.

[0165] In other words, the feature point FP with two-dimensional positioning elements can be estimated in two dimensions by the characteristic boundary line BL after sewing. Therefore, the displacement of the feature point FP can be calculated based on the position data of the feature point FP before sewing and the estimated position of the feature point FP after sewing.

[0166] Furthermore, regarding the interpolation point IP, which has only one-dimensional positioning features, the displacement of the interpolation point IP in the intersecting direction CD is calculated by taking into account the displacement of the surface of the sewn object S approximately orthogonal to the sewing direction SD due to the sewing process. The one-dimensional characteristic boundary line BL is a characteristic boundary line in the intersecting direction CD. Therefore, at the position of the interpolation point IP relative to the position before the sewing process in the intersecting direction CD, the position of the interpolation point IP after the sewing process can be estimated by the boundary line BL. Thus, the displacement of the interpolation point IP can be calculated based on the position data of the interpolation point IP before the sewing process and the estimated position of the interpolation point IP after the sewing process.

[0167] Therefore, the displacement of the correction point CP, including the feature point FP and the interpolation point IP, from before to after the sewing process, is calculated. By calculating the displacement of the correction point CP, the displacement of the surface of the sewn object S caused by the sewing process is appropriately identified. By appropriately identifying the displacement of the surface of the sewn object S, even if the surface of the sewn object S is displaced, the position of the target stitch line RL can be appropriately corrected. Therefore, the control device 40 can form the stitch CH based on the target stitch line RL.

[0168] [Other Implementation Methods]

[0169] The embodiments of this application have been described above, but the present invention is not limited to these embodiments. The above embodiments and variations can be appropriately combined within a scope that does not contradict the intended meaning. Furthermore, the aforementioned structural elements include elements of substantially the same, so-called equal scope, which can be easily conceived by those skilled in the art. Moreover, the aforementioned structural elements can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of structural elements can be made without departing from the spirit of the aforementioned embodiments.

[0170] For example, in the above embodiment, when the image SM of the object is acquired by the imaging device 30, the holding member 15 that holds the sewing object S moves in the XY plane while the position of the imaging device 30 is fixed. While the position of the sewing object S is fixed, the imaging area FA of the imaging device 30 may move in the XY plane, or both the imaging area FA and the sewing object S may move in the XY plane.

[0171] Furthermore, in the above embodiment, the boundary line BL is generated such that it passes through the outer edge of the outermost hole 7 disposed in the texture region TA. The boundary line BL can also be generated such that it passes through the outermost hole 7 disposed in the texture region TA.

[0172] Furthermore, all or part of the processes described in the above embodiments that are set to be performed automatically can also be performed manually. Alternatively, all or part of the processes described in the above embodiments that are set to be performed manually can be performed automatically using known methods. In addition, the processing order, specific names, and information including various data and parameters shown in the above documents and figures can be arbitrarily changed, unless specifically described otherwise. For example, the various information shown in the figures is not limited to the information shown in the figures.

[0173] Furthermore, the structural elements of each device illustrated are functional conceptual structural elements and do not necessarily need to be constructed as shown in the physical diagram. That is, the specific method of distributing and merging the devices is not limited to the illustrated method, and all or part of them can be distributed and merged functionally or physically in any unit according to various loads, usage conditions, etc. For example, the boundary point calculation unit 105 and the intermediate point calculation unit 106 may also be included in the interpolation point calculation unit 107. In addition, the learning model storage unit 101 may have two learning models obtained by machine learning on a two-dimensional geospatial feature learning image SW with set feature points FP and a one-dimensional geospatial feature learning image SW with set interpolation points IP. In addition, the control device 40 and the image processing device 100 described above may also be constructed by multiple computers divided into several functions. In addition, a cloud server that performs various functions through cloud computing may also have several functions of a computer.

[0174] Alternatively, the image processing device 100 can be configured as a computer completely independent of the sewing machine 1. The image processing device 100 is connected to the control devices 40 of multiple sewing machines 1, and performs processing to calculate correction points CP based on object images SM output from each control device 40. Furthermore, the image processing device 100 can also be connected to a communication network including short-range wireless communication such as Bluetooth (registered trademark) or wireless LAN (Local Area Network), and connected to the sewing machine 1 via the communication network.

[0175] The present invention includes the following methods. (1)

[0177] An image processing apparatus comprising: The object information acquisition unit acquires object information, which includes an object image and prior correction point information. The object image represents an image of the sewn object after sewing, and the prior correction point information represents prior sewing information for correcting the target stitch line defined in the area where the stitch is formed, i.e., the stitch area. The feature point calculation unit, when the prior information for the correction point includes position data of feature points with two-dimensional positioning elements, calculates the feature points after the sewing process based on the object image after the sewing process and the position data before the sewing process; and The interpolation point calculation unit calculates the interpolation point after sewing processing based on the object image after sewing processing, the position data of the interpolation point with one-dimensional positioning elements and the sewing direction data indicating the sewing direction, when the prior information of the correction point includes position data of the interpolation point with one-dimensional positioning elements and sewing direction data indicating the sewing direction. (2)

[0179] In the image processing apparatus described in (1), It also includes a boundary line extraction unit that, based on the object image, divides the surface of the sewn object into texture regions and stitch regions, and extracts the boundary lines between the texture regions and the stitch regions. The feature point calculation unit calculates the feature points after sewing based on the boundary line extracted from the object image after sewing and the position data before sewing. The interpolation point calculation unit calculates the interpolation point after sewing based on the boundary line extracted from the object image after sewing, the position data before sewing, and the sewing direction data. (3)

[0181] In the image processing apparatus described in (2), The interpolation point calculation unit calculates two boundary points, which represent the points where a straight line passing through the interpolation point and parallel to the intersection direction orthogonal to the sewing direction intersects the boundary line. Calculate the midpoint representing the midpoint of the two boundary lines. Based on the interpolation point and the intermediate point before the sewing process, the displacement of the interpolation point in the intersection direction is calculated. (4)

[0183] In the image processing apparatus described in (2) or (3), The interpolation point calculation unit operates under the condition that the displacement of the interpolation point in the sewing direction is less than 1 / 2 of the distance between the holes of the plurality of holes contained in the texture region. Based on the displacement of the interpolation point and the multiple holes contained in the texture area before the sewing process, the displacement of the interpolation point in the sewing direction is calculated. (5)

[0185] A sewing machine having: A retaining component that can hold and move the sewing object within a defined surface, including the sewing position directly below the sewing machine needle; An actuator that generates power to move the retaining member; and The control device outputs a control command to control the actuator based on the processing result of the image processing device described in any one of (1) to (4). (6)

[0187] In the sewing machines described in (5), The control device includes: A correction point information acquisition unit acquires correction point information from the image processing device. This correction point information represents information about multiple correction points, including the displacement of the feature points and the interpolation points after the sewing process. The target line trace correction unit corrects the target line trace based on the correction point information. (7)

[0189] In the sewing machines described in (5) or (6), The control device further includes a needle point correction unit, which recalculates the needle points for the corrected target stitch by setting the interval of the needle points to a fixed value and varying the number of needle points. (8)

[0191] An image processing method comprising the following steps: Based on the object image representing the image involved in the sewing object, the surface of the sewing object is divided into a texture region and a stitch region, and the boundary line between the texture region and the stitch region is extracted; The prior information for correction points refers to information prior to sewing used to correct the target stitch line defined in the stitch area. When this prior information includes position data of feature points with two-dimensional positioning elements, the feature points after sewing are calculated based on the boundary line after sewing and the position data before sewing; and When the prior information of the correction point includes position data of the interpolation point with one-dimensional positioning elements and sewing direction data indicating the sewing direction, the interpolation point after the sewing process is calculated based on the boundary line after the sewing process, the position data before the sewing process, and the sewing direction data. (9)

[0193] A sewing correction method further includes the following steps: The correction point information represents information on multiple correction points, including the displacement of the feature points and the interpolation points calculated in the image processing method described in (8), and the target line trace is corrected based on the correction point information.

[0194] This application is based on Japanese Patent Application No. 2023-110448, filed on July 5, 2023, the contents of which are incorporated herein by reference.

Claims

1. An image processing apparatus comprising: The object information acquisition unit acquires object information, which includes an object image and prior correction point information. The object image represents an image of the sewn object after sewing, and the prior correction point information represents prior sewing information for correcting the target stitch line defined in the area where the stitch is formed, i.e., the stitch area. The feature point calculation unit calculates the feature points after sewing based on the object image after sewing and the position data before sewing, when the prior information of the correction point includes position data of feature points with two-dimensional positioning elements. as well as The interpolation point calculation unit calculates the interpolation point after sewing processing based on the object image after sewing processing, the position data of the interpolation point with one-dimensional positioning elements and the sewing direction data indicating the sewing direction, when the prior information of the correction point includes position data of the interpolation point with one-dimensional positioning elements and sewing direction data indicating the sewing direction.

2. The image processing apparatus according to claim 1, wherein, It also includes a boundary line extraction unit that, based on the object image, divides the surface of the sewn object into texture regions and stitch regions, and extracts the boundary lines between the texture regions and the stitch regions. The feature point calculation unit calculates the feature points after sewing based on the boundary line extracted from the object image after sewing and the position data before sewing. The interpolation point calculation unit calculates the interpolation point after sewing based on the boundary line extracted from the object image after sewing, the position data before sewing, and the sewing direction data.

3. The image processing apparatus according to claim 2, wherein, The interpolation point calculation unit calculates two boundary points, which represent the points where a straight line passing through the interpolation point and parallel to the intersection direction orthogonal to the sewing direction intersects the boundary line. Calculate the midpoint representing the midpoint of the two boundary lines. Based on the interpolation point and the intermediate point before the sewing process, the displacement of the interpolation point in the intersection direction is calculated.

4. The image processing apparatus according to claim 2, wherein, The interpolation point calculation unit operates under the condition that the displacement of the interpolation point in the sewing direction is less than 1 / 2 of the distance between the holes of the plurality of holes contained in the texture region. Based on the displacement of the interpolation point and the multiple holes contained in the texture area before the sewing process, the displacement of the interpolation point in the sewing direction is calculated.

5. A sewing machine, comprising: A retaining component that can hold and move the sewing object within a defined surface, including the sewing position directly below the sewing machine needle; An actuator that generates power to move the retaining member; and A control device that outputs control commands to control the actuator based on the processing results of the image processing device described in any one of claims 1 to 4.

6. The sewing machine according to claim 5, wherein, The control device includes: The correction point information acquisition unit acquires correction point information from the image processing device. The correction point information represents information on multiple correction points, including the displacement of the feature points and the interpolation points after the sewing process. as well as The target line trace correction unit corrects the target line trace based on the correction point information.

7. The sewing machine according to claim 6, wherein, The control device further includes a needle point correction unit, which recalculates the needle points for the corrected target stitch by setting the interval of the needle points to a fixed value and varying the number of needle points.

8. An image processing method comprising the following steps: Based on the object image representing the image involved in the sewing object, the surface of the sewing object is divided into a texture region and a stitch region, and the boundary line between the texture region and the stitch region is extracted; The prior information for correction points refers to the information before sewing of correction points used to correct the target stitch line defined in the stitch area. When the prior information for correction points includes the position data of feature points with two-dimensional positioning elements, the feature points after sewing are calculated based on the boundary line after sewing and the position data before sewing. as well as When the prior information of the correction point includes position data of the interpolation point with one-dimensional positioning elements and sewing direction data indicating the sewing direction, the interpolation point after the sewing process is calculated based on the boundary line after the sewing process, the position data before the sewing process, and the sewing direction data.

9. A sewing correction method, further comprising the following steps: The correction point information represents information on multiple correction points, including the displacement of the feature points and the interpolation points calculated in the image processing method of claim 8 after the sewing process, and the target line trace is corrected based on the correction point information.

Images