Image processing device, sewing machine, image processing method, and sewing correction method

Abstract

An image processing device (100) includes an object information acquisition unit (102) that acquires an object image (SM) representing an image relating to an object to be sewn (S) after sewing processing, and correction point prior information on a correction point (CP) before the sewing processing, the correction point (CP) being for correcting a target stitch line (RL) defined in a stitch area (SA) that is an area in which a stitch (CH) is formed; a feature point calculation unit (104) that, when the correction point prior information includes position data of a feature point, calculates the feature point after the sewing processing on the basis of the object image (SM) after the sewing processing and the position data before the sewing processing; and an interpolation point calculation unit (107) that, when the correction point prior information includes position data and sewing direction data of an interpolation point (IP), calculates the interpolation point (IP) after the sewing processing on the basis of the object image after the sewing processing, and the position data and the sewing direction data before the sewing processing.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an image processing device, a sewing machine, an image processing method, and a sewing correction method.BACKGROUND ART

[0002] In order to enhance the design property of an object to be sewn, stitches may be formed on the object to be sewn. Patent Literature 1 discloses a technique for forming stitches on a skin material used for a vehicle seat. However, when stitches are formed on a thick and elastic object to be sewn, such as a skin material used for a vehicle seat, the object to be sewn may shrink, thereby causing the surface of the object to be sewn to be displaced. In response to this, Patent Literature 2 discloses a technique for appropriately detecting the displacement of the surface of the object to be sewn and forming stitches on the basis of a target stitch line.CITATION LISTPATENT LITERATURE

[0003] Patent Literature 1: JP2013-162957A Patent Literature 2: JP2021-074074A SUMMARY OF INVENTIONTECHNICAL PROBLEM

[0004] However, in Patent Literature 2, positional deviation is detected based on a perforation pattern, and therefore a reference pattern needs to have a two-dimensionally unique shape feature. For this reason, there has been a problem that it is difficult to apply the stitch formation technique disclosed in Patent Literature 2 to a simple design pattern such as a band-shaped curve.

[0005] An aspect of the present disclosure is to appropriately detect displacement of a surface of an object to be sewn and to form a stitch on the basis of a target stitch line.SOLUTION TO PROBLEM

[0006] According to an aspect of the present disclosure, there is provided an image processing device including: an object information acquisition unit configured to acquire object information including an object image representing an image relating to an object to be sewn after sewing processing, and correction point prior information representing information on a correction point before the sewing processing, the correction point being for correcting a target stitch line defined in a stitch area that is an area in which a stitch is formed; a feature point calculation unit configured, when the correction point prior information includes position data of a feature point having a two-dimensional positioning element, to calculate the feature point after the sewing processing on the basis of the object image after the sewing processing and the position data before the sewing processing; and an interpolation point calculation unit configured, when the correction point prior information includes position data and sewing direction data of an interpolation point having a one-dimensional positioning element and sewing direction data the sewing direction data representing a sewing direction, to calculate the interpolation point after the sewing processing on the basis of the object image after the sewing processing and the position data and the sewing direction data before the sewing processing.ADVANTAGEOUS EFFECTS OF INVENTION

[0007] According to an aspect of the present disclosure, it is possible to appropriately detect displacement of a surface of an object to be sewn and to form a stitch on the basis of a target stitch line.BRIEF DESCRIPTION OF DRAWINGS

[0008] Fig. 1 is a perspective view illustrating a sewing machine according to the present embodiment. Fig. 2 is a perspective view illustrating a part of the sewing machine according to the present embodiment. Fig. 3 is a cross-sectional view illustrating a part of an object to be sewn according to the present embodiment. Fig. 4 is a plan view illustrating a part of the object to be sewn according to the present embodiment. Fig. 5 is a cross-sectional view illustrating a part of the object to be sewn according to the present embodiment. Fig. 6 is a plan view illustrating a part of the object to be sewn according to the present embodiment. Fig. 7 is a functional block diagram illustrating the sewing machine according to the present embodiment. Fig. 8 is an illustrative diagram of correction points according to the present embodiment. Fig. 9 is an illustrative diagram of an interpolation point according to the present embodiment. Fig. 10 is an illustrative diagram of a target stitch line correction method according to the present embodiment. Fig. 11 is an illustrative diagram of learning processing according to the present embodiment. Fig. 12 is an illustrative diagram of a reference point according to the present embodiment. Fig. 13 is an illustrative diagram of a boundary point calculation method according to the present embodiment. Fig. 14 is an illustrative diagram of an intermediate point calculation method according to the present embodiment. Fig. 15 is an illustrative diagram of an example of a correction point calculation method according to the present embodiment. Fig. 16 is a flowchart showing a sewing method according to the present embodiment. Fig. 17 is a flowchart showing a correction point calculation method according to the present embodiment. Fig. 18 is a block diagram illustrating a computer system according to the present embodiment. DESCRIPTION OF EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings, but the present disclosure is not limited thereto. Components of the embodiments described below can be combined appropriately. In addition, some components may not be used.

[0010] In the present embodiment, a local coordinate system is defined for a sewing machine 1. In the following description, the local coordinate system defined for the sewing machine 1 is appropriately referred to as a sewing machine coordinate system. The sewing machine coordinate system is defined by the XYZ orthogonal coordinate system. In the present embodiment, the positional relationship of each part will be described based on the sewing machine coordinate system. A direction parallel to an X-axis in a predetermined plane is defined as an X-axis direction. A direction parallel to a Y-axis in the predetermined plane orthogonal to the X-axis is defined as a Y-axis direction. A direction parallel to a Z-axis orthogonal to the predetermined plane is defined as a Z-axis direction. In addition, a rotational direction or an inclination direction about the X-axis is defined as a θX direction. A rotational direction or an inclination direction about the Y-axis is defined as a θY direction. A rotational direction or an inclination direction about the Z-axis is defined as a θZ direction. In addition, in the present embodiment, a plane including the X-axis and the Y-axis is appropriately referred to as an XY plane. A plane including the X-axis and the Z-axis is appropriately referred to as an XZ-plane. A plane including the Y-axis and the Z-axis is appropriately referred to as a YZ-plane. The XY plane is parallel to the predetermined plane. The XY plane, the XZ plane, and the YZ plane are orthogonal to one another. Additionally, in the present embodiment, the XY plane is parallel to a horizontal plane. The Z-axis direction is an up-down direction. The +Z direction is an upward direction. The -Z direction is a downward direction. Note that the XY plane may be inclined with respect to the horizontal plane.[Physical Configuration of Sewing Machine]

[0011] Fig. 1 is a perspective view illustrating a sewing machine 1 according to the present embodiment. Fig. 2 is a perspective view illustrating a part of the sewing machine 1 according to the present embodiment. In the present embodiment, the sewing machine 1 is an electronic cycle sewing machine. The sewing machine 1 includes a sewing machine body 10, an operation device 20, and an imaging device 30. The operation device 20 is operated by an operator. The imaging device 30 is capable of capturing an image of an object to be sewn S.

[0012] The sewing machine body 10 is mounted on an upper surface of a table 2. The sewing machine body 10 includes a sewing machine frame 11, a needle bar 12, a throat plate 13, a holding member 15, an actuator 16, an actuator 17, and an actuator 18. The needle bar 12 and the throat plate 13 are each supported by the sewing machine frame 11. The holding member 15 is supported by the sewing machine frame 11 via a support member 14. The actuator 16 generates power to move the needle bar 12. The actuator 17 generates power to move the holding member 15. The actuator 18 generates power to move at least a portion of the holding member 15.

[0013] The sewing machine frame 11 includes a horizontal arm 11A, a bed 11B, a vertical arm 11C, and a head 11D. The horizontal arm 11A extends in the Y-axis direction. The bed 11B is arranged lower than horizontal arm 11A. The vertical arm 11C is arranged to connect an end portion on the +Y side of the horizontal arm 11A and the bed 11B. The head 11D is arranged on the -Y side of the horizontal arm 11A.

[0014] The needle bar 12 holds a sewing needle 3. The needle bar 12 holds the sewing needle 3 such that the sewing needle 3 is parallel to the Z-axis. The needle bar 12 is supported by the head 11D so as to be movable in the Z-axis direction.

[0015] The throat plate 13 supports the object to be sewn S. The throat plate 13 supports the holding member 15. The throat plate 13 is supported by the bed 11B. The throat plate 13 is arranged lower than the holding member 15.

[0016] The holding member 15 holds the object to be sewn S. The holding member 15 is capable of moving while holding the object to be sewn S, in the XY plane including a sewing position Ps directly below the sewing needle 3. The holding member 15 is capable of moving while holding the object to be sewn S, in the XY plane including an imaging position Pf directly below the imaging device 30. When the holding member 15 moves based on sewing data while holding the object to be sewn S, in the XY plane including the sewing position Ps, a stitch CH is formed on the object to be sewn S. The holding member 15 is supported by the horizontal arm 11A via the support member 14.

[0017] The holding member 15 includes a pressing member 15A and a lower plate 15B facing the pressing member 15A. The pressing member 15A is a frame-shaped member. The pressing member 15A is movable in the Z-axis direction. The lower plate 15B is arranged below the pressing member 15A. The holding member 15 holds the object to be sewn S by sandwiching the object to be sewn S with the pressing member 15A and the lower plate 15B.

[0018] When the pressing member 15A moves in the +Z direction, the pressing member 15A and the lower plate 15B are separated from each other. As a result, the operator can arrange the object to be sewn S between the pressing member 15A and the lower plate 15B. When the pressing member 15A moves in the -Z direction while the object to be sewn S is arranged between the pressing member 15A and the lower plate 15B, the object to be sewn S is sandwiched between the pressing member 15A and the lower plate 15B. As a result, the object to be sewn S is held by the holding member 15. In addition, when the pressing member 15A moves in the +Z direction, the object to be sewn S is released from being held by the holding member 15. As a result, the operator can take out the object to be sewn S from between the pressing member 15A and the lower plate 15B.

[0019] The actuator 16 generates power to move the needle bar 12 in the Z-axis direction. The actuator 16 includes a pulse motor. The actuator 16 is arranged on the horizontal arm 11A.

[0020] Inside the horizontal arm 11A, a horizontal arm shaft extending in the Y-axis direction is arranged. The actuator 16 is connected to an end portion on the +Y side of the horizontal arm shaft. An end portion on the -Y side of the horizontal arm shaft is connected to the needle bar 12 via a first power transmission mechanism. The first power transmission mechanism is arranged inside the head 11D. By an operation of the actuator 16, the horizontal arm shaft rotates. The power generated by the actuator 16 is transmitted to the needle bar 12 via the horizontal arm shaft and the first power transmission mechanism. As a result, the sewing needle 3 held by the needle bar 12 reciprocates in the Z-axis direction.

[0021] Inside the vertical arm 11C, a timing belt extending in the Z-axis direction is arranged. In addition, inside the bed 11B, a bed shaft extending in the Y-axis direction is arranged. A pulley is arranged on each of the horizontal arm shaft and the bed shaft. The timing belt is looped around each of the pulley arranged on the horizontal arm shaft and the pulley arranged on the bed shaft. The horizontal arm shaft and the bed shaft are connected to each other via the first power transmission mechanism including the timing belt.

[0022] Inside the bed 11B, a shuttle is arranged. In the shuttle, a bobbin contained in a bobbin case is accommodated. By an operation of the actuator 16, each of the horizontal arm shaft and the bed shaft rotates. The power generated by the actuator 16 is transmitted to the shuttle via the horizontal arm shaft, the timing belt, and the bed shaft. As a result, the shuttle rotates in synchronization with reciprocation of the needle bar 12 in the Z-axis direction.

[0023] The actuator 17 generates power to move the holding member 15 in the XY plane. The actuator 17 includes a pulse motor. The actuator 17 includes an X-axis motor 17X and a Y-axis motor 17Y. The X-axis motor 17X generates power to move the holding member 15 in the X-axis direction. The Y-axis motor 17Y generates power to move the holding member 15 in the Y-axis direction. The actuator 17 is arranged inside the bed 11B.

[0024] The power generated by the actuator 17 is transmitted to the holding member 15 via the support member 14. As a result, the holding member 15 can move in the X-axis direction and the Y-axis direction, respectively, between the sewing needle 3 and the throat plate 13. By an operation of the actuator 17, the holding member 15 can move while holding the object to be sewn S, in the XY plane including the sewing position Ps directly below the sewing needle 3.

[0025] The actuator 18 generates power to move the pressing member 15A of the holding member 15 in the Z-axis direction. The actuator 18 includes a pulse motor. When the pressing member 15A moves in the +Z direction, the pressing member 15A and the lower plate 15B are separated from each other. When the pressing member 15A moves in the -Z direction, the object to be sewn S is sandwiched between the pressing member 15A and the lower plate 15B.

[0026] As illustrated in Fig. 2, the sewing machine body 10 includes an intermediate pressing member 19 arranged around the sewing needle 3. The intermediate pressing member 19 restrains the object to be sewn S around the sewing needle 3. The intermediate pressing member 19 is supported by the head 11D so as to be movable in the Z-axis direction. The intermediate pressing member 19 is connected to the horizontal arm shaft arranged inside the horizontal arm 11A via a second power transmission mechanism. The second power transmission mechanism is a power transmission mechanism different from the first power transmission mechanism that transmits power to the needle bar 12. The second power transmission mechanism that transmits power to the intermediate pressing member 19 is arranged inside the head 11D. By an operation of the actuator 16, the intermediate pressing member 19 moves in the Z-axis direction in conjunction with the needle bar 12. The intermediate pressing member 19 suppresses lifting of the object to be sewn S caused by movement of the sewing needle 3.

[0027] The operation device 20 is operated by the operator. When the operation device 20 is operated, the sewing machine 1 is operated. In the present embodiment, the operation device 20 includes an operation panel 21 and an operation pedal 22. The operation panel 21 is mounted on the upper surface of the table 2. The operation pedal 22 is arranged below the table 2. The operator operates the operation pedal 22 with a foot. The sewing machine 1 operates when at least one of the operation panel 21 and the operation pedal 22 is operated by the operator.

[0028] The imaging device 30 captures an image of the object to be sewn S held by the holding member 15. The imaging device 30 includes an optical system and an image sensor that receives light incident through the optical system. The image sensor includes a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor.

[0029] The imaging device 30 is arranged higher than the throat plate 13 and the holding member 15. An imaging position Pf includes a position of an optical axis AX of the optical system of the imaging device 30. An imaging region FA is defined in the imaging device 30. The imaging region FA includes a field of view of the optical system of the imaging device 30. The imaging region FA includes the imaging position Pf. The imaging device 30 acquires an image of at least a portion of the object to be sewn S arranged in the imaging region FA. The imaging device 30 captures, from above, an image of at least a portion of the object to be sewn S arranged on an inner side of the pressing member 15A.

[0030] A position of the imaging device 30 is fixed. A relative position between the imaging device 30 and the sewing machine frame 11 is fixed. A relative position between the optical axis AX of the optical system of the imaging device 30 and the sewing needle 3 in the XY plane is fixed. Relative position data representing the relative position between the optical axis AX of the optical system of the imaging device 30 and the sewing needle 3 in the XY plane is known data that can be derived from design data of the sewing machine 1.

[0031] A position of an image acquired by the imaging device 30 is defined in the camera coordinate system. The position of the image defined in the camera coordinate system is converted into a position of the image defined in the sewing machine coordinate system by a predetermined transformation formula or transformation matrix.

[0032] Note that when a difference occurs between an actual position of the imaging device 30 and a position in the design data due to an attachment error of the imaging device 30, an accurate relative position between the optical axis AX of the optical system of the imaging device 30 and the sewing needle 3 can be calculated based on the difference between the actual position of the imaging device 30 in the XY plane and a position of the sewing needle 3 after movement. The difference between the actual position of the imaging device 30 in the XY plane and the position of the sewing needle 3 after movement is calculated by measuring a position of the sewing needle 3 in the XY plane after attachment of the imaging device 30 and moving the measured position of the sewing needle 3 toward the imaging device 30 by the known data.

[0033] The imaging device 30 captures an image of the object to be sewn S and outputs an object image SM representing an image relating to the object to be sewn S to a control device 40. In the present embodiment, the object image SM includes at least an image of the object to be sewn S after sewing processing. Note that the object image SM may include an image of the object to be sewn S before the sewing processing. In the present embodiment, the imaging device 30 captures an image of the object to be sewn S before the sewing processing. In addition, the imaging device 30 captures an image of the object to be sewn S after the sewing processing.

[0034] The imaging device 30 captures an imaging region FA including a correction point CP set on a target stitch line RL, as illustrated in Fig. 8 described below. The imaging device 30 sequentially captures a plurality of imaging regions FA corresponding to a plurality of correction points CP set on the target stitch line RL to be subjected to the sewing processing next, and outputs the object image SM to the control device 40.[Object To Be Sewn]

[0035] Fig. 3 is a cross-sectional view illustrating a part of the object to be sewn S according to the present embodiment. Fig. 4 is a plan view illustrating a part of the object to be sewn S according to the present embodiment. Fig. 5 is a cross-sectional view illustrating a part of the object to be sewn S according to the present embodiment. Fig. 6 is a plan view illustrating a part of the object to be sewn S according to the present embodiment. Figs. 3 and 4 illustrate the object to be sewn S before the sewing processing. Figs. 5 and 6 illustrate the object to be sewn S after the sewing processing. In the present embodiment, the object to be sewn S is a skin material used for a vehicle seat.

[0036] As illustrated in Fig. 3, the object to be sewn S includes a surface material 4, a padding material 5, and a back material 6. A hole 7 is provided in the surface material 4. The hole 7 may be provided to penetrate not only the surface material 4 but also the padding material 5 and the back material 6.

[0037] A surface of the surface material 4 is a seating surface that comes into contact with a passenger when the passenger sits on a vehicle seat. The surface material 4 includes at least one of woven fabric, nonwoven fabric, and leather. The padding material 5 has elasticity. The padding material 5 includes, for example, a urethane resin. The back material 6 includes at least one of woven fabric, nonwoven fabric, and leather.

[0038] As illustrated in Fig. 4, the hole 7 is provided in plurality in the surface material 4. The surface material 4 includes a texture area TA that is an area in which the holes 7 are formed at substantially equal intervals in the surface, and a stitch area SA that is an area in which the holes 7 are not formed in the surface. In the stitch area SA, a target stitch line RL of a stitch CH to be formed on the object to be sewn S is defined. The target stitch line RL is stored in a sewing data storage unit 51 of a storage device 50 described below.[Displacement of Surface of Object To Be Sewn]

[0039] The object to be sewn S has thickness and elasticity. As a stitch CH is formed on the object to be sewn S having thickness and elasticity, the object to be sewn S is compressed and contracted in the Z-axis direction at a position where the stitch CH is formed, as illustrated in Fig. 5. When the object to be sewn S is contracted, the surface of the object to be sewn S is pulled and displaced. As illustrated in Fig. 6, when the stitch CH is formed on the object to be sewn S, the surface of the object to be sewn S is displaced in the XY plane with respect to the target stitch line RL.

[0040] When the surface of the object to be sewn S is displaced in the XY plane with respect to the target stitch line RL, if the holding member 15 is moved along the target stitch line RL, it becomes difficult to form the stitch CH at a desired position on the surface of the object to be sewn S.

[0041] In the present embodiment, when the object to be sewn S is contracted due to the formation of the stitch CH and the surface of the object to be sewn S is thus displaced, the position of the target stitch line RL is corrected based on a displacement amount of the surface of the object to be sewn S. The holding member 15 is moved based on the corrected target stitch line RL.[Control Configuration of Sewing Machine]

[0042] Fig. 7 is a functional block diagram illustrating the sewing machine 1 according to the present embodiment. The sewing machine 1 includes a control device 40, a storage device 50, an input device 60, an output device 70, and an image processing device 100.

[0043] The control device 40 includes a computer system. As illustrated in Fig. 7, the control device 40 is connected to each of the actuator 16, the actuator 17, the actuator 18, the operation device 20, the imaging device 30, the storage device 50, the input device 60, the output device 70, and the image processing device 100. The actuator 16 moves the sewing needle 3 in the Z-axis direction. The actuator 17 moves the holding member 15 in the XY plane. The actuator 18 moves the pressing member 15A of the holding member 15 in the Z-axis direction. Additionally, the control device 40 is connected to a drive amount sensor 31 and a drive amount sensor 32. The drive amount sensor 31 detects a drive amount of the actuator 16. The drive amount sensor 32 detects a drive amount of the actuator 17.

[0044] The control device 40 outputs a control command to control the actuator 17 that moves the holding member 15, based on a processing result of the image processing device 100. The drive amount sensor 31 includes an encoder that detects a rotation amount of a pulse motor serving as the actuator 16. Detection data of the drive amount sensor 31 is output to the control device 40.

[0045] The drive amount sensor 32 includes an X-axis sensor 32X that detects a rotation amount of the X-axis motor 17X, and a Y-axis sensor 32Y that detects a rotation amount of the Y-axis motor 17Y. The X-axis sensor 32X includes an encoder that detects a rotation amount of the X-axis motor 17X. The Y-axis sensor 32Y includes an encoder that detects a rotation amount of the Y-axis motor 17Y. Detection data of the drive amount sensor 32 is output to the control device 40.

[0046] The drive amount sensor 32 serves as a position sensor that detects a position of the holding member 15 in the XY plane. The drive amount of the actuator 17 and a movement amount of holding member 15 correspond to each other on a one-to-one basis.

[0047] The X-axis sensor 32X can detect a movement amount of the holding member 15 in the X-axis direction from the origin in the sewing machine coordinate system by detecting a rotation amount of the X-axis motor 17X. The Y-axis sensor 32Y can detect a movement amount of the holding member 15 in the Y-axis direction from the origin in the sewing machine coordinate system by detecting a rotation amount of the Y-axis motor 17Y.

[0048] The control device 40 controls the actuator 16 on the basis of the detection data from the drive amount sensor 31. The control device 40 determines, for example, an operation timing of the actuator 16 on the basis of the detection data from the drive amount sensor 31.

[0049] The control device 40 controls the actuator 17 on the basis of the detection data from the drive amount sensor 32. The control device 40 feedback-controls the actuator 17 so that the holding member 15 moves to a desired position, based on the detection data from the drive amount sensor 32.

[0050] The control device 40 calculates a position of the holding member 15 in the XY plane on the basis of the detection data from the drive amount sensor 32. Based on the detection data from the drive amount sensor 32, the movement amount of the holding member 15 from the origin in the XY plane is detected. The control device 40 calculates a position of the holding member 15 in the XY plane on the basis of the detected movement amount of the holding member 15.

[0051] The control device 40 further includes an object image acquisition unit 41, a correction point setting unit 42, an object information output unit 43, a correction point information acquisition unit 44, a target stitch line correction unit 45, and a stitch point correction unit 46.

[0052] The object image acquisition unit 41 acquires an object image SM representing an image relating to the object to be sewn S. The object image acquisition unit 41 acquires the object image SM from the imaging device 30.

[0053] In the present embodiment, the object image acquisition unit 41 acquires the object image SM captured before the sewing processing from the imaging device 30. Note that when the object image SM captured by the imaging device 30 before the sewing processing is the same as design data of the object to be sewn S in an initial state stored in a design data storage unit 52, the object image acquisition unit 41 may acquire, as the object image SM before the sewing processing, the design data of the object to be sewn S in the initial state from the design data storage unit 52.

[0054] The correction point setting unit 42 sets a correction point CP that is set at an arbitrary position so as to correct the target stitch line RL defined in the stitch area SA. The correction point CP is set for the object to be sewn S before the sewing processing. When the stitch CH is formed, the surface of the object to be sewn S is displaced with respect to the correction point CP and the target stitch line RL. Accordingly, after forming the stitch CH on one target stitch line RL, a position of the target stitch line RL to be subjected to the sewing processing next is corrected based on a position of the correction point CP set on the target stitch line RL to be subjected to the sewing processing next.

[0055] Fig. 8 is an illustrative diagram of the correction points CP according to the present embodiment. Fig. 9 is an illustrative diagram of an interpolation point IP according to the present embodiment. As illustrated in Fig. 8, the correction points CP are set on the object to be sewn S. The correction points CP are used to correct the target stitch line RL. The correction points CP are set to overlap the target stitch line RL to be corrected in the stitch area SA. The positions of the correction points CP are defined in the sewing machine coordinate system. The correction points CP include a feature point FP and an interpolation point IP.

[0056] The feature point FP represents, for example, a correction point CP that is a position at which the target stitch line RL intersects, where the stitch area SA has a characteristic shape and that has a two-dimensional positioning element. The characteristic shape of the stitch area SA is, for example, an X shape. Additionally, feature points FP are set at both ends of the target stitch line RL. In the correction point setting unit 42, a position of the feature point FP on the object to be sewn S before the sewing processing is set together with information indicating that the correction point is the feature point FP.

[0057] The interpolation point IP represents a correction point CP that has no positioning element in a sewing direction, which is a direction along the target stitch line RL, and has a positioning element in a cross direction (e.g., see a cross direction CD in Fig. 13) intersecting the sewing direction, i.e., has only a one-dimensional positioning element. The sewing direction, which is a direction along the target stitch line RL, is, for example, a straight line section, a gently curved section, or the like (see a sewing direction SD in Fig. 9). In the present embodiment, the positioning element includes a hole pattern formed by the plurality of holes 7. Specifically, in the cross direction intersecting the sewing direction, hole patterns are provided on both sides of a stitch area SA in which the holes 7 are not formed, and the pattern is not uniform. Since the pattern is not uniform, positioning is possible in the cross direction intersecting the sewing direction. In contrast, in the sewing direction, similar hole patterns are continuous, and therefore positioning is difficult. The correction point setting unit 42 sets a position of the interpolation point IP on the object to be sewn S before the sewing processing, together with data of the sewing direction SD for the interpolation point IP and information indicating that the correction point is the interpolation point IP. The sewing direction SD is a tangential direction at the interpolation point IP with respect to the target stitch line RL.

[0058] The correction point CP is set to an arbitrary position by the operator. The operator can set the correction point CP to an arbitrary position on the object to be sewn S by operating the input device 60. The correction point information acquisition unit 44 sets the correction point CP on the basis of input data generated by the operation of the input device 60. The correction point CP may also be set automatically based on the design data. Additionally, after the operator sets the feature point FP to an arbitrary position, the interpolation point IP may be set based on the design data.

[0059] The correction point CP set by the operator or automatically is stored as correction point data in a correction point data storage unit 53 of the storage device 50 described below. In the sewing processing described below, in order to calculate a position after displacement of the correction point CP on the target stitch line RL, which is to be subjected to the sewing processing next, of the object to be sewn S whose surface has been displaced by sewing, the object image SM of the imaging region FA including the correction point CP set by the correction point setting unit 42 is acquired by the imaging device 30.

[0060] The object image acquisition unit 41 acquires the object image SM of the imaging region FA including the correction point CP set by the correction point setting unit 42, before the sewing processing. More specifically, the object image acquisition unit 41 sequentially acquires object images SM of a plurality of imaging regions FA corresponding to a plurality of correction points CP set on the target stitch line RL to be subjected to the next sewing processing. The object image acquisition unit 41 converts the position of the object image SM defined in the camera coordinate system into a position of the object image SM defined in the sewing machine coordinate system by a predetermined transformation formula or transformation matrix.

[0061] The object information output unit 43 outputs object information including the object image SM acquired by the object image acquisition unit 41 to the image processing device 100. The object information has coordinate data of the imaging region FA. The object information is associated with correction point prior information representing information on the correction point CP included in the object image SM. The correction point prior information includes an identifier that identifies each of the plurality of correction points CP.

[0062] The correction point prior information includes information on whether the correction point CP is a feature point FP or an interpolation point IP. When the correction point CP is a feature point FP, the correction point prior information includes position data of the feature point FP. When the correction point CP is an interpolation point IP, the correction point prior information includes position data of the interpolation point IP and data of the sewing direction SD.

[0063] Here, the positions of the feature point FP and the interpolation point IP indicate positions before displacement caused by the sewing processing. Therefore, for example, when the target stitch line RL is set at a width center of the stitch area SA, it is assumed that the position of the correction point CP in the object image SM output by the object information output unit 43 is at a position offset from the width center of the stitch area SA.

[0064] In addition, the correction point data may include arrangement data of the holes 7 in the texture area TA. For example, the correction point data may include relative position data of a hole 7A closest to the interpolation point IP with respect to the interpolation point IP, and a hole-to-hole distance DH between the hole 7A and a hole 7B closest to the hole 7A. The position of the hole 7A and the hole-to-hole distance DH may be information set by the operator, or may be extracted from the object image SM by the image processing device 100 when the object image SM is acquired before the sewing processing.

[0065] The correction point information acquisition unit 44 acquires correction point information output from the image processing device 100. The correction point information acquisition unit 44 acquires correction point information corresponding to the object information including the object image SM output by the object information output unit 43. The correction point information includes an identifier that identifies each of the plurality of correction points CP and displacement amount data of the correction point CP displaced due to the sewing processing. The correction point information acquisition unit 44 acquires displacement amount data of the correction points CP including all feature points FP and interpolation points IP corresponding to the target stitch line RL to be subjected to the sewing processing next.

[0066] The target stitch line correction unit 45 corrects a position of the target stitch line RL on the basis of the correction point information on the correction points CP acquired by the correction point information acquisition unit 44. Due to the displacement of the surface of the object to be sewn S caused by the sewing processing, the set correction points CP and the target stitch line RL defined by sewing data are displaced. The displacement amounts of the correction points CP are calculated by the image processing device 100 described below. The target stitch line correction unit 45 corrects the target stitch line RL on the basis of the calculated displacement amounts of the correction points CP.

[0067] Fig. 10 is an illustrative diagram of a method for correcting the target stitch line RL according to the present embodiment. The target stitch line correction unit 45 displaces the target stitch line RL by a displacement amount equal to, for example, the displacement amounts of the correction points CP.

[0068] The target stitch line correction unit 45 uses two adjacent correction points CP as references, and displaces, transforms, and rotates the target stitch line RL on the basis of a displacement amount, a transformation amount, and a rotation amount of a vector connecting the correction points CP. For example, as illustrated in Fig. 10, a pre-sewing vector SV11 connecting a correction point CP1 and a correction point CP2 changes to a post-sewing vector SV12 due to the displacement of the surface of the object to be sewn S. The target stitch line RL between the correction point CP1 and the correction point CP2 is transformed by a transformation amount equal to that from the pre-sewing vector SV11 to the post-sewing vector SV12. Additionally, the target stitch line RL between the correction point CP1 and the correction point CP2 is rotated by a rotation amount equal to that from the pre-sewing vector SV11 to the post-sewing vector SV12.

[0069] In addition, a pre-sewing vector SV21 connecting the correction point CP2 and a correction point CP3 changes to a post-sewing vector SV22 due to the displacement of the surface of the object to be sewn S. The target stitch line RL between the correction point CP2 and the correction point CP3 is transformed by a displacement amount equal to a displacement amount and a transformation amount from the pre-sewing vector SV21 to the post-sewing vector SV22. Additionally, the target stitch line RL between the correction point CP2 and the correction point CP3 is rotated by a rotation amount equal to that from the pre-sewing vector SV21 to the post-sewing vector SV22. Similarly, the target stitch line RL between the correction point CP3 and a correction point CP4 is also displaced, transformed, and rotated. In addition, the target stitch line RL between the correction point CP4 and a correction point CP5 is similarly displaced, transformed, and rotated. As a result, the target stitch line RL after correction can be obtained.

[0070] The stitch point correction unit 46 corrects a position at which the sewing needle 3 is lowered when forming a stitch CH on the target stitch line RL. When the target stitch line correction unit 45 corrects the target stitch line RL, a length of the target stitch line RL between the correction points CP changes. In addition, the target stitch line RL between different correction points CP may have variations in a rate of change of length. As a result, variations occur in the spacing of stitch points, which are positions at which the sewing needle 3 is lowered when forming a stitch CH on the target stitch line RL.

[0071] The stitch point correction unit 46 recalculates stitch points for one target stitch line RL after being corrected by the target stitch line correction unit 45 so that an interval between stitch points is fixed and the number of stitch points varies.

[0072] The storage device 50 includes a non-volatile memory and a volatile memory. The non-volatile memory is, for example, a read only memory (ROM), a storage, or the like. The volatile memory is, for example, a random access memory (RAM) or the like. As illustrated in Fig. 7, the storage device 50 is connected to the control device 40. The storage device 50 includes a sewing data storage unit 51, a design data storage unit 52, a correction point data storage unit 53, and a program storage unit 54.

[0073] The sewing data storage unit 51 stores sewing data that is referenced in the sewing processing. The sewing processing refers to processing for forming a stitch CH (see Fig. 6) on the object to be sewn S. The sewing data includes the target stitch line RL (see Figs. 4 and 6) of the stitch CH that is formed on the object to be sewn S and movement conditions of the holding member 15. The target stitch line RL defines a target shape of the stitch CH that is formed on the object to be sewn S and a target position of the stitch CH in the sewing machine coordinate system.

[0074] For the object to be sewn S, a plurality of target stitch lines RL are defined to form a plurality of stitches CH. In addition, the sewing processing includes a plurality of sewing processes corresponding to the plurality of stitches CH. A part of the object to be sewn S illustrated in Fig. 4 includes a first target stitch line RL1, a second target stitch line RL2, a third target stitch line RL3, and a fourth target stitch line RL4. In the state illustrated in Fig. 6, the stitch CH is formed on the first target stitch line RL1.

[0075] The movement conditions of the holding member 15 include a movement trajectory of the holding member 15 defined in the sewing machine coordinate system. The movement trajectory of the holding member 15 includes a movement trajectory of the holding member 15 in the XY plane. The movement conditions of the holding member 15 are determined based on the target stitch line RL.

[0076] For example, a first sewing process includes a process for forming a first stitch CH1 on the object to be sewn S on the basis of the first target stitch line RL1. The first sewing process is first performed after the object to be sewn S is held by the holding member 15. After the first sewing process, a second sewing process for forming a second stitch on the object to be sewn S on the basis of the second target stitch line RL2 is performed. Similarly, a third sewing process, a fourth sewing process, and a sewing process for forming a stitch CH on the corresponding target stitch line RL are sequentially performed.

[0077] The design data storage unit 52 stores design data of the object to be sewn S. The design data of the object to be sewn S includes a position and a range of the texture area TA and a position and a range of the stitch area SA on the surface of the object to be sewn S . When the object to be sewn S is designed by CAD (Computer Aided Design), the design data of the object to be sewn S includes CAD data.

[0078] The design data of the object to be sewn S is design data of the object to be sewn S in an initial state. The initial state of the object to be sewn S refers to a state before the first sewing process. That is, the initial state of the object to be sewn S refers to a state in which a stitch CH has not yet been formed on the object to be sewn S.

[0079] The correction point data storage unit 53 stores correction point data, which is data of the correction point CP, so as to correct the target stitch line RL defined in the stitch area SA. The correction point data includes data on the position of the feature point FP, and data on the position and the sewing direction SD of the interpolation point IP. In addition, the correction point data may include arrangement data of the holes 7 in the texture area TA. For example, the correction point data may include relative position data of a hole 7A closest to the interpolation point IP with respect to the interpolation point IP, and a hole-to-hole distance DH between the hole 7A and a hole 7B closest to the hole 7A.

[0080] The correction point data storage unit 53 stores the data of the correction point CP input by an operation of an operator on the input device 60. In addition, the correction point data storage unit 53 updates and stores the data of the correction point CP acquired by the correction point information acquisition unit 44 of the control device 40. The data of the correction point CP acquired by the correction point information acquisition unit 44 is data of the correction point CP that has moved in the sewing machine coordinate system due to the deformation of the object to be sewn S caused by formation of the stitch CH on the target stitch line RL one position prior to the target stitch line RL on which the correction point CP is set.

[0081] The program storage unit 54 stores a computer program for controlling the sewing machine 1. The computer program is read into the control device 40. The control device 40 controls the sewing machine 1 in accordance with the computer program stored in the program storage unit 54.

[0082] The input device 60 receives various input operations by the operator. The input data is generated by an operation on the input device 60. Examples of the input devices 60 include a computer keyboard, a mouse, operation keys, a microphone that receives voice input, and a touch panel.

[0083] The output device 70 outputs output data. Examples of the output device 70 include a display device and a printing device. The display device outputs display data as the output data. The printing device outputs print data as the output data. Examples of the display device include a flat panel display such as a liquid crystal monitor (LCD: Liquid Crystal Display) or an organic EL Display (OELD: Organic Electroluminescence Display). Examples of the printing device include an inkjet printer.

[0084] The image processing device 100 includes a computer system. As illustrated in Fig. 7, the image processing device 100 is connected to the control device 40. The image processing device 100 processes an image relating to the object to be sewn S. The image processing device 100 includes a learning model storage unit 101, an object information acquisition unit 102, a boundary line extraction unit 103, a feature point calculation unit 104, a boundary point calculation unit 105, an intermediate point calculation unit 106, an interpolation point calculation unit 107, and a correction point information output unit 108.

[0085] The learning model storage unit 101 stores a comparison model that captures, as two-dimensional data such as image data, a hole pattern on the surface of an object to be sewn S before sewing, or a learning model generated through machine learning.

[0086] The comparison model may be generated from one or more images and does not require a discriminator. The comparison model may use the image data itself as comparison data, or may use, as comparison data, two or more shape attributes serving as features and their representative coordinates, which are extracted in advance from the image data. The two or more shape attributes serving as features are, for example, perforation holes. The representative coordinates of the shape attributes are, for example, a center or a centroid.

[0087] In the present embodiment, a case in which learning data generated through machine learning is used will be described. The learning model generated through machine learning includes a discriminator for distinguishing between the texture area TA and the stitch area SA.

[0088] Fig. 11 is an illustrative diagram of learning processing according to the present embodiment. As training data, a large number of training images SW are collected. The training image SW represents training data relating to the object image SM after the sewing processing. Each of the training images SW is provided with boundary line data BW representing a boundary line BL between the texture area TA and the stitch area SA.

[0089] A learning model is generated by performing machine learning, using a predetermined machine learning algorithm, on the training images SW provided with the boundary line data BW serving as teacher data. Examples of the machine learning algorithm include at least one of a decision tree, a random forest, and a neural network.

[0090] The object information acquisition unit 102 acquires object information including the object image SM output by the object information output unit 43 of the control device 40. The object information is associated with correction point prior information representing information on the correction point CP included in the object image SM. The object image SM acquired by the object information acquisition unit 102 is an image after the surface of the object to be sewn S has been displaced due to the sewing processing. The correction point prior information acquired by the object information acquisition unit 102 is information on the correction point CP before the surface of the object to be sewn S is displaced due to the sewing processing. Therefore, a discrepancy occurs between the position of the correction point CP after the sewing processing and the acquired position data of the correction point CP.

[0091] Fig. 12 is an illustrative diagram of a reference point RP according to the present embodiment. Figs. 9 and 12 illustrate the object image SM of the same imaging region FA. Fig. 9 illustrates the object image SM before the sewing processing. Fig. 12 illustrates the object image SM after the surface of the object to be sewn S has been displaced due to the sewing processing. The reference point RP is a point indicating a position of an intermediate point HP before the surface of the object to be sewn S is displaced due to the sewing processing. The position data of the correction point CP acquired by the object information acquisition unit 102 is treated as position data of the reference point RP. The image processing device 100 calculates a correction point CP after displacement and outputs a displacement from the reference point RP to the correction point CP.

[0092] The boundary line extraction unit 103, based on the object image SM after the sewing processing acquired by the object information acquisition unit 102 and the learning model stored in the learning model storage unit 101, divides the surface of the object to be sewn S into the texture area TA and the stitch area SA and extracts the boundary line BL between the texture area TA and the stitch area SA.

[0093] The boundary line extraction unit 103 does not necessarily need to capture all contour edges of the boundary line BL, and may extract a centroid point of a hole 7 located on or in the vicinity of the boundary line BL. Matching processing is performed using, as a template, image data having the reference point RP as a center and including the texture region TA and the stitch region SA, or using, as a template, a centroid point coordinate arrangement pattern (coordinate point sequence data) of the holes 7. By performing these matching processes, the positional relationship between the boundary line BL and the reference point RP can be identified, and a sequence of centroid points of the holes 7 near the boundary line BL can be extracted.

[0094] As illustrated in Fig. 13 described below, the boundary line BL is a line passing through an edge of the texture area TA. When the texture area TA includes the holes 7, the boundary line BL is generated to connect outer edges of the holes 7 or centroid points of the holes 7 located on the outermost side in the texture area TA. The boundary line extraction unit 103 extracts the boundary line BL from any of the object image SM including the feature point FP illustrated in Fig. 8 and the object image SM including the interpolation point IP illustrated in Fig. 8.

[0095] The feature point calculation unit 104, when the boundary line extraction unit 103 extracts the boundary line BL from the object image SM including the feature point FP, calculates the position of the feature point FP on the basis of the boundary line BL. The feature point calculation unit 104, for example, as described in Patent Literature 2, generates the feature point FP on the basis of a plurality of points such as corner points, local maximum points, local minimum points, and inflection points on the boundary line BL, or centroid points of the holes 7, from their mutual positional relationship.

[0096] The boundary point calculation unit 105, when the boundary line extraction unit 103 extracts the boundary line BL from the object image SM including the interpolation point IP, calculates, based on the boundary line BL, boundary points BP indicating both end points in a width direction of the stitch area SA passing through the reference point RP. The width direction of the stitch area SA is a direction orthogonal to the sewing direction SD. The interpolation point IP is set in the stitch area SA having a linear or gently curved form. Accordingly, the boundary point calculation unit 105 can acquire and fix the width direction of the stitch area SA, and calculate two boundary points BP on the boundary line BL, which are two ends of the stitch area SA.

[0097] Fig. 13 is an illustrative diagram of a method for calculating the boundary point BP according to the present embodiment. The correction point prior information acquired by the object information acquisition unit 102 is associated with the sewing direction SD. The surface of the object to be sewn S is displaced in a cross direction CD, which is approximately orthogonal to the sewing direction SD, due to the sewing processing. Accordingly, first, a displacement amount of the correction point CP in the cross direction CD is calculated. The cross direction CD is the width direction of the stitch area SA.

[0098] As illustrated in Fig. 13, the boundary point calculation unit 105 first calculates a straight line passing through the reference point RP and parallel to the cross direction CD orthogonal to the sewing direction SD on the basis of the position of the reference point RP, which is the position of the correction point CP acquired by the object information acquisition unit 102, and the sewing direction SD. The boundary point calculation unit 105 then calculates, as the boundary point BP, a position where the straight line intersects the boundary line BL.

[0099] Note that the significance of the boundary point calculation unit 105 calculating the boundary point BP lies in calculating scale variation and angle variation of the object image SM. Without being limited to the method of calculating the boundary point BP, for example, when matching processing of image data or coordinate point sequence data is used to grasp the positional relationship between the boundary line BL and the reference point RP, the scale variation and angle variation of the object image SM can be calculated by adding scale and angle to variation parameters.

[0100] The intermediate point calculation unit 106 calculates an intermediate point HP, which indicates a midpoint in the width direction of the stitch area SA passing through the reference point RP.

[0101] Fig. 14 is an illustrative diagram of a method for calculating the intermediate point HP according to the present embodiment. The width direction of the stitch area SA passing through the reference point RP is the cross direction CD. In addition, both ends in the width direction of the stitch area SA are the two boundary points BP calculated by the boundary point calculation unit 105. Therefore, the intermediate point HP indicates a midpoint of the two boundary points BP calculated by the boundary point calculation unit 105. The intermediate point calculation unit 106 calculates the intermediate point HP on the basis of the two boundary points BP calculated by the boundary point calculation unit 105. Alternatively, when the positional relationship between the boundary line BL and the reference point RP is calculated by a matching process, the intermediate point calculation unit 106 calculates the intermediate point HP on the basis of the reference point RP and the scale variation and angle variation of the object image SM.

[0102] Since the target stitch line RL passes through the center in the width direction of the stitch area SA, in the present disclosure, the intermediate point HP may be regarded as the interpolation point IP after the sewing processing. That is, the interpolation point calculation unit 107 may calculate the interpolation point IP only by the correction of the cross direction CD. For example, when the interpolation points IP are densely set in the curved portion of the target stitch line RL, the target stitch line RL can be sufficiently corrected by the feature point FP and the intermediate point HP. However, when further precise correction is required, correction of the sewing direction SD is also performed.

[0103] The interpolation point calculation unit 107 calculates the interpolation point IP after the sewing processing. The interpolation point calculation unit 107 calculates a displacement in the sewing direction SD caused by the sewing processing, with respect to the intermediate point HP calculated by the intermediate point calculation unit 106. The interpolation point calculation unit 107 calculates the displacement in the sewing direction SD on the basis of the displacement of the perforation pattern, that is, the holes 7, in the texture area TA.

[0104] Fig. 15 is an illustrative diagram of an example of a method for calculating the interpolation point IP according to the present embodiment. The interpolation point calculation unit 107 first calculates a position of a reference hole 7R based on a relative position of the hole 7A (see Fig. 9) with respect to the interpolation point IP acquired by the object information acquisition unit 102. The relative position of the reference hole 7R with respect to the intermediate point HP is the same as that of the hole 7A with respect to the correction point CP.

[0105] The interpolation point calculation unit 107 then searches for the hole 7A in the cross direction CD with respect to the reference hole 7R. The search method for the hole 7A may be any known method such as pattern matching. Additionally, a search range may be set based on the hole-to-hole distance DH (see Fig. 9) between the hole 7B closest to the hole 7A and the hole 7A. Specifically, for example, the search range in the cross direction CD is set to a distance equal to one-half of the hole-to-hole distance DH on each side with the reference hole 7R as a center.

[0106] The interpolation point calculation unit 107 calculates a distance from the reference hole 7R to the detected hole 7A. The distance from the reference hole 7R to the detected hole 7A is a displacement amount in the cross direction CD of the interpolation point IP. The interpolation point calculation unit 107 calculates the interpolation point IP on the basis of the intermediate point HP and the displacement amount in the cross direction CD of the interpolation point IP.

[0107] Note that, when calculating the boundary line BL and the boundary point BP by matching processing using image data or coordinate point sequence data, a variation region of the interpolation point IP is predicted and a search region is set from the cross direction CD, the hole-to-hole distance DH, the maximum scale, and the angle variation amount, so that the interpolation point IP can be calculated only by the matching processing instead of the processes performed by the boundary line extraction unit 103, the feature point calculation unit 104, the boundary point calculation unit 105, the intermediate point calculation unit 106, and the interpolation point calculation unit 107.

[0108] The correction point information output unit 108 outputs, to the control device 40, correction point information representing information on a new correction point CP including the feature point FP calculated by the feature point calculation unit 104 and the interpolation point IP calculated by the interpolation point calculation unit 107. The correction point information includes an identifier that identifies each of the plurality of correction points CP. The correction point information includes displacement amount data of a newly calculated correction point CP with respect to the correction point CP acquired by the object information acquisition unit 102.[Control Device Processing]

[0109] Fig. 16 is a flowchart showing a sewing method according to the present embodiment. In the present embodiment, the sewing method includes an alignment process S10, an object image acquisition process S11, an object information output process S12, a correction point information acquisition process S13, a target stitch line correction process S14, a stitch point correction process S15, a sewing process S16, and an end determination process S17. The processes shown in Fig. 16 are executed by the control device 40.

[0110] The alignment process S10 is a process for associating the texture area TA and the stitch area SA of the object to be sewn S held by the holding member 15 with the sewing machine coordinate system. After the object to be sewn S before the sewing processing is held by the holding member 15, the imaging device 30 captures an image of the object to be sewn S. The imaging device 30 captures, for example, the correction points CP including a plurality of feature points FP and interpolation points IP of the object to be sewn S. Note that when an alignment mark is provided on the object to be sewn S, the imaging device 30 may capture an image of the alignment mark. The position of the image of the object to be sewn S captured by the imaging device 30 is defined in the camera coordinate system. The position of the image defined in the camera coordinate system is converted into a position of the image defined in the sewing machine coordinate system by a predetermined transformation formula or transformation matrix. As a result, the position of the texture area TA and the position of the stitch area SA of the object to be sewn S are defined in the sewing machine coordinate system.

[0111] The object image acquisition process S11 is a process for acquiring the object image SM. Before the first sewing process, the object image acquisition unit 41 may acquire the object image SM in the initial state from the imaging device 30, or may acquire the object image SM in the initial state from the design data storage unit 52.

[0112] After the first sewing process, the object image SM is acquired by the imaging device 30. The imaging device 30 acquires the object image SM after the first sewing process and before the second sewing process. Similarly, the imaging device 30 acquires the object image SM between respective sewing processes from the second sewing process to the final sewing process.

[0113] The object information output process S12 is a process for outputting object information including the object image SM acquired by the object image acquisition process S11 to the image processing device 100. The object information output unit 43 outputs object information including the object image SM acquired by the object image acquisition unit 41 to the image processing device 100. In the object information output process S12, the object images SM corresponding to one target stitch line RL may be output collectively, or each object image SM may be output individually.

[0114] The correction point information acquisition process S13 is a process for acquiring correction point information output from the image processing device 100. The correction point information acquisition unit 44 acquires correction point information corresponding to the object information including the object image SM output by the object information output unit 43. Note that, in the object information output process S12, after outputting the object information corresponding to one target stitch line RL, the control device 40 enters a standby state for the correction point information acquisition process S13 until the correction point information corresponding to the target stitch line RL is acquired.

[0115] The target stitch line correction process S14 is a process for correcting the target stitch line RL on the basis of the correction points CP acquired by the correction point information acquisition process S13. Due to the displacement of the surface of the object to be sewn S caused by the sewing processing, the correction points CP set by the operator and the target stitch line RL defined by the sewing data are displaced. The displacement amounts of the correction points CP are calculated by the image processing device 100 and acquired by the correction point information acquisition process S13. The target stitch line correction unit 45 corrects the target stitch line RL on the basis of the displacement amounts of the correction points CP acquired from the image processing device 100. The target stitch line correction unit 45 displaces the target stitch line RL by a displacement amount equal to, for example, the displacement amounts of the correction points CP.

[0116] For the target stitch line RL, a plurality of correction points CP are set. The target stitch line RL is corrected based on the displacement amount of each of the plurality of correction points CP. The target stitch line RL is corrected by using two adjacent correction points CP as references, and displacing, transforming, and rotating the target stitch line RL on the basis of a displacement amount, a transformation amount, and a rotation amount of a vector connecting the correction points CP. The position of the target stitch line RL in the sewing machine coordinate system is corrected.

[0117] The target stitch line correction process S14 is not performed before the first sewing process. The first sewing process is performed based on the target stitch line RL in the initial state defined by the sewing data. The target stitch line correction process S14 is performed after the first sewing process and before the second sewing process. Similarly, the target stitch line correction process S14 is performed between respective sewing processes from the second sewing process to the final sewing process.

[0118] The stitch point correction process S15 is a process for correcting the position at which the sewing needle 3 is lowered when forming a stitch CH on the target stitch line RL. By the target stitch line correction process S14, each target stitch line RL between adjacent correction points CP is corrected. As a result, variations occur in the spacing of the stitch points set on the target stitch line RL between different correction points CP. The stitch point correction unit 46 recalculates stitch points for one target stitch line RL after being corrected by the target stitch line correction process S14 so that an interval between stitch points is fixed and the number of stitch points varies.

[0119] The sewing process S16 is a process for forming a stitch CH on the basis of the target stitch line RL. The first sewing process is performed based on the target stitch line RL in the initial state defined by the sewing data. The sewing processes from the second sewing process to the final sewing process are performed based on the target stitch line RL after being corrected by the target stitch line correction process S14 and the stitch points after being corrected by the stitch point correction process S15. The control device 40 outputs a control command to the actuator 17 so that a stitch CH is formed along the target stitch line RL.

[0120] The end determination process S17 is a process for determining whether the sewing processing for the object to be sewn S has been completed. The control device 40 determines whether the sewing processing for the object to be sewn S has been completed on the basis of the sewing data. In a state in which the sewing processes from the first sewing process to the one immediately before the final sewing process have been completed, the control device 40 determines in the end determination process S17 that the sewing processing has not been completed. When the control device 40 determines that the sewing processing has not been completed, it proceeds to the object image acquisition process S11. In a state in which the final sewing process has been completed, the control device 40 determines in the end determination process S17 that the sewing processing has been completed.[Image Processing Device Processing]

[0121] Fig. 17 is a flowchart showing a correction point calculation method according to the present embodiment. In the present embodiment, the correction point calculation method includes an object information acquisition process S21, a correction point determination process S22, a boundary line extraction process S23, a feature point calculation process S24, a boundary line extraction process S25, an interpolation point calculation process S26, and a correction point information output process S27. The processes shown in Fig. 17 are executed by the image processing device 100.

[0122] The object information acquisition process S21 is a process for acquiring object information including the object image SM output in the object information output process S12 shown in Fig. 16 by the control device 40. The image processing device 100 starts a series of processes shown in Fig. 17 by acquiring the object information.

[0123] The correction point determination process S22 is a process for determining whether the correction point CP included in the object image SM acquired in the object information acquisition process S21 is a feature point FP or an interpolation point IP. The object information includes information on whether the correction point CP is either a feature point FP or an interpolation point IP. When the object information including the object image SM of the feature point FP is acquired, the control device 40 determines in the correction point determination process S22 that the correction point is a feature point FP, and proceeds to the boundary line extraction process S23. When the object information including the object image SM of the interpolation point IP is acquired, the control device 40 determines in the correction point determination process S22 that the correction point is an interpolation point IP, and proceeds to the boundary line extraction process S25.

[0124] The boundary line extraction process S23 is a process for dividing the surface of the object to be sewn S into a texture area TA and a stitch area SA and calculating a boundary line BL between the texture area TA and the stitch area SA, based on the object image SM including the feature point FP and a learning model,. The boundary line extraction process S23 is sequentially performed on the object images SM each including the feature point FP among the object images SM acquired in the object information acquisition process S21.

[0125] In the boundary line extraction process S23, the boundary line extraction unit 103 acquires a learning model from the learning model storage unit 101. The boundary line extraction unit 103, based on the object image SM including the feature point FP acquired in the object information acquisition process S21 and the acquired learning model, divides the surface of the object to be sewn S into a texture area TA and a stitch area SA and calculates a boundary line BL between the texture area TA and the stitch area SA.

[0126] The feature point calculation process S24 is a process for calculating a position of the feature point FP after the sewing processing on the basis of the feature point FP set on the object to be sewn S before the sewing processing and the boundary line BL extracted in the boundary line extraction process S23. The feature point calculation process S24 is performed according to a procedure referred to in the correction point calculation process S3 described in Patent Literature 2, for example.

[0127] The boundary line extraction process S25 is a process for dividing the surface of the object to be sewn S into a texture area TA and a stitch area SA and calculating a boundary line BL between the texture area TA and the stitch area SA, based on the object image SM including the interpolation point IP and a learning model. The boundary line extraction process S25 is sequentially performed on the object images SM each including the interpolation point IP among the object images SM acquired in the object information acquisition process S21.

[0128] In the boundary line extraction process S25, the boundary line extraction unit 103 acquires a learning model from the learning model storage unit 101. The boundary line extraction unit 103, based on the object image SM including the interpolation point IP acquired in the object information acquisition process S21 and the acquired learning model, divides the surface of the object to be sewn S into a texture area TA and a stitch area SA and calculates a boundary line BL between the texture area TA and the stitch area SA.

[0129] The interpolation point calculation process S26 is a process for calculating a position of the interpolation point IP after the sewing processing on the basis of the interpolation point IP set on the object to be sewn S before the sewing processing and the boundary line BL extracted in the boundary line extraction process S25. In the interpolation point calculation process S26 of the present embodiment, two boundary points BP and an intermediate point HP, which is the center between the boundary points, are calculated based on the boundary line BL, and the interpolation point IP is calculated based on a deviation amount of the hole 7 near the interpolation point IP and the intermediate point HP.

[0130] In the interpolation point calculation process S26, the boundary point calculation unit 105 calculates the cross direction CD, which is a direction orthogonal to the sewing direction SD passing through the reference point RP. The cross direction CD passing through the reference point RP is the width direction of the stitch area SA. The boundary point calculation unit 105 calculates two boundary points BP, which indicate both end points in the width direction of the stitch area SA, on the basis of the reference point RP and the cross direction CD.

[0131] The intermediate point calculation unit 106 calculates the intermediate point HP, which indicates a center in the width direction of the stitch area SA passing through the reference point RP, on the basis of the two boundary points BP. The intermediate point HP calculated by the interpolation point calculation process S26 may be regarded as the interpolation point IP after the sewing processing. That is, the interpolation point calculation unit 107 may calculate the interpolation point IP only by correction of the cross direction CD.

[0132] In the interpolation point calculation process S26 of the present embodiment, the intermediate point HP, the deviation amount of the hole 7 near the intermediate point HP, and the interpolation point IP are calculated. In other words, the correction of the sewing direction SD is also performed. The interpolation point calculation unit 107 calculates a position of the reference hole 7R with respect to the intermediate point HP on the basis of the relative position of the hole 7A with respect to the interpolation point IP acquired in the object information acquisition process S21. The interpolation point calculation unit 107 searches for the hole 7A in the cross direction CD with respect to the reference hole 7R, and calculates a distance from the reference hole 7R to the detected hole 7A. Based on the intermediate point HP and the distance from the reference hole 7R to the detected hole 7A, which represents the displacement amount of the interpolation point IP in the cross direction CD, the interpolation point calculation unit 107 calculates the interpolation point IP.

[0133] The correction point information output process S27 is a process for outputting, to the control device 40, correction point information representing information on a new correction point CP including the feature point FP calculated by the feature point calculation process S24 and the interpolation point IP calculated by the interpolation point calculation process S26. In the correction point information output process S27, correction point information corresponding to the object information acquired by the object information acquisition process S21 is output. The correction point information includes displacement amount data of a newly calculated correction point CP with respect to the correction point prior information on the correction point CP acquired by the object information acquisition process S21.[Computer System]

[0134] Fig. 18 is a block diagram illustrating an example of a computer system 1000. Each of the control device 40 and the image processing device 100 includes a computer system 1000. The computer system 1000 includes a processor 1001 such as a central processing unit (CPU), a main memory 1002 including a non-volatile memory and a volatile memory, a storage 1003, and an interface 1004 including an input / output circuit. The non-volatile memory is, for example, a read only memory (ROM) or the like. The volatile memory is, for example, a random access memory (RAM) or the like. Each of the functions of the control device 40 and the functions of the image processing device 100 is stored in the storage 1003 as a computer program. The processor 1001 reads a computer program from the storage 650, loads the computer program into the main memory 1002, and executes the above-described processes in accordance with the computer program. Note that the computer program may be transmitted to the computer system 1000 via a network.

[0135] The computer program includes various application software for implementing the functions described above. In accordance with the above embodiments, the computer program may perform the following: dividing the surface of the object to be sewn S into a texture area TA and a stitch area SA and extracting a boundary line BL between the texture area TA and the stitch area SA, based on the object image SM representing an image relating to the object to be sewn S; acquiring object information including the object image SM after the sewing processing and the correction point prior information representing information on the correction point CP before the sewing processing, the correction point CP being for correcting the target stitch line RL defined in the stitch area SA; when the correction point prior information includes position data of the feature point FP having a two-dimensional positioning element, calculating the feature point FP after the sewing processing on the basis of the boundary line BL after the sewing processing and the position data before the sewing processing; and when the correction point prior information includes position data and sewing direction data of the interpolation point IP having a one-dimensional positioning element, the sewing direction data representing a sewing direction SD, calculating the interpolation point IP after the sewing processing on the basis of the boundary line BL after the sewing processing, and the position data and the sewing direction data before the sewing processing.[Effects]

[0136] As described above, according to the present embodiment, the boundary line BL between the texture area TA in which the holes 7 are provided and the stitch area SA in which no hole 7 is provided on the surface of the object to be sewn S is calculated based on the object image SM of the object to be sewn S. The correction point prior information is acquired which represents information on the correction point CP before the sewing processing, the correction point CP being for correcting the target stitch line RL defined in the stitch area SA. The correction point CP includes the feature point FP having a two-dimensional positioning element, and the interpolation point IP having a one-dimensional positioning element but not a two-dimensional positioning element.

[0137] When the correction point CP is a feature point FP, the correction point prior information includes position data of the feature point FP. Based on the boundary line BL after the sewing processing and the position data of the feature point FP before the sewing processing, the feature point FP after the sewing processing is calculated. When the correction point CP is the interpolation point IP, the correction point prior information includes the position data and sewing direction data of the interpolation point IP, the sewing direction data representing the sewing direction SD. Based on the boundary line BL after the sewing processing and the position data and sewing direction data of the interpolation point IP, the interpolation point IP after the sewing processing is calculated.

[0138] In other words, the feature point FP having a two-dimensional positioning element is such that the position of the feature point FP after the sewing processing can be estimated from a two-dimensionally characteristic boundary line BL. Therefore, the displacement amount of the feature point FP can be calculated based on the position data of the feature point FP before the sewing processing and the estimated position of the feature point FP after the sewing processing.

[0139] In addition, for the interpolation point IP having only a one-dimensional positioning element, the displacement of the interpolation point IP in the cross direction CD is calculated by utilizing the fact that, due to the sewing processing, the surface of the object to be sewn S is displaced in the cross direction CD substantially orthogonal to the sewing direction SD. The one-dimensionally characteristic boundary line BL is a boundary BL that is characteristic in the cross direction CD. Accordingly, since the position of the interpolation point IP after the sewing processing can be estimated from the boundary line BL at a position in the cross direction CD with respect to the interpolation point IP before the sewing processing, the displacement amount of the interpolation point IP can be calculated based on the position data of the interpolation point IP before the sewing processing and the estimated position of the interpolation point IP after the sewing processing.

[0140] As a result, the displacement amount of the correction point CP, including the feature point FP and the interpolation point IP, from before to after the sewing processing, is calculated. The displacement amount of the correction point CP is calculated, so that the displacement of the surface of the object to be sewn S caused by the sewing processing is properly recognized. The displacement of the surface of the object to be sewn S is properly recognized, so that even when the surface of the object to be sewn S is displaced, the position of the target stitch line RL can be appropriately corrected. Accordingly, the control device 40 can form the stitch CH on the basis of the target stitch line RL.[Other Embodiments]

[0141] Although the embodiments of the present invention have been described, the present disclosure is not limited to the contents of the embodiments. The embodiments and modifications described above can be appropriately combined as long as the processing contents are not made inconsistent. In addition, the components described above include those that can be easily conceived by one skilled in the art, those that are substantially identical, and those that fall within the scope of equivalents. Additionally, the components described above can be appropriately combined. Further, various omissions, substitutions, or changes of the components can be made without departing from the gist of the embodiments described above.

[0142] For example, in the embodiments described above, when acquiring the object image SM by the imaging device 30, the holding member 15 holding the object to be sewn S moves in the XY plane while the position of the imaging device 30 remains fixed. The imaging region FA of the imaging device 30 may move in the XY plane while the position of the object to be sewn S remains fixed, or both the imaging region FA and the object to be sewn S may move in the XY plane.

[0143] In addition, in the embodiments described above, the boundary line BL is generated so as to pass through the outer edges of the holes 7 located on the outermost side in the texture area TA. The boundary line BL may be generated so as to pass through the holes 7 located on the outermost side in the texture area TA.

[0144] In addition, among the processes described in the above embodiment, all or some of the processes described as being performed automatically may be performed manually. Alternatively, all or some of the processes described as being performed manually in the above embodiment may be performed automatically by a known method. In addition, unless otherwise specified, the processing procedures, specific names, and information including various data and parameters described in the above text and drawings may be arbitrarily changed. For example, the various types of information illustrated in the respective drawings are not limited to the information illustrated.

[0145] In addition, each component of each device illustrated is functionally conceptual and does not necessarily need to be physically configured as illustrated. That is, the specific form of distribution and integration of each device is not limited to that illustrated, and all or some of the devices can be functionally or physically distributed and integrated into arbitrary units according to various loads, usage situations, and the like. For example, the boundary point calculation unit 105 and the intermediate point calculation unit 106 may be included in the interpolation point calculation unit 107. In addition, the learning model storage unit 101 may include two machine-learned learning models, respectively for a two-dimensionally characteristic training image SW in which a feature point FP is set and for a one-dimensionally characteristic training image SW in which an interpolation point IP is set. In addition, the control device 40 and the image processing device 100 may be configured by a plurality of computers divided into several functions. Additionally, some of the functions of the computers may be provided by a cloud server that executes various functions in the form of cloud computing.

[0146] In addition, the image processing device 100 may be configured by a computer that is completely independent of the sewing machine 1. The image processing device 100 may be connected to the control devices 40 of a plurality of sewing machines 1 and may execute a process for calculating a correction point CP for the object image SM output from each control device 40. In addition, the image processing device 100 may be connected to a communication network including short-range wireless communication such as Bluetooth (registered trademark) or wireless LAN (Local Area Network), and may be connected to the sewing machine 1 via the communication network.

[0147] The present disclosure includes the following aspects. (1) An image processing device including: an object information acquisition unit configured to acquire object information including an object image representing an image relating to an object to be sewn after sewing processing, and correction point prior information representing information on a correction point before the sewing processing, the correction point being for correcting a target stitch line defined in a stitch area that is an area in which a stitch is formed; a feature point calculation unit configured, when the correction point prior information includes position data of a feature point having a two-dimensional positioning element, to calculate the feature point after the sewing processing on the basis of the object image after the sewing processing and the position data before the sewing processing; and an interpolation point calculation unit configured, when the correction point prior information includes position data and sewing direction data of an interpolation point having a one-dimensional positioning element, the sewing direction data representing a sewing direction, to calculate the interpolation point after the sewing processing on the basis of the object image after the sewing processing and the position data and the sewing direction data before the sewing processing. (2) The image processing device according to the above (1), further including a boundary line extraction unit configured to divide a surface of the object to be sewn into a texture area and a stitch area and to extract a boundary line between the texture area and the stitch area, based on the object image representing an image relating to the object to be sewn, wherein the feature point calculation unit calculates the feature point after the sewing processing on the basis of the boundary line extracted from the object image after the sewing processing and the position data before the sewing processing, and the interpolation point calculation unit calculates the interpolation point after the sewing processing on the basis of the boundary line extracted from the object image after the sewing processing and the position data before the sewing processing. (3) The image processing device according to the above (2), wherein the interpolation point calculation unit calculates two boundary points indicating points where the boundary line intersects a straight line passing through the interpolation point and parallel to a cross direction orthogonal to the sewing direction, calculates an intermediate point indicating a midpoint of the two boundary points, and calculates a displacement amount of the interpolation point in the cross direction on the basis of the interpolation point and the intermediate point before the sewing processing. (4) The image processing device according to the above (2), wherein the interpolation point calculation unit, under a condition that a displacement amount of the interpolation point in the sewing direction is equal to or less than one half of a hole-to-hole distance among a plurality of holes included in the texture area, calculates the displacement amount of the interpolation point in the sewing direction on the basis of the interpolation point before the sewing processing and displacement amounts of the plurality of holes included in the texture area. (5) A sewing machine including: a holding member configured to be movable while holding an object to be sewn in a predetermined plane including a sewing position directly below a sewing needle, an actuator configured to generate power to move the holding member, and a control device configured to output a control command to control the actuator on the basis of a processing result of the image processing device according to any one of the above (1) to (4). (6) The sewing machine according to the above (5), wherein the control device includes a correction point information acquisition unit configured to acquire, from the image processing device, correction point information representing information on a plurality of correction points including displacement amounts of the feature point and the interpolation point after the sewing processing, and a target stitch line correction unit configured to correct the target stitch line on the basis of the correction point information. (7) The sewing machine according to the above (5) or (6), wherein the control device further includes a stitch point correction unit configured to recalculate stitch points for the target stitch line after being corrected so that an interval between the stitch points is fixed and the number of the stitch points varies. (8) An image processing method including: dividing a surface of an object to be sewn into a texture area and a stitch area and extracting a boundary line between the texture area and the stitch area, based on an object image representing an image relating to the object to be sewn; when correction point prior information representing information on a correction point before sewing processing, the correction point being for correcting a target stitch line defined in the stitch area, includes position data of a feature point having a two-dimensional positioning element, calculating the feature point after the sewing processing on the basis of the boundary line after the sewing processing and the position data before the sewing processing; and when the correction point prior information includes position data and sewing direction data of an interpolation point having a one-dimensional positioning element, the sewing direction data representing a sewing direction, calculating the interpolation point after the sewing processing on the basis of the boundary line after the sewing processing, and the position data and the sewing direction data before the sewing processing. (9) A sewing correction method further including correcting the target stitch line on the basis of correction point information representing information on a plurality of correction points including displacement amounts of the feature point and interpolation point after the sewing processing calculated in the image processing method according to the above (8).

[0148] The present application is based on Japanese Patent Application No. 2023-110448 filed on July 5, 2023, the contents of which are incorporated herein by reference.

Claims

1. An image processing device comprising: an object information acquisition unit configured to acquire object information comprising an object image representing an image relating to an object to be sewn after sewing processing, and correction point prior information representing information on a correction point before the sewing processing, the correction point being for correcting a target stitch line defined in a stitch area that is an area in which a stitch is formed; a feature point calculation unit configured, when the correction point prior information comprises position data of a feature point having a two-dimensional positioning element, to calculate the feature point after the sewing processing on the basis of the object image after the sewing processing and the position data before the sewing processing; and an interpolation point calculation unit configured, when the correction point prior information comprises position data and sewing direction data of an interpolation point having a one-dimensional positioning element, the sewing direction data representing a sewing direction, to calculate the interpolation point after the sewing processing on the basis of the object image after the sewing processing and the position data and the sewing direction data before the sewing processing.

2. The image processing device according to claim 1, further comprising a boundary line extraction unit configured to divide a surface of the object to be sewn into a texture area and a stitch area and to extract a boundary line between the texture area and the stitch area, based on the object image, wherein the feature point calculation unit calculates the feature point after the sewing processing on the basis of the boundary line extracted from the object image after the sewing processing and the position data before the sewing processing, and the interpolation point calculation unit calculates the interpolation point after the sewing processing on the basis of the boundary line extracted from the object image after the sewing processing and the position data before the sewing processing.

3. The image processing device according to claim 2, wherein the interpolation point calculation unit calculates two boundary points indicating points where the boundary line intersects a straight line passing through the interpolation point and parallel to a cross direction orthogonal to the sewing direction, calculates an intermediate point indicating a midpoint of the two boundary points, and calculates a displacement amount of the interpolation point in the cross direction on the basis of the interpolation point and the intermediate point before the sewing processing.

4. The image processing device according to claim 2, wherein the interpolation point calculation unit, under a condition that a displacement amount of the interpolation point in the sewing direction is equal to or less than one half of a hole-to-hole distance among a plurality of holes included in the texture area, calculates the displacement amount of the interpolation point in the sewing direction on the basis of the interpolation point before the sewing processing and displacement amounts of the plurality of holes included in the texture area.

5. A sewing machine comprising: a holding member configured to be movable while holding an object to be sewn in a predetermined plane comprising a sewing position directly below a sewing needle, an actuator configured to generate power to move the holding member, and a control device configured to output a control command to control the actuator on the basis of a processing result of the image processing device according to any one of claims 1 to 4.

6. The image processing device according to claim 5, wherein the control device comprises a correction point information acquisition unit configured to acquire, from the image processing device, correction point information representing information on a plurality of correction points comprising displacement amounts of the feature point and the interpolation point after the sewing processing, and a target stitch line correction unit configured to correct the target stitch line on the basis of the correction point information.

7. The image processing device according to claim 6, wherein the control device further comprises a stitch point correction unit configured to recalculate stitch points for the target stitch line after being corrected so that an interval between the stitch points is fixed and the number of the stitch points varies.

8. An image processing method comprising: dividing a surface of an object to be sewn into a texture area and a stitch area and extracting a boundary line between the texture area and the stitch area, based on an object image representing an image relating to the object to be sewn; when correction point prior information representing information on a correction point before sewing processing, the correction point being for correcting a target stitch line defined in the stitch area, comprises position data of a feature point having a two-dimensional positioning element, calculating the feature point after the sewing processing on the basis of the boundary line after the sewing processing and the position data before the sewing processing; and when the correction point prior information comprises position data and sewing direction data of an interpolation point having a one-dimensional positioning element, the sewing direction data representing a sewing direction, calculating the interpolation point after the sewing processing on the basis of the boundary line after the sewing processing, and the position data and the sewing direction data before the sewing processing.

9. A sewing correction method further comprising correcting the target stitch line on the basis of correction point information representing information on a plurality of correction points comprising displacement amounts of the feature point and interpolation point after the sewing processing calculated in the image processing method according to claim 8.

Images