Image processing device, sewing machine, image processing method and sewing correction method

Abstract

To form stitches based on a target stitch line by properly recognizing the displacement of a surface of a sewing object.SOLUTION: An image processing device includes: an object information acquisition part for acquiring an object image showing an image related to a sewing object S after sewing processing and correction point prior information before sewing processing of a correction point for correcting a target stitch line prescribed in a stitch region which is a region where stitches are formed; a characteristic point calculation part for, in the case where the correction point prior information includes position data of a characteristic point, calculating the characteristic point after sewing processing based on the object image after sewing processing and the position data before sewing processing; and an interpolation point calculation part for, in the case where the correction point prior information includes the position data of the interpolation point and the sewing direction data, calculating the interpolation point after sewing processing, based on the object image after sewing processing and position data and the sewing direction data before sewing processing.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device, a sewing machine, an image processing method, and a sewing correction method. [Background technology]

[0002] In order to improve the design of a sewing object, stitches may be formed on the sewing object. Patent Document 1 discloses a technique for forming stitches on a cover material used for a vehicle seat. However, when stitches are formed on a sewing object having a thickness and elasticity, such as a cover material used for a vehicle seat, the sewing object may shrink and the surface of the sewing object may be displaced. In response to this, Patent Document 2 discloses a technique for appropriately recognizing the displacement of the surface of the sewing object and forming stitches based on a target stitch line. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-162957 A [Patent Document 2] Patent Publication No. 2021-074074 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Document 2, since the positional deviation is detected based on the perforation pattern, the reference pattern needs to have a two-dimensional unique shape feature, which makes it difficult to apply the method to simple design patterns such as belt-like curves.

[0005] An aspect of the present invention is to appropriately recognize displacements on the surface of a sewing object and form stitches based on a target stitch line. [Means for solving the problem]

[0006] According to an aspect of the present invention, there is provided an image processing device comprising: an object information acquisition unit that acquires object information including an object image showing an image of a sewn object after a sewing process; and advance correction point information showing information before the sewing process of a correction point for correcting a target stitch line defined in a stitch area, which is an area where stitches are formed; a feature point calculation unit that calculates the feature point after the sewing process based on the object image after the sewing process and the position data before the sewing process when the advance correction point information includes position data of a feature point having a two-dimensional positioning element; and an interpolation point calculation unit that calculates the interpolation point after the sewing process based on the object image after the sewing process and the position data before the sewing process and the sewing direction data when the advance correction point information includes position data of an interpolation point having a one-dimensional positioning element and sewing direction data showing a sewing direction. Effect of the Invention

[0007] According to an aspect of the present invention, the displacement of the surface of the sewing object can be appropriately recognized and stitches can be formed based on a target stitch line. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a perspective view showing a sewing machine according to the present embodiment. [Diagram 2] FIG. 2 is a perspective view showing a part of the sewing machine according to the present embodiment. [Diagram 3] FIG. 3 is a cross-sectional view showing a part of the sewing object according to the present embodiment. [Figure 4] FIG. 4 is a plan view showing a part of the sewing object according to the present embodiment. [Diagram 5] FIG. 5 is a cross-sectional view showing a part of the sewing object according to the present embodiment. [Figure 6] FIG. 6 is a plan view showing a part of the sewing object according to the present embodiment. [Figure 7] FIG. 7 is a functional block diagram showing the sewing machine according to the present embodiment. [Figure 8] FIG. 8 is a diagram for explaining the correction points according to the present embodiment. [Figure 9] FIG. 9 is a diagram for explaining the interpolation points according to the present embodiment. [Figure 10] FIG. 10 is a diagram for explaining a method of correcting the target stitch line according to the present embodiment. [Figure 11] FIG. 11 is a diagram for explaining the learning process according to the present embodiment. [Figure 12] FIG. 12 is a diagram for explaining the reference point according to the present embodiment. [Figure 13] FIG. 13 is a diagram for explaining a method of calculating a boundary point according to this embodiment. [Figure 14] FIG. 14 is a diagram for explaining a method of calculating the midpoint according to this embodiment. [Figure 15] FIG. 15 is a diagram for explaining an example of a method for calculating a correction point according to the present embodiment. [Figure 16] FIG. 16 is a flowchart showing the sewing method according to this embodiment. [Figure 17] FIG. 17 is a flowchart showing a correction point calculation method according to this embodiment. [Figure 18] FIG. 18 is a block diagram showing a computer system according to this embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings, but the present invention is not limited thereto. The components of the embodiments described below can be appropriately combined. In addition, some components may not be used.

[0010] In this embodiment, a local coordinate system is defined for the sewing machine 1. In the following description, the local coordinate system defined for the sewing machine 1 is appropriately referred to as the sewing machine coordinate system. The sewing machine coordinate system is defined by an XYZ orthogonal coordinate system. In this embodiment, the positional relationship of each part will be described based on the sewing machine coordinate system. The direction parallel to the X axis in a predetermined plane is defined as the X axis direction. The direction parallel to the Y axis in a predetermined plane perpendicular to the X axis is defined as the Y axis direction. The direction parallel to the Z axis perpendicular to the predetermined plane is defined as the Z axis direction. In addition, the rotation direction or tilt direction around the X axis is defined as the θX direction. The rotation direction or tilt direction around the Y axis is defined as the θY direction. The rotation direction or tilt direction around the Z axis is defined as the θZ direction. In this embodiment, the plane including the X axis and the Y axis is appropriately referred to as the XY plane. The plane including the X axis and the Z axis is appropriately referred to as the XZ plane. The plane including the Y axis and the Z axis is appropriately referred to as the YZ plane. The XY plane is parallel to the predetermined plane. The XY plane, the XZ plane, and the YZ plane are orthogonal to each other. In this embodiment, the XY plane and the horizontal plane are parallel to each other. The Z axis direction is the up-down direction. The +Z direction is the upward direction, and the -Z direction is the downward direction. The XY plane may be inclined with respect to the horizontal plane.

[0011] [Physical configuration of the sewing machine] Fig. 1 is a perspective view showing a sewing machine 1 according to this embodiment. Fig. 2 is a perspective view showing a part of the sewing machine 1 according to this embodiment. In this embodiment, the sewing machine 1 is an electronic cycle sewing machine. The sewing machine 1 includes a sewing machine body 10, an operation device 20 operated by an operator, and an imaging device 30 capable of capturing an image of a sewing object S.

[0012] Sewing machine body 10 is mounted on the upper surface of table 2. Sewing machine body 10 has sewing machine frame 11, needle bar 12 supported by sewing machine frame 11, needle plate 13 supported by sewing machine frame 11, holding member 15 supported by sewing machine frame 11 via support member 14, actuator 16 that generates power to move needle bar 12, actuator 17 that generates power to move holding member 15, and actuator 18 that generates power to move at least a portion of holding member 15.

[0013] The sewing machine frame 11 has a horizontal arm 11A extending in the Y-axis direction, a bed 11B arranged below the horizontal arm 11A, a vertical arm 11C connecting the +Y side end of the horizontal arm 11A to the bed 11B, and a head 11D arranged on the -Y side of the horizontal arm 11A.

[0014] Needle bar 12 holds sewing needle 3. Needle bar 12 holds sewing needle 3 so that sewing needle 3 is parallel to the Z-axis. Needle bar 12 is supported by head 11D so as to be movable in the Z-axis direction.

[0015] The needle plate 13 supports a sewing object S. The needle plate 13 supports a holding member 15. The needle plate 13 is supported by the bed 11B. The needle plate 13 is disposed below the holding member 15.

[0016] The holding member 15 holds the sewing object S. The holding member 15 is capable of moving while holding the sewing object S within an XY plane including a sewing position Ps directly below the sewing needle 3. The holding member 15 is capable of moving while holding the sewing object S within an XY plane including an imaging position Pf directly below the imaging device 30. With the holding member 15 holding the sewing object S, the holding member 15 moves within the XY plane including the sewing position Ps based on the sewing data, thereby forming a stitch CH on the sewing object S. The holding member 15 is supported by the horizontal arm 11A via the support member 14.

[0017] The holding member 15 has a presser member 15A and a lower plate 15B facing the presser member 15A. The presser member 15A is a frame-shaped member. The presser member 15A is movable in the Z-axis direction. The lower plate 15B is disposed below the presser member 15A. The holding member 15 holds the sewing object S by sandwiching the sewing object S between the presser member 15A and the lower plate 15B.

[0018] When the presser member 15A moves in the +Z direction, the presser member 15A and the lower plate 15B are separated. This allows the operator to place the sewing object S between the presser member 15A and the lower plate 15B. When the presser member 15A moves in the -Z direction with the sewing object S placed between the presser member 15A and the lower plate 15B, the sewing object S is sandwiched between the presser member 15A and the lower plate 15B. This allows the sewing object S to be held by the holding member 15. Furthermore, when the presser member 15A moves in the +Z direction, the holding of the sewing object S by the holding member 15 is released. This allows the operator to remove the sewing object S from between the presser member 15A and the lower plate 15B.

[0019] The actuator 16 generates power to move the needle bar 12 in the Z-axis direction. The actuator 16 includes a pulse motor. The actuator 16 is disposed on the horizontal arm 11A.

[0020] A horizontal arm shaft extending in the Y-axis direction is disposed inside horizontal arm 11A. Actuator 16 is connected to the +Y side end of the horizontal arm shaft. The -Y side end of the horizontal arm shaft is connected to needle bar 12 via a first power transmission mechanism disposed inside head 11D. Operation of actuator 16 rotates the horizontal arm shaft. Power generated by actuator 16 is transmitted to needle bar 12 via the horizontal arm shaft and the first power transmission mechanism. As a result, sewing machine needle 3 held by needle bar 12 moves back and forth in the Z-axis direction.

[0021] A timing belt extending in the Z-axis direction is disposed inside the vertical arm 11C. A bed shaft extending in the Y-axis direction is disposed inside the bed 11B. A pulley is disposed on each of the horizontal arm shaft and the bed shaft. The timing belt is hung on each of the pulleys disposed on the horizontal arm shaft and the bed shaft. The horizontal arm shaft and the bed shaft are connected via a first power transmission mechanism including the timing belt.

[0022] A shuttle is disposed inside bed 11B. A bobbin placed in a bobbin case is housed in the shuttle. The horizontal arm shaft and the bed shaft are each rotated by the operation of actuator 16. Power generated by actuator 16 is transmitted to the shuttle via the horizontal arm shaft, a timing belt, and the bed shaft. As a result, the shuttle rotates in synchronization with the reciprocating movement of needle bar 12 in the Z-axis direction.

[0023] Actuator 17 generates power to move holding member 15 in the XY plane. Actuator 17 includes a pulse motor. Actuator 17 includes an X-axis motor 17X that generates power to move holding member 15 in the X-axis direction, and a Y-axis motor 17Y that generates power to move holding member 15 in the Y-axis direction. Actuator 17 is disposed inside bed 11B.

[0024] The power generated by the actuator 17 is transmitted to the holding member 15 via the support member 14. This allows the holding member 15 to move in both the X-axis and Y-axis directions between the sewing needle 3 and the needle plate 13. By operating the actuator 17, the holding member 15 can move while holding the sewing object S within an XY plane including the sewing position Ps directly below the sewing needle 3.

[0025] The actuator 18 generates power to move the presser member 15A of the holding member 15 in the Z-axis direction. The actuator 18 includes a pulse motor. When the presser member 15A moves in the +Z direction, the presser member 15A and the lower plate 15B are separated from each other. When the presser member 15A moves in the -Z direction, the sewing object S is sandwiched between the presser member 15A and the lower plate 15B.

[0026] As shown in FIG. 2, the sewing machine body 10 has an intermediate presser member 19 arranged around the sewing needle 3. The intermediate presser member 19 holds down the sewing object S around the sewing needle 3. The intermediate presser member 19 is supported by the head 11D so as to be movable in the Z-axis direction. The intermediate presser member 19 is connected to a horizontal arm shaft arranged inside the horizontal arm 11A via a second power transmission mechanism that is different from the first power transmission mechanism that transmits power to the needle bar 12. The second power transmission mechanism that transmits power to the intermediate presser member 19 is arranged inside the head 11D. When the actuator 16 is actuated, the intermediate presser member 19 moves in the Z-axis direction in conjunction with the needle bar 12. The intermediate presser member 19 prevents the sewing object S from floating up due to the movement of the sewing needle 3.

[0027] The operating device 20 is operated by an operator. The sewing machine 1 is operated by operating the operating device 20. In this embodiment, the operating device 20 includes an operating panel 21 and an operating pedal 22. The operating panel 21 is mounted on the upper surface of the table 2. The operating pedal 22 is disposed below the table 2. The operator operates the operating pedal 22 with his / her foot. The sewing machine 1 is operated by the operator operating at least one of the operating panel 21 and the operating pedal 22.

[0028] The imaging device 30 captures an image of the sewing object S held by the holding member 15. The imaging device 30 has an optical system and an image sensor that receives light incident via the optical system. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0029] The imaging device 30 is disposed above the needle plate 13 and the holding member 15. The imaging position Pf includes the position of the optical axis AX of the optical system of the imaging device 30. An imaging area FA is defined in the imaging device 30. The imaging area FA includes the field of view of the optical system of the imaging device 30. The imaging area FA includes the imaging position Pf. The imaging device 30 captures an image of at least a portion of the sewing object S disposed in the imaging area FA. The imaging device 30 captures an image of at least a portion of the sewing object S disposed inside the presser member 15A from above.

[0030] The position of the imaging device 30 is fixed. The relative position of the imaging device 30 and the sewing machine frame 11 is fixed. The relative position of the optical axis AX of the optical system of the imaging device 30 and the sewing needle 3 in the XY plane is fixed. Relative position data indicating the relative position of the optical axis AX of the optical system of the imaging device 30 and the sewing needle 3 in the XY plane is known data that can be derived from the design data of the sewing machine 1.

[0031] The position of the image captured by the imaging device 30 is defined in the camera coordinate system. The position of the image defined in the camera coordinate system is transformed into the position of the image defined in the sewing machine coordinate system by a defined transformation formula or transformation matrix.

[0032] In addition, if a difference occurs between the actual position of the imaging device 30 and its position in the design data due to an installation error of the imaging device 30, after installing the imaging device 30, the position of the sewing needle 3 in the XY plane is measured, and the measured position of the sewing needle 3 is moved toward the imaging device 30 by a known data amount, and the difference between the actual position of the imaging device 30 in the XY plane and the position of the sewing needle 3 after the movement is calculated, and based on this difference, the accurate relative position of the optical axis AX of the optical system of the imaging device 30 and the sewing needle 3 can be calculated.

[0033] The imaging device 30 captures an image of the sewing object S and outputs an object image SM showing an image of the sewing object S to the control device 40. In this embodiment, the object image SM includes at least an image of the sewing object S after the sewing process. Note that the object image SM may also include an image of the sewing object S before the sewing process. In this embodiment, the imaging device 30 captures an image of the sewing object S before the sewing process. Moreover, the imaging device 30 captures an image of the sewing object S after the sewing process.

[0034] The imaging device 30 captures an image of an imaging area FA including a correction point CP set on a target stitch line RL as shown in Fig. 8. The imaging device 30 sequentially captures images of the imaging areas FA corresponding to the correction points CP set on the target stitch line RL for which the next sewing process is to be performed, and outputs an object image SM to the control device 40.

[0035] [Items to be sewn] FIG. 3 is a cross-sectional view showing a portion of the sewing object S according to the present embodiment. FIG. 4 is a plan view showing a portion of the sewing object S according to the present embodiment. FIG. 5 is a cross-sectional view showing a portion of the sewing object S according to the present embodiment. FIG. 6 is a plan view showing a portion of the sewing object S according to the present embodiment. FIGS. 3 and 4 show the sewing object S before the sewing process. FIGS. 5 and 6 show the sewing object S after the sewing process. In the present embodiment, the sewing object S is a cover material used for a vehicle seat.

[0036] 3, the sewing object S has a surface material 4, a padding material 5, and a backing material 6. A hole 7 is provided in the surface material 4. The hole 7 may be provided penetrating not only the surface material 4 but also the padding material 5 and the backing material 6.

[0037] The surface of the cover material 4 is a seating surface that comes into contact with an occupant when the occupant sits on the vehicle seat. The cover material 4 includes at least one of a woven fabric, a nonwoven fabric, and leather. The pad material 5 has elasticity. The pad material 5 includes, for example, a urethane resin. The backing material 6 includes at least one of a woven fabric, a nonwoven fabric, and leather.

[0038] As shown in Fig. 4, a plurality of holes 7 are provided in the surface material 4. The material 4 includes a texture area TA, which is an area on the surface where the holes 7 are formed at approximately equal intervals, and a stitch area SA, which is an area on the surface where the holes 7 are not formed. A target stitch line RL of a stitch CH to be formed on the sewing object S is defined in the stitch area SA. The target stitch line RL is stored in a sewing data storage unit 51 of the storage device 50, which will be described later.

[0039] [Displacement of the surface of the sewing object] The sewing object S is thick and elastic. When a stitch CH is formed on the sewing object S, which is thick and elastic, the sewing object S is crushed and shrunk in the Z-axis direction at the position where the stitch CH is formed, as shown in Fig. 5. When the sewing object S shrinks, the surface of the sewing object S is pulled and displaced. When a stitch CH is formed on the sewing object S, the surface of the sewing object S is displaced in the XY plane with respect to the target stitch line RL, as shown in Fig. 6.

[0040] If the surface of the sewing object S is displaced within the XY plane relative to the target stitch line RL, it becomes difficult to form the stitch CH at the desired position on the surface of the sewing object S when the holding member 15 is moved according to the target stitch line RL.

[0041] In this embodiment, when the sewing object S shrinks due to the formation of the stitch CH and the surface of the sewing object S is displaced, the position of the target stitch line RL is corrected based on the amount of displacement of the surface of the sewing object S. The holding member 15 is moved based on the corrected target stitch line RL.

[0042] [Sewing machine control configuration] 7 is a functional block diagram showing the sewing machine 1 according to the present embodiment. The sewing machine 1 includes a control device 40, a storage device 50, an input device 60, an output device 70, and an image processing device 100.

[0043] The control device 40 includes a computer system. As shown in Fig. 7, the control device 40 is connected to the actuator 16 that moves the sewing needle 3 in the Z-axis direction, the actuator 17 that moves the holding member 15 in the XY plane, the actuator 18 that moves the presser member 15A of the holding member 15 in the Z-axis direction, the operation device 20, the imaging device 30, the storage device 50, the input device 60, the output device 70, and the image processing device 100. The control device 40 is also connected to a drive amount sensor 31 that detects the drive amount of the actuator 16, and a drive amount sensor 32 that detects the drive amount of the actuator 17.

[0044] The control device 40 outputs a control command to control the actuator 17 that moves the holding member 15, based on the processing result of the image processing device 100. The drive amount sensor 31 includes an encoder that detects the amount of rotation of the pulse motor that is the actuator 16. Detection data of the drive amount sensor 31 is output to the control device 40.

[0045] The drive amount sensor 32 includes an X-axis sensor 32X that detects the amount of rotation of the X-axis motor 17X and a Y-axis sensor 32Y that detects the amount of rotation of the Y-axis motor 17Y. The X-axis sensor 32X includes an encoder that detects the amount of rotation of the X-axis motor 17X. The Y-axis sensor 32Y includes an encoder that detects the amount of rotation of the Y-axis motor 17Y. Detection data of the drive amount sensor 32 is output to the control device 40.

[0046] The drive amount sensor 32 functions as a position sensor that detects the position in the XY plane of the holding member 15. The drive amount of the actuator 17 and the movement amount of the holding member 15 correspond one-to-one.

[0047] The X-axis sensor 32X detects the amount of rotation of the X-axis motor 17X, thereby detecting the amount of movement of the holding member 15 in the X-axis direction from the origin in the sewing machine coordinate system. The Y-axis sensor 32Y detects the amount of rotation of the Y-axis motor 17Y, thereby detecting the amount of movement of the holding member 15 in the Y-axis direction from the origin in the sewing machine coordinate system.

[0048] The control device 40 controls the actuator 16 based on the detection data of the drive amount sensor 31. The control device 40 determines, for example, the operation timing of the actuator 16 based on the detection data of the drive amount sensor 31.

[0049] The control device 40 controls the actuator 17 based on the detection data of the drive amount sensor 32. The control device 40 feedback-controls the actuator 17 based on the detection data of the drive amount sensor 32 so that the holding member 15 moves to a desired position.

[0050] The control device 40 calculates the position of the holding member 15 in the XY plane based on the detection data of the drive amount sensor 32. The amount of movement of the holding member 15 from the origin in the XY plane is detected based on the detection data of the drive amount sensor 32. The control device 40 calculates the position of the holding member 15 in the XY plane based on the detected amount of movement of the holding member 15.

[0051] The control device 40 further includes an object image acquisition unit 41 , a correction point setting unit 42 , an object information output unit 43 , a correction point information acquisition unit 44 , a target stitch line correction unit 45 , and a needle drop point correction unit 46 .

[0052] The object image acquisition unit 41 acquires an object image SM indicating an image related to the sewing object S. The object image acquisition unit 41 acquires an object image SM from the imaging device 30 .

[0053] In this embodiment, the object image acquisition unit 41 acquires an object image SM captured before the sewing process from the imaging device 30. When the object image SM captured by the imaging device 30 before the sewing process is identical to the design data of the sewing object S in the initial state stored in the design data storage unit 52, the object image acquisition unit 41 may acquire the object image SM before the sewing process from the design data storage unit 52 of the sewing object S in the initial state.

[0054] The correction point setting unit 42 sets a correction point CP that is set at an arbitrary position in order to correct a target stitch line RL defined in the stitch area SA. The correction point CP is set with respect to the sewing object S before the sewing process. When a stitch CH is formed, the surface of the sewing object S is displaced with respect to the correction point CP and the target stitch line RL. Therefore, after a stitch CH is formed on one target stitch line RL, the position of the target stitch line RL to be sewn next is corrected based on the position of the correction point CP set on the target stitch line RL to be sewn next.

[0055] FIG. 8 is a diagram for explaining the correction points CP according to this embodiment. FIG. 9 is a diagram for explaining the interpolation points IP according to this embodiment. As shown in FIG. 8, the correction points CP are set on the sewing object S. The correction points CP are used to correct the target stitch lines RL. The correction points CP are set in the stitch area SA so as to overlap with the target stitch lines RL to be corrected. The positions of the correction points CP are defined in the sewing machine coordinate system. The correction points CP include a feature point FP and an interpolation point IP.

[0056] The characteristic point FP is, for example, a position where the target stitch line RL intersects with the stitch area SA, and indicates a correction point CP having a characteristic shape, such as an X-shape, and having a two-dimensional positioning element. In addition, the characteristic point FP is set at both ends of the target stitch line RL. The correction point setting unit 42 sets the position of the characteristic point FP on the sewing object S before the sewing process, together with information that it is the characteristic point FP.

[0057] The interpolation point IP indicates a correction point CP that has no positioning element in the sewing direction (e.g., see the sewing direction SD in FIG. 9), which is a direction along the target stitch line RL, such as a straight line or a gentle curved section, but has a positioning element in the cross direction (e.g., see the cross direction CD in FIG. 13), which crosses the sewing direction, that is, has only a one-dimensional positioning element. In this embodiment, the positioning element includes a hole pattern consisting of a plurality of holes 7. Specifically, in the cross direction crossing the sewing direction, the hole pattern is present on both sides of the stitch area SA where no holes 7 are formed, and the pattern is not uniform, so positioning is possible. On the other hand, in the sewing direction, similar hole patterns continue, so positioning is difficult. The correction point setting unit 42 sets the position of the interpolation point IP on the sewing object S before the sewing process together with data on the sewing direction SD for the interpolation point IP and information that it is an interpolation point IP. The sewing direction SD is a tangential direction on the interpolation point IP with respect to the target stitch line RL.

[0058] The correction point CP is set at a desired position by the operator. The operator can set the correction point CP at a desired position on the sewing object S by operating the input device 60. The correction point information acquisition unit 44 sets the correction point CP based on input data generated by operating the input device 60. The correction point CP may be set automatically based on the design data. In addition, after the operator sets the characteristic point FP at a desired position, the interpolation point IP may be set based on the design data.

[0059] The correction point CP set by the operator or automatically is stored as correction point data in a correction point data storage unit 53 of the storage device 50, which will be described later. In the sewing process, which will be described later, in order to calculate the position after the displacement of the correction point CP of the target stitch line RL where the next sewing process is to be performed of the sewing object S, whose surface has been displaced by sewing, the imaging device 30 acquires an object image SM of the imaging area FA including the correction point CP set by the correction point setting unit 42.

[0060] Before the sewing process, the object image acquisition unit 41 acquires an object image SM of an imaging area FA including the correction points CP set by the correction point setting unit 42. More specifically, the object image acquisition unit 41 sequentially acquires object images SM of a plurality of imaging areas FA corresponding to a plurality of correction points CP set on a target stitch line RL for performing the next sewing process. The object image acquisition unit 41 converts the position of the object image SM defined in the camera coordinate system into the position of the object image SM defined in the sewing machine coordinate system by a predetermined conversion formula or conversion matrix.

[0061] The object information output unit 43 outputs object information including the object image SM acquired by the object image acquisition unit 41 to the image processing device 100. The object information has coordinate data of the imaging area FA. The object information is linked to correction point prior information indicating information of the correction points CP included in the object image SM. The correction point prior information includes an identifier for identifying each of the multiple correction points CP.

[0062] The correction point prior information includes information on whether the correction point CP is a characteristic point FP or an interpolation point IP. If the correction point CP is a characteristic point FP, the correction point prior information includes position data of the characteristic point FP. If the correction point CP is an interpolation point IP, the correction point prior information includes position data of the interpolation point IP and data of the sewing direction SD.

[0063] Here, the positions of the feature point FP and the interpolation point IP indicate the positions before being displaced by the sewing process. Therefore, for example, when the target stitch line RL is set at the width center of the stitch area SA, the position of the correction point CP in the object image SM output by the object information output unit 43 is assumed to be shifted from the width center of the stitch area SA.

[0064] The correction point data may also include arrangement data of the holes 7 in the texture area TA. For example, the correction point data may include data on the relative position of the hole 7A closest to the interpolation point IP with respect to the interpolation point IP, and the hole distance DH between the hole 7A and the hole 7B closest to the hole 7A. The position of the hole 7A and the hole distance DH may be information set by the operator, or may be extracted from the object image SM by the image processing device 100 when the object image SM is acquired before the sewing process.

[0065] The correction point information acquisition unit 44 acquires the correction point information output from the image processing device 100. The correction point information acquisition unit 44 acquires the correction point information corresponding to the object information including the object image SM output by the object information output unit 43. The correction point information includes an identifier for identifying each of the multiple correction points CP, and displacement amount data of the correction points CP displaced by the sewing process. The correction point information acquisition unit 44 acquires the displacement amount data of the correction points CP including all the feature points FP and the interpolation points IP corresponding to the target stitch line RL for which the sewing process is to be performed next.

[0066] The target stitch line correction unit 45 corrects the position of the target stitch line RL based on the correction point information of the correction point CP acquired by the correction point information acquisition unit 44. The target stitch line RL defined by the set correction point CP and the sewing data is displaced due to displacement of the surface of the sewing object S caused by the sewing process. The displacement amount of the correction point CP is calculated by the image processing device 100, which will be described later. The target stitch line correction unit 45 corrects the target stitch line RL based on the calculated displacement amount of the correction point CP.

[0067] 10 is a diagram for explaining a method for correcting the target stitch line RL according to the present embodiment. The target stitch line correcting unit 45 displaces the target stitch line RL by, for example, the same amount of displacement as the amount of displacement of the correction point CP.

[0068] The target stitch line correction unit 45 displaces, deforms and rotates the target stitch line RL based on the displacement, transformation and rotation amounts of the vector connecting the correction points CP, using two adjacent correction points CP as references. For example, as shown in Fig. 10, a pre-sewing vector SV11 connecting the correction points CP1 and CP2 changes to a post-sewing vector SV12 due to the displacement of the surface of the sewing object S. The target stitch line RL between the correction points CP1 and CP2 is deformed and rotated by the same deformation and rotation amounts as the deformation and rotation amounts from the pre-sewing vector SV11 to the post-sewing vector SV12.

[0069] Furthermore, the pre-sewing vector SV21 connecting the correction points CP2 and CP3 changes to a post-sewing vector SV22 due to the displacement of the surface of the sewing object S. The target stitch line RL between the correction points CP2 and CP3 is displaced, deformed and rotated by the same displacement, deformation and rotation amounts as the displacement, deformation and rotation amounts from the pre-sewing vector SV21 to the post-sewing vector SV22. Similarly, the target stitch line RL between the correction points CP3 and CP4 and the target stitch line RL between the correction points CP4 and CP5 are also displaced, deformed and rotated. As a result, the corrected target stitch line RL is obtained.

[0070] The needle drop point correction unit 46 corrects the position where the sewing needle 3 is lowered when forming the stitch CH on the target stitch line RL. When the target stitch line correction unit 45 corrects the target stitch line RL, the length of the target stitch line RL between the correction points CP changes. In addition, the rate of change of the length of the target stitch line RL between different correction points CP may vary. This causes variation in the intervals between the needle drop points, which are the positions where the sewing needle 3 is lowered when forming the stitch CH on the target stitch line RL.

[0071] The needle drop point correction unit 46 recalculates the needle drop points for one target stitch line RL corrected by the target stitch line correction unit 45 so that the interval between the needle drop points is fixed and the number of needle drop points is variable.

[0072] The storage device 50 includes a non-volatile memory such as a read only memory (ROM) or storage, and a volatile memory such as a random access memory (RAM). As shown in Fig. 7, the storage device 50 is connected to the control device 40. The storage device 50 has a sewing data storage unit 51, a design data storage unit 52, a correction point data storage unit 53, and a program storage unit 54.

[0073] The sewing data storage unit 51 stores sewing data to be referenced in the sewing process. The sewing process refers to a process for forming a stitch CH (see FIG. 6) on the sewing object S. The sewing data includes a target stitch line RL (see FIGS. 4 and 6) of the stitch CH to be formed on the sewing object S, and movement conditions of the holding member 15. The target stitch line RL specifies a target shape of the stitch CH to be formed on the sewing object S and a target position of the stitch CH in the sewing machine coordinate system.

[0074] A plurality of target stitch lines RL for forming a plurality of stitches CH are defined for the sewing object S. The sewing process includes a plurality of sewing processes corresponding to the plurality of stitches CH. A portion of the sewing object S shown in FIG. 4 includes a first target stitch line RL1, a second target stitch line RL2, a third target stitch line RL3, and a fourth target stitch line RL4. In the state shown in FIG. 6, a stitch CH is formed on the first target stitch line RL1.

[0075] The movement conditions of the holding member 15 include a movement trajectory of the holding member 15 that is defined in the sewing machine coordinate system. The movement trajectory of the holding member 15 includes a movement trajectory of the holding member 15 in the XY plane. The movement conditions of the holding member 15 are determined based on the target stitch line RL.

[0076] For example, the first sewing process includes a process of forming a first stitch CH1 on the sewing object S based on a first target stitch line RL1. The first sewing process is first performed after the sewing object S is held by the holding member 15. The first sewing process is followed by a second sewing process of forming a second stitch on the sewing object S based on a second target stitch line RL2. Similarly, a third sewing process, a fourth sewing process, and a sewing process of forming a stitch CH for the corresponding target stitch line RL are sequentially performed.

[0077] The design data storage unit 52 stores design data of the sewing object S. The design data of the sewing object S includes the position and range of a texture area TA and the position and range of a stitch area SA on the surface of the sewing object S. When the sewing object S is designed by CAD (Computer Aided Design), the design data of the sewing object S includes CAD data.

[0078] The design data of the sewing object S is design data of the sewing object S in an initial state. The initial state of the sewing object S refers to the state before the first sewing process. In other words, the initial state of the sewing object S refers to a state in which the stitch CH has not yet been formed on the sewing object S.

[0079] The correction point data storage unit 53 stores correction point data, which is data on correction points CP for correcting a target stitch line RL defined in the stitch area SA. The correction point data includes data on the positions of feature points FP, and data on the positions of interpolation points IP and sewing directions SD. The correction point data may also include arrangement data of holes 7 in the texture area TA. For example, the correction point data may include data on the relative position of hole 7A closest to the interpolation point IP with respect to the interpolation point IP, and the inter-hole distance DH between hole 7A and hole 7B closest to hole 7A.

[0080] The correction point data storage unit 53 stores data of the correction point CP input by the operator through the input device 60. The correction point data storage unit 53 also updates and stores data of the correction point CP acquired by the correction point information acquisition unit 44 of the control device 40. The data of the correction point CP acquired by the correction point information acquisition unit 44 is data of the correction point CP that has moved within the sewing machine coordinate system as a result of a stitch CH being formed on the target stitch line RL immediately preceding the target stitch line RL on which the correction point CP is set, thereby deforming the sewing object S.

[0081] The program storage unit 54 stores a computer program for controlling the sewing machine 1. The computer program is read into the control device 40. The control device 40 controls the sewing machine 1 in accordance with the computer program stored in the program storage unit 54.

[0082] The input device 60 accepts various input operations by an operator. Input data is generated by operating the input device 60. Examples of the input device 60 include a computer keyboard, a mouse, operation keys, a microphone that accepts voice input, and a touch panel.

[0083] The output device 70 outputs output data. Examples of the output device 70 include a display device and a printing device. The display device outputs display data as the output data. The printing device outputs print data as the output data. Examples of the display device include a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OELD). Examples of the printing device include an inkjet printer.

[0084] The image processing device 100 includes a computer system. As shown in Fig. 7, the image processing device 100 is connected to a control device 40. The image processing device 100 processes an image related to a sewing object S. The image processing device 100 includes a learning model storage unit 101, an object information acquisition unit 102, a boundary extraction unit 103, a feature point calculation unit 104, a boundary point calculation unit 105, a midpoint calculation unit 106, an interpolation point calculation unit 107, and a correction point information output unit 108.

[0085] The learning model storage unit 101 stores a matching model that captures the hole pattern on the surface of the sewing object S before sewing as two-dimensional data such as image data, or a learning model that is generated by machine learning.

[0086] The matching model can be generated from one or more images and does not require a classifier. The matching model may use the image data itself as matching data, or may use two or more characteristic shape attributes (e.g., perforation holes, etc.) and their representative coordinates (e.g., center or center of gravity, etc.) extracted in advance from the image data as matching data.

[0087] In this embodiment, a case will be described in which learning data generated by machine learning is used. The learning model generated by machine learning includes a classifier for discriminating between the texture region TA and the stitch region SA.

[0088] 11 is a diagram for explaining the learning process according to this embodiment. A large number of learning images SW are collected as learning data. The learning images SW represent learning data related to an object image SM after sewing processing. Boundary data BW indicating a boundary line BL between a texture area TA and a stitch area SA is assigned to each of the learning images SW.

[0089] A learning model is generated by machine learning the learning image SW to which the boundary data BW, which is the teacher data, is added using a specified machine learning algorithm. As the machine learning algorithm, at least one of a decision tree, a random forest, and a neural network is exemplified.

[0090] The object information acquisition unit 102 acquires object information including an object image SM output by the object information output unit 43 of the control device 40. The object information is linked to prior correction point information indicating information of the correction point CP included in the object image SM. The object image SM acquired by the object information acquisition unit 102 is an image after the surface of the sewing object S has been displaced by the sewing process. The prior correction point information acquired by the object information acquisition unit 102 is information of the correction point CP before the surface of the sewing object S is displaced by the sewing process. Therefore, a discrepancy occurs between the position of the correction point CP after the sewing process and the acquired position data of the correction point CP.

[0091] FIG. 12 is a diagram for explaining the reference point RP according to this embodiment. FIG. 9 and FIG. 12 are object images SM in the same imaging area FA. FIG. 9 is the object image SM before the sewing process. FIG. 12 is the object image SM after the surface of the sewing object S is displaced by the sewing process. The reference point RP is a point that indicates the position of the midpoint CP before the surface of the sewing object S is displaced by the sewing process. The position data of the correction point CP acquired by the object information acquisition unit 102 is treated as the position data of the reference point RP. The image processing device 100 calculates the correction point CP after the displacement, and outputs the displacement from the reference point RP to the correction point CP.

[0092] The boundary extraction unit 103 divides the surface of the sewing object S into a texture area TA and a stitch area SA based on the object image SM after the sewing process acquired by the object information acquisition unit 102 and the learning model stored in the learning model memory unit 101, and extracts the boundary line BL between the texture area TA and the stitch area SA.

[0093] The boundary extraction unit 103 does not necessarily need to capture all of the contour edges of the boundary line BL, but only needs to extract the center of gravity of the hole 7 on or near the boundary line BL. By performing a matching process using image data including the texture area TA and stitch area SA, centered on the reference point RP, as a template, or by performing a matching process using the center of gravity coordinate arrangement pattern (coordinate point sequence data) of the hole 7 as a template, the positional relationship between the boundary line BL and the reference point RP can be determined, and the center of gravity sequence of the hole 7 near the boundary line BL can be extracted.

[0094] 13, which will be described later, the boundary line BL is a line passing through the edge of the texture area TA. When the texture area TA includes a hole 7, the boundary line BL is generated so as to connect the outer edge of the hole 7 located at the outermost position in the texture area TA, or the center of gravity of the hole 7. The boundary line extraction unit 103 extracts the boundary line BL from both the object image SM including the feature point FP shown in FIG. 8 and the object image SM including the interpolation point IP shown in FIG. 8.

[0095] When the boundary line extraction unit 103 extracts a boundary line BL from the object image SM including the feature point FP, the feature point calculation unit 104 calculates the position of the feature point FP based on the boundary line BL. As described in Patent Document 2, for example, the feature point calculation unit 104 calculates the feature point FP based on a plurality of points such as corners, maximum points, minimum points, and inflection points on the boundary line BL, or the center of gravity of the hole 7, from their mutual positional relationships.

[0096] When the boundary extraction unit 103 extracts a boundary line BL from the object image SM including the interpolation points IP, the boundary point calculation unit 105 calculates boundary points BP indicating both ends of the width direction of the stitch area SA that passes through the reference point RP based on the boundary line BL. The width direction of the stitch area SA is a direction perpendicular to the sewing direction SD. The interpolation points IP are set in a straight line or gently curved stitch area SA. Therefore, the boundary point calculation unit 105 can obtain and fix the width direction of the stitch area SA, and calculates two boundary points BP on the boundary line BL that are both ends of the stitch area SA.

[0097] 13 is a diagram for explaining a method for calculating the boundary point BP according to this embodiment. The sewing direction SD is associated with the correction point advance information acquired by the object information acquisition unit 102. The surface of the sewing object S is displaced in the cross direction CD that is generally perpendicular to the sewing direction SD by the sewing process. Therefore, first, the displacement amount of the correction point CP in the cross direction CD is calculated. The cross direction CD is the width direction of the stitch area SA.

[0098] 13, the boundary point calculation unit 105 first calculates a straight line that passes through the reference point BP and is parallel to the cross direction CD perpendicular to the sewing direction SD, based on the position of the reference point RP, which is the position of the correction point CP acquired by the object information acquisition unit 102, and the sewing direction SD. The boundary point calculation unit 105 then calculates the position where the straight line intersects with the boundary line BL as the boundary point BP.

[0099] The significance of the boundary point calculation unit 105 calculating the boundary point BP is to calculate the scale variation and angle variation of the object image SW. There is no limitation to the method of calculating the boundary point BP, and for example, when a matching process based on image data or a coordinate point sequence is used to grasp the positional relationship between the boundary line BL and the reference point RP, the scale variation and angle variation of the object image SW can be calculated by adding the scale and angle to the variation parameters.

[0100] The midpoint calculation unit 106 calculates a midpoint HP that indicates the midpoint in the width direction of the stitch area SA that passes through the reference point RP.

[0101] FIG. 14 is a diagram for explaining a method of calculating the midpoint HP according to this embodiment. The width direction of the stitch area SA that passes through the reference point RP is the cross direction CD. In addition, both ends of the width direction of the stitch area SA are the two boundary points BP calculated by the boundary point calculation unit 105. Therefore, the midpoint HP indicates the midpoint of the two boundary points BP calculated by the boundary point calculation unit 105. The midpoint calculation unit 106 calculates the midpoint HP based on the two boundary points BP calculated by the boundary point calculation unit 105. Alternatively, when the positional relationship between the boundary line BL and the reference point RP is calculated by a matching process, the midpoint HP is calculated based on the reference point RP and the scale variation and angle variation of the object image SW.

[0102] Since the target stitch line RL passes through the center of the stitch area SA in the width direction, in the present invention, the midpoint HP may be regarded as the interpolation point IP after the sewing process. In other words, the interpolation point IP may be calculated only by correcting the cross direction CD. For example, if the interpolation points IP are densely set in the curved portion of the target stitch line RL, the target stitch line RL can be sufficiently corrected by the characteristic point FP and the midpoint HP. However, if more precise correction is required, the sewing direction SD is also corrected.

[0103] The interpolation point calculation unit 107 calculates the interpolation point IP after the sewing process. The interpolation point calculation unit 107 calculates the displacement in the sewing direction SD due to the sewing process with respect to the midpoint HP calculated by the midpoint calculation unit 106. The interpolation point calculation unit 107 calculates the displacement in the sewing direction SD based on the perforation pattern of the texture area TA, i.e., the displacement of the hole 7.

[0104] 15 is a diagram for explaining an example of a calculation method of the interpolation point IP according to this embodiment. The interpolation point calculation unit 107 first calculates the position of the reference hole 7R based on the relative position of the hole 7A (see FIG. 9) with respect to the interpolation point IP acquired by the object information acquisition unit 102. The relative position of the reference hole 7R with respect to the intermediate point HP is the same as the relative position of the hole 7A with respect to the correction point CP.

[0105] The interpolation point calculation unit 107 then searches for the hole 7A in the cross direction CD relative to the reference hole 7R. The search method for the hole 7A may be a known method such as pattern matching. The search range may be set based on the inter-hole distance DH (see FIG. 9) between the hole 7A and the hole 7B closest to the hole 7A. Specifically, for example, the search range in the cross direction CD is set to a distance of 1 / 2 the inter-hole distance DH with the reference hole 7R as the center.

[0106] The interpolation point calculation unit 107 calculates the distance to the detected hole 7A relative to the reference hole 7R. The distance to the detected hole 7A relative to the reference hole 7R is the displacement of the interpolation point IP in the cross direction CD. The interpolation point calculation unit 107 calculates the interpolation point IP based on the intermediate point HP and the displacement of the interpolation point IP in the cross direction CD.

[0107] In addition, when the calculation of the boundary line BL and boundary point BP is performed by a matching process using image data or coordinate point sequence data, the fluctuation area of ​​the interpolation point IP is predicted from the cross direction CD, the hole distance DH, the maximum scale, and the angle fluctuation amount, and a search area is set, so that the interpolation point IP can be calculated by the matching process alone, instead of processing by the boundary line extraction unit 103, the feature point calculation unit 104, the boundary point calculation unit 105, the midpoint calculation unit 106, and the interpolation point calculation unit 107.

[0108] The correction point information output unit 108 outputs correction point information indicating information of new correction points CP including the feature points FP calculated by the feature point calculation unit 104 and the interpolation points IP calculated by the interpolation point calculation unit 107 to the control device 40. The correction point information includes identifiers for identifying each of the multiple correction points CP. The correction point information includes displacement amount data of the newly calculated correction points CP with respect to the correction points CP acquired by the object information acquisition unit 102.

[0109] [Control device processing] Fig. 16 is a flow chart showing the sewing method according to this embodiment. In this embodiment, the sewing method includes alignment processing S10, object image acquisition processing S11, object information output processing S12, correction point information acquisition processing S13, target stitch line correction processing S14, needle drop point correction processing S15, sewing processing S16, and end determination processing S17. The processing shown in Fig. 16 is executed by the control device 40.

[0110] The alignment process S10 is a process for associating the texture area TA and stitch area SA of the sewing object S held by the holding member 15 with the sewing machine coordinate system. After the sewing object S before the sewing process is held by the holding member 15, the imaging device 30 images the sewing object S. The imaging device 30 images, for example, a plurality of characteristic points FP and correction points CP including the interpolation points IP of the sewing object S. Note that, if an alignment mark is provided on the sewing object S, the imaging device 30 may image the alignment mark. The position of the image of the sewing object S imaged by the imaging device 30 is defined in the camera coordinate system. The position of the image defined in the camera coordinate system is converted to the position of the image defined in the sewing machine coordinate system by a defined transformation formula or transformation matrix. As a result, the position of the texture area TA and the position of the stitch area SA of the sewing object S are defined in the sewing machine coordinate system.

[0111] The object image acquisition process S11 is a process for acquiring an object image SM. Before the first sewing process, the object image acquisition unit 41 may acquire the object image SM in an initial state from the imaging device 30, or may acquire the object image SM in an initial state from the design data storage unit 52.

[0112] After the first sewing process, the object image SM is captured by the imaging device 30. The imaging device 30 captures the object image SM after the first sewing process and before the second sewing process. Similarly, the imaging device 30 captures the object image SM between the second sewing process and the final sewing process.

[0113] The object information output process S12 is a process of outputting object information including the object image SM acquired by the object image acquisition process S11 to the image processing device 100. The object information output unit 43 outputs the object information including the object image SM acquired by the object image acquisition unit 41 to the image processing device 100. In the object information output process S12, the object images SM corresponding to one target stitch line RL may be output together, or each object image SM may be output.

[0114] The correction point information acquisition process S13 is a process for acquiring correction point information output from the image processing device 100. The correction point information acquisition unit 44 acquires correction point information corresponding to object information including the object image SM output by the object information output unit 43. After the object information corresponding to one target stitch line RL is output in the object information output process S12, the control device 40 enters a standby state for the correction point information acquisition process S13 until the correction point information corresponding to the target stitch line RL is acquired.

[0115] The target stitch line correction process S14 is a process for correcting the target stitch line RL based on the correction point CP acquired by the correction point information acquisition process S13. The target stitch line RL, which is defined by the correction point CP set by the operator and the sewing data, is displaced due to the displacement of the surface of the sewing object S caused by the sewing process. The displacement amount of the correction point CP is calculated by the image processing device 100 and acquired by the correction point information acquisition process S13. The target stitch line correction unit 45 corrects the target stitch line RL based on the displacement amount of the correction point CP acquired from the image processing device 100. The target stitch line correction unit 45 displaces the target stitch line RL by, for example, the same displacement amount as the displacement amount of the correction point CP.

[0116] A plurality of correction points CP are set for the target stitch line RL. The target stitch line RL is corrected based on the respective displacement amounts of the plurality of correction points CP. The target stitch line RL is corrected by displacing, deforming and rotating based on the displacement amounts, deformation amounts and rotation amounts of the vector connecting the correction points CP with respect to two adjacent correction points CP. The position of the target stitch line RL in the sewing machine coordinate system is corrected.

[0117] The target stitch line correction process S14 is not performed before the first sewing process. The first sewing process is performed based on the initial target stitch line RL defined by the sewing data. The target stitch line correction process S14 is performed after the first sewing process and before the second sewing process. Similarly, the target stitch line correction process S14 is performed between the second sewing process and the final sewing process.

[0118] The needle point correction process S15 is a process for correcting the position where the sewing machine needle 3 is lowered when forming a stitch CH on the target stitch line RL. The target stitch line RL between adjacent correction points CP is corrected by the target stitch line correction process S14. This causes variation in the intervals between the needle points set on the target stitch line RL between different correction points CP. The needle point correction unit 46 recalculates the needle points for one target stitch line RL after correction by the target stitch line correction process S14 so that the intervals between the needle points are fixed and the number of needle points varies.

[0119] The sewing process S16 is a process for forming a stitch CH based on a target stitch line RL. The first sewing process is performed based on the target stitch line RL in an initial state defined by the sewing data. The second sewing process to the final sewing process are performed based on the target stitch line RL corrected by the target stitch line correction process S14 and the needle drop points corrected by the needle drop point correction process S15. The control device 40 outputs a control command to the actuator 17 so that the stitch CH is formed according to the target stitch line RL.

[0120] The end determination process S17 is a process for determining whether or not the sewing process of the sewing object S has been completed. The control device 40 determines whether or not the sewing process of the sewing object S has been completed based on the sewing data. In a state where the sewing process from the first sewing process to the sewing process immediately before the final sewing process has been completed, the control device 40 determines in the end determination process S17 that the sewing process has not been completed. If the control device 40 determines that the sewing process has not been completed, the control device 40 proceeds to the object image acquisition process S11. In a state where the final sewing process has been completed, the control device 40 determines in the end determination process S17 that the sewing process has been completed.

[0121] [Image processing device processing] Fig. 17 is a flowchart showing a correction point calculation method according to this embodiment. In this embodiment, the correction point calculation method includes an object information acquisition process S21, a correction point determination process S22, a boundary line extraction process S23, a feature point calculation process S24, a boundary line extraction process S25, an interpolation point calculation process S26, and a correction point information output process S27. The processes shown in Fig. 17 are executed by the image processing device 100.

[0122] The object information acquisition process S21 is a process for acquiring object information including the object image SM output by the control device 40 in the object information output process S12 shown in Fig. 16. The image processing device 100 acquires the object information, thereby starting a series of processes shown in Fig. 17.

[0123] The correction point determination process S22 is a process for determining whether the correction point CP included in the object image SM acquired in the object information acquisition process S21 is a feature point FP or an interpolation point IP. The object information includes information as to whether the correction point CP is a feature point FP or an interpolation point IP. When object information including the object image SM of the feature point FP is acquired, the control device 40 determines that it is the feature point FP in the correction point determination process S22, and proceeds to a boundary extraction process S23. When object information including the object image SM of the interpolation point IR is acquired, the control device 40 determines that it is the interpolation point IR in the correction point determination process S22, and proceeds to a boundary extraction process S25.

[0124] The boundary extraction process S23 is a process for dividing the surface of the sewing object S into a texture area TA and a stitch area SA based on the object image SM including the feature point FP and a learning model, and for calculating the boundary line BL between the texture area TA and the stitch area SA. The boundary extraction process S23 is performed sequentially on the object images SM including the feature point FP among the object images SM acquired in the object information acquisition process S21.

[0125] In the boundary extraction process S23, the boundary extraction unit 103 acquires a learning model from the learning model storage unit 101. Based on the object image SM including the feature points FP acquired in the object information acquisition process S21 and the acquired learning model, the boundary extraction unit 103 divides the surface of the sewing object S into a texture area TA and a stitch area SA, and calculates a boundary line BL between the texture area TA and the stitch area SA.

[0126] The feature point calculation process S24 is a process for calculating the position of the feature point FP after the sewing process based on the feature point FP set on the sewing object S before the sewing process and the boundary line BL extracted in the boundary line extraction process S23. The feature point calculation process S24 is performed, for example, in a procedure referenced by the correction point calculation process S3 described in Patent Document 2.

[0127] The boundary extraction process S25 is a process for dividing the surface of the sewing object S into a texture area TA and a stitch area SA based on the object image SM including the interpolation points IP and a learning model, and for calculating the boundary line BL between the texture area TA and the stitch area SA. The boundary extraction process S25 is performed sequentially on the object images SM including the interpolation points IP among the object images SM acquired in the object information acquisition process S21.

[0128] In the boundary extraction process S25, the boundary extraction unit 103 acquires a learning model from the learning model storage unit 101. Based on the object image SM including the interpolation points IP acquired in the object information acquisition process S21 and the acquired learning model, the boundary extraction unit 103 divides the surface of the sewing object S into a texture area TA and a stitch area SA, and calculates a boundary line BL between the texture area TA and the stitch area SA.

[0129] The interpolation point calculation process S26 is a process for calculating the position of the interpolation point IR after the sewing process, based on the interpolation point IR set on the sewing object S before the sewing process and the boundary line BL extracted in the boundary line extraction process S25. In the interpolation point calculation process S26 of the present embodiment, two boundary points BP and a midpoint HP which is the center between them are calculated based on the boundary line BL, and the interpolation point IP is calculated based on the deviation amount of the hole 7 near the interpolation point IP and the midpoint HP.

[0130] In the interpolation point calculation process S26, the boundary point calculation unit 105 calculates the cross direction CD, which is a direction perpendicular to the sewing direction SD and passes through the reference point RP. The cross direction CD that passes through the reference point RP is the width direction of the stitch area SA. The boundary point calculation unit 105 calculates two boundary points BP that indicate both ends of the stitch area SA in the width direction based on the reference point RP and the cross direction CD.

[0131] Based on the two boundary points BP, the midpoint calculation unit 106 calculates a midpoint HP that indicates the center in the width direction of the stitch area SA that passes through the reference point RP. The midpoint HP calculated by the interpolation point calculation process S26 may be regarded as the interpolation point IP after the sewing process. In other words, the interpolation point IP may be calculated by only correcting the cross direction CD.

[0132] In the interpolation point calculation process S26 of this embodiment, the amount of deviation of the hole 7 near the intermediate point HP and the interpolation point IP and the interpolation point IP are calculated. That is, the sewing direction SD is also corrected. The interpolation point calculation unit 107 calculates the position of the reference hole 7R relative to the intermediate point HP based on the relative position of the hole 7A with respect to the interpolation point IP acquired in the object information acquisition process S21. The interpolation point calculation unit 107 searches for the hole 7A in the cross direction CD with respect to the reference hole 7R and calculates the distance to the detected hole 7A with respect to the reference hole 7R. The interpolation point calculation unit 107 calculates the interpolation point IP based on the intermediate point HP and the distance to the detected hole 7A with respect to the reference hole 7R, which indicates the amount of displacement of the interpolation point IP in the cross direction CD.

[0133] The correction point information output process S27 is a process for outputting, to the control device 40, correction point information indicating information on a new correction point CP including the feature point FP calculated by the feature point calculation process S24 and the interpolation point IR calculated by the interpolation point calculation process S26. The correction point information output process S27 outputs correction point information corresponding to the object information acquired by the object information acquisition process S21. The correction point information includes displacement amount data of the newly calculated correction point CP with respect to the prior correction point information of the correction point CP acquired by the object information acquisition process S21.

[0134] [Computer System] FIG. 18 is a block diagram showing an example of a computer system 1000. Each of the above-mentioned control device 40 and image processing device 100 includes a computer system 1000. The computer system 1000 has a processor 1001 such as a CPU (Central Processing Unit), a main memory 1002 including a non-volatile memory such as a ROM and a volatile memory such as a RAM, a storage 1003, and an interface 1004 including an input / output circuit. The functions of the control device 40 and the functions of the image processing device 100 are stored in the storage 1003 as computer programs. The processor 1001 reads the computer program from the storage 1003, expands it in the main memory 1002, and executes the above-mentioned processing according to the computer program. The computer program may be distributed to the computer system 1000 via a network.

[0135] The computer program includes various application software for realizing the above-mentioned functions. The computer program can execute the following operations according to the above-mentioned embodiment: based on an object image SM showing an image related to the sewing object S, divide the surface of the sewing object S into a texture area TA and a stitch area SA, extract a boundary line BL between the texture area TA and the stitch area SA, acquire object information including the object image SM after the sewing process and correction point advance information showing information before the sewing process of a correction point CP for correcting a target stitch line RP defined in the stitch area SA, calculate a feature point FP after the sewing process based on the boundary line BL after the sewing process and the position data before the sewing process when the correction point advance information includes position data of a feature point FP having a two-dimensional positioning element, and calculate an interpolation point IP after the sewing process based on the boundary line BL after the sewing process and the position data and sewing direction data before the sewing process when the correction point advance information includes position data of an interpolation point IP having a one-dimensional positioning element and sewing direction data showing a sewing direction SD.

[0136] [effect] As described above, according to this embodiment, a boundary line BL between a texture area TA in which holes 7 are provided and a stitch area SA in which holes 7 are not provided is calculated on the surface of the sewing object S based on an object image SM of the sewing object S. Correction point prior information indicating information before sewing processing of correction points CP for correcting a target stitch line RL defined in the stitch area SA is acquired. The correction points CP include feature points FP having two-dimensional positioning elements and interpolation points IP having one-dimensional positioning elements but no two-dimensional positioning elements.

[0137] When the correction point CP is a characteristic point FP, the correction point prior information includes position data of the characteristic point FP. The characteristic point FP after the sewing process is calculated based on the boundary line BL after the sewing process and the position data of the characteristic point FP before the sewing process. When the correction point CP is an interpolation point IP, the correction point prior information includes position data of the interpolation point IP and sewing direction data indicating the sewing direction SD. The interpolation point IP after the sewing process is calculated based on the boundary line BL after the sewing process and the position data and sewing direction data of the interpolation point IP.

[0138] In other words, since the characteristic point FP has a two-dimensional positioning element, the position of the characteristic point FP after the sewing process can be estimated using the two-dimensionally characteristic boundary line BL, and the displacement amount of the characteristic point FP can be calculated based on the position data of the characteristic point FP before the sewing process and the estimated position of the characteristic point FP after the sewing process.

[0139] Furthermore, for the interpolation point IP that only has a one-dimensional positioning element, the displacement of the interpolation point IP in the cross direction CD is calculated by utilizing the fact that the sewing process displaces the surface of the sewing object S in the cross direction CD that is roughly perpendicular to the sewing direction SD. A one-dimensionally characteristic boundary line BL is a boundary line BL that is characteristic in the cross direction CD. Therefore, since the position of the interpolation point IP after the sewing process can be estimated by the boundary line BL at the position of the interpolation point IP before the sewing process in the cross direction CD, the displacement amount of the interpolation point IP can be calculated based on the position data of the interpolation point IP before the sewing process and the estimated position of the interpolation point IP after the sewing process.

[0140] This allows the amount of displacement of the correction point CP, which includes the feature point FP and the interpolation point IP, from before the sewing process to after the sewing process to be calculated. Calculating the amount of displacement of the correction point CP allows the displacement of the surface of the sewing object S caused by the sewing process to be properly recognized. Properly recognizing the displacement of the surface of the sewing object S allows the position of the target stitch line RL to be properly corrected even if the surface of the sewing object S is displaced. Therefore, the control device 40 can form the stitch CH based on the target stitch line RL.

[0141] [Other embodiments] Although the embodiments of the present application have been described above, the present invention is not limited by the contents of these embodiments. The above-described embodiments and modified examples can be appropriately combined within a range that does not cause contradictions in the processing contents. In addition, the above-described components include those that a person skilled in the art can easily imagine, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be appropriately combined. Furthermore, various omissions, substitutions, or modifications of the components can be made within a range that does not deviate from the gist of the above-described embodiments.

[0142] For example, in the above embodiment, when the object image SM is acquired by the imaging device 30, the holding member 15 that holds the sewing object S moves within the XY plane while the position of the imaging device 30 is fixed. With the position of the sewing object S fixed, the imaging area FA of the imaging device 30 may move within the XY plane, or both the imaging area FA and the sewing object S may move within the XY plane.

[0143] In the above embodiment, the boundary line BL is generated so as to pass through the outer edge of the hole 7 that is located on the outermost side in the texture area TA. The boundary line BL may be generated so as to pass through the hole 7 that is located on the outermost side in the texture area TA.

[0144] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0145] In addition, each component of each device shown in the figure is functionally conceptual, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. For example, the boundary point calculation unit 105 and the midpoint calculation unit 106 may be included in the interpolation point calculation unit 107, and the learning model storage unit 101 may have two learning models in which a two-dimensionally characteristic learning image SW in which a feature point FP is set and a one-dimensionally characteristic learning image SW in which an interpolation point IP is set are machine-learned, respectively. In addition, the above-mentioned control device 40 and image processing device 100 may be configured by a plurality of computers divided into several functions, and some functions of the computer may be possessed by a cloud server that executes various functions in the form of cloud computing.

[0146] Furthermore, the image processing device 100 may be configured as a computer that is completely independent of the sewing machine 1. The image processing device 100 may be connected to the control devices 40 of a plurality of sewing machines 1 and execute a process of calculating a correction point CP for the object image SM output from each of the control devices 40. The image processing device 100 may also be connected to a communication network including short-range wireless communication such as Bluetooth (registered trademark) or wireless LAN (Local Area Network), and connected to the sewing machine 1 via the communication network. [Explanation of symbols]

[0147] 1... sewing machine, 2... table, 3... sewing machine needle, 4... surface material, 5... padding material, 6... backing material, 7... hole, 10... sewing machine body, 11... sewing machine frame, 11A... horizontal arm, 11B... bed, 11C... vertical arm, 11D... head, 12... needle bar, 13... needle plate, 14... support member, 15... holding member, 15A... presser member, 15B... lower plate, 16... actuator, 17... actuator, 17X... X-axis motor, 17Y... Y-axis motor, 18... actuator, 19... intermediate presser member, 20... operation device, 21...operation panel, 22...operation pedal, 30...imaging device, 31...drive amount sensor, 32...drive amount sensor, 32X...X-axis sensor, 32Y...Y-axis sensor, 40...control device, 41...object image acquisition unit, 42...correction point setting unit, 43...object information output unit, 44...correction point information acquisition unit, 45...target stitch line correction unit, 46...needle drop point correction unit, 50...storage device, 51...sewing data storage unit, 52...design data storage unit, 53...correction point data storage unit, 54...program storage unit, 6 0...input device, 70...output device, 100...image processing device, 101...learning model storage unit, 102...object information acquisition unit, 103...boundary extraction unit, 104...feature point calculation unit, 105...boundary point calculation unit, 106...midpoint calculation unit, 107...interpolation point calculation unit, 108...correction point information output unit, 1000...computer system, 1001...processor, 1002...main memory, 1003...storage, 1004...interface, AX...optical axis, BL...boundary line, BP...boundary point, BW...boundary Boundary data, CD...crossing direction, CH...stitch, CP...correction point, DH...hole distance, FA...imaging area, FP...feature point, HP...midpoint, IP...interpolation point, Pf...imaging position, Ps...sewing position, RL...target stitch line, RP...reference point, S...sewing object, SA...stitch area, SD...sewing direction, SM...object image, SV11...before-sewing vector, SV12...after-sewing vector, SV21...before-sewing vector, SV22...after-sewing vector, SW...learning image, TA...texture area.

Claims

1. an object information acquiring unit that acquires object information including an object image showing an image of a sewing object after sewing processing and correction point advance information showing information before sewing processing of correction points for correcting a target stitch line defined in a stitch area, which is an area where stitches are formed; a feature point calculation unit that calculates the feature point after the sewing process based on the object image after the sewing process and the position data before the sewing process when the correction point advance information includes position data of a feature point having a two-dimensional positioning element; an interpolation point calculation unit that calculates the interpolation point after the sewing process based on the object image after the sewing process and the position data and the sewing direction data before the sewing process when the correction point advance information includes position data of an interpolation point having a one-dimensional positioning element and sewing direction data indicating a sewing direction, Image processing device.

2. a boundary extraction unit that divides a surface of the sewing object into a texture area and a stitch area based on an object image showing an image related to the sewing object, and extracts a boundary line between the texture area and the stitch area; the characteristic point calculation unit calculates the characteristic points after the sewing process based on the boundary line extracted from the object image after the sewing process and the position data before the sewing process; the interpolation point calculation unit calculates the interpolation point after the sewing process based on the boundary line extracted from the object image after the sewing process, the position data before the sewing process, and the sewing direction data; The image processing device according to claim 1 .

3. The interpolation point calculation unit Two boundary points are calculated, which indicate points where a straight line passing through the interpolation points and parallel to a cross direction perpendicular to the sewing direction intersects with the boundary line; Calculating a midpoint indicating the midpoint of the boundary line between the two points; calculating a displacement amount of the interpolation point in the intersecting direction based on the interpolation point and the intermediate point before the sewing process; The image processing device according to claim 2 .

4. The interpolation point calculation unit Under the condition that the displacement amount of the interpolation point in the sewing direction is equal to or less than ½ of the inter-hole distance between the plurality of holes included in the texture area, calculating a displacement amount of the interpolation point in the sewing direction based on the interpolation point before the sewing process and displacement amounts of a plurality of holes included in the texture area; The image processing device according to claim 2 .

5. a holding member capable of holding and moving an object to be sewn within a predetermined plane including a sewing position directly below a sewing machine needle; An actuator that generates power to move the holding member; and a control device that outputs a control command for controlling the actuator based on a processing result of the image processing device according to any one of claims 1 to 4. sewing machine.

6. The control device includes: a correction point information acquiring unit that acquires correction point information indicating information on a plurality of correction points including displacement amounts of the feature points and the interpolation points after the sewing process from the image processing device; a target stitch line correction unit that corrects the target stitch line based on the correction point information.

6. The sewing machine according to claim 5.

7. The control device includes: and a needle point correction unit that recalculates the needle points so that the interval between the needle points is fixed and the number of needle points is variable for the corrected target stitch line.

7. The sewing machine according to claim 6.

8. Dividing a surface of the sewing object into a texture region and a stitch region based on an object image showing an image related to the sewing object, and extracting a boundary line between the texture region and the stitch region; When correction point prior information indicating information before a sewing process of a correction point for correcting a target stitch line defined in the stitch area includes position data of a feature point having a two-dimensional positioning element, calculating the feature point after the sewing process based on the boundary line after the sewing process and the position data before the sewing process; and when the correction point advance information includes position data of an interpolation point having a one-dimensional positioning element and sewing direction data indicating a sewing direction, calculating the interpolation point after the sewing process based on the boundary line after the sewing process and the position data and the sewing direction data before the sewing process. Image processing methods.

9. 9. The image processing method according to claim 8, further comprising: correcting the target stitch line based on correction point information indicating information on a plurality of correction points including amounts of displacement of the feature points and the interpolation points after the sewing process calculated in the image processing method. Sewing correction method.

Images