Image processing apparatus, sewing machine, and image processing method

Abstract

An image processing device (40) includes a plurality of image capturing devices (41) arranged at intervals in a horizontal plane so as to be aligned with a sewing machine body (10) including a head (11D) for performing sewing and a holding member (15) for holding and moving an object to be sewn (S), and an image processing unit (42) for detecting a recognition object of an object to be sewn (S), based on images of the object to be sewn (S) captured by the plurality of image capturing devices (41). The image processing unit (42) includes a composition processing unit (44B) for stitching together the images captured by the plurality of imaging devices (41), and a detection processing unit (44C) for detecting a plurality of recognition objects (UP) from a composite image (75) obtained through the stitching by the composition processing unit (44B).

Description

TECHNICAL FIELD

[0001] The present disclosure relates to an image processing device, a sewing machine, and an image processing method.BACKGROUND ART

[0002] In order to enhance the design property of an object to be sewn, stitches may be formed on the object to be sewn. Patent Literature 1 discloses a technique for forming stitches on a skin material used for a vehicle seat.CITATION LISTPATENT LITERATURE

[0003] Patent Literature 1: JP2013-162957ASUMMARY OF INVENTIONTECHNICAL PROBLEM

[0004] A skin material used for a vehicle seat has thickness and elasticity. When a stitch is formed on an object to be sewn having thickness and elasticity, the object to be sewn may be contracted, thereby causing the surface of the object to be sewn to be displaced. For example, when forming a second stitch after a first stitch is formed, it is preferable to form the second stitch at a target position on the object to be sewn according to the displacement of the surface of the object to be sewn caused by the formation of the first stitch.

[0005] As a measure for forming a stitch at a target position of an object to be sewn, the surface of the object to be sewn may be photographed before sewing processing to detect displacement of the surface of the object to be sewn. If multiple points along a stitch line are imaged one location at a time in sequence and displacement detection is performed for each, the imaging and image recognition processing take time. The displacement changes, each time a stitch is formed, due to the stitch. Therefore, if the imaging and the displacement detection are performed each time a stitch is formed, the total processing time for the sewing operation increases.

[0006] An aspect of the present disclosure is to shorten the total processing time for a sewing operation.SOLUTION TO PROBLEM

[0007] According to a first aspect of the present disclosure, there is provided an image processing device including: a plurality of imaging devices arranged at intervals in a horizontal plane so as to be aligned with a sewing machine body including a head configured to perform sewing and a holding member configured to hold and move an object to be sewn; and an image processing unit configured to detect a recognition object of the object to be sewn on the basis of images of the object to be sewn captured by the plurality of imaging devices, in which the image processing unit includes a composition processing unit configured to stitch together the images captured by the plurality of imaging devices, and a detection processing unit configured to detect a plurality of the recognition objects from a composite image obtained through the stitching by the composition processing unit.

[0008] According to a second aspect of the present disclosure, there is provided a sewing machine including: a sewing machine body including a head configured to perform sewing and a holding member configured to hold and move an object to be sewn; the image processing device according to the first aspect; and a control device configured to control the sewing machine body on the basis of a processing result of the image processing device.

[0009] According to a third aspect of the present disclosure, there is provided an image processing method including: stitching together images of an object to be sewn captured by a plurality of imaging devices arranged at intervals in a predetermined direction in a horizontal plane so as to be aligned with a sewing machine body including a head configured to perform sewing and a holding member configured to hold and move an object to be sewn; and detecting a plurality of recognition objects of the object to be sewn from a composite image obtained through the stitching.ADVANTAGEOUS EFFECTS OF INVENTION

[0010] According to the aspects of the present disclosure, the total processing time for a sewing operation can be shortened.BRIEF DESCRIPTION OF DRAWINGS

[0011] Fig. 1 is a perspective view illustrating a sewing machine according to the present embodiment. Fig. 2 is a perspective view illustrating a part of the sewing machine according to the present embodiment. Fig. 3 is a cross-sectional view illustrating a part of an object to be sewn according to the present embodiment. Fig. 4 is a plan view illustrating the object to be sewn according to the present embodiment. Fig. 5 is a cross-sectional view illustrating a part of the object to be sewn according to the present embodiment. Fig. 6 is a plan view illustrating attachment positions of imaging devices according to the present embodiment. Fig. 7 is a plan view illustrating an arrangement example of the imaging devices according to the present embodiment. Fig. 8 is a functional block diagram illustrating the sewing machine according to the present embodiment. Fig. 9 is a flowchart showing a sewing operation of the sewing machine according to the present embodiment. Fig. 10 is a flowchart showing an image stitching process according to the present embodiment. Fig. 11 is a schematic diagram illustrating an image positioning process in the image stitching process. Fig. 12 is a schematic diagram illustrating a mask region setting process in the image stitching process. Fig. 13 is a schematic diagram illustrating a composition process in the image stitching process. Fig. 14 is a schematic diagram illustrating a recognition object detection process. DESCRIPTION OF EMBODIMENTS

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings, but the present disclosure is not limited thereto. Components of the embodiments described below can be combined appropriately. In addition, some components may not be used.

[0013] In the present embodiment, a local coordinate system is defined for a sewing machine 1. In the following description, the local coordinate system defined for the sewing machine 1 is appropriately referred to as a sewing machine coordinate system. The sewing machine coordinate system is defined by the XYZ orthogonal coordinate system. In the present embodiment, the positional relationship of each part will be described based on the sewing machine coordinate system. A direction parallel to an X-axis in a predetermined plane is defined as an X-axis direction. A direction parallel to a Y-axis in the predetermined plane orthogonal to the X-axis is defined as a Y-axis direction. A direction parallel to a Z-axis orthogonal to the predetermined plane is defined as a Z-axis direction. In addition, a rotational direction or an inclination direction about the X-axis is defined as a θX direction. A rotational direction or an inclination direction about the Y-axis is defined as a θY direction. A rotational direction or an inclination direction about the Z-axis is defined as a θZ direction. In addition, in the present embodiment, a plane including the X-axis and the Y-axis is appropriately referred to as an XY plane. A plane including the X-axis and the Z-axis is appropriately referred to as an XZ-plane. A plane including the Y-axis and the Z-axis is appropriately referred to as a YZ-plane. The XY plane is parallel to the predetermined plane. The XY plane, the XZ plane, and the YZ plane are orthogonal to one another. Additionally, in the present embodiment, the XY plane is parallel to a horizontal plane. The Z-axis direction is an up-down direction. The +Z direction is an upward direction, and the -Z direction is a downward direction. Note that the XY plane may be inclined with respect to the horizontal plane.[Sewing Machine]

[0014] Fig. 1 is a perspective view illustrating a sewing machine 1 according to the present embodiment. Fig. 2 is a perspective view illustrating a part of the sewing machine 1 according to the present embodiment. In the present embodiment, the sewing machine 1 is an electronic cycle sewing machine. The sewing machine 1 includes a sewing machine body 10, an operating device 20, a control device 30, and an image processing device 40. The operation device 20 is operated by an operator. The control device 30 controls the sewing machine body 10. The image processing device 40 captures an image of an object to be sewn S and performs image recognition. The image processing device 40 includes a plurality of imaging devices 41 and an image processing unit 42.

[0015] The sewing machine body 10 is mounted on an upper surface of a table 2. As illustrated in Fig. 2, the sewing machine body 10 includes a sewing machine frame 11, a needle bar 12, a throat plate 13, a holding member 15, an actuator 16, an actuator 17, and an actuator 18. The needle bar 12 and the throat plate 13 are each supported by the sewing machine frame 11. The holding member 15 is supported by the sewing machine frame 11 via a support member 14. The actuator 16 (see Fig. 1) generates power to move the needle bar 12. The actuator 17 (see Fig. 1) generates power to move the holding member 15. The actuator 18 (see Fig. 1) generates power to move at least a portion of the holding member 15.

[0016] The sewing machine frame 11 includes a horizontal arm 11A, a bed 11B, a vertical arm 11C, and a head 11D. The horizontal arm 11A extends in the Y-axis direction. The bed 11B (see Fig. 1) is arranged lower than the horizontal arm 11A. The vertical arm 11C (see Fig. 1) is arranged to connect an end portion on the +Y side of the horizontal arm 11A and the bed 11B. The head 11D is arranged on the -Y side of the horizontal arm 11A.

[0017] The needle bar 12 holds a sewing needle 3. The needle bar 12 holds the sewing needle 3 such that the sewing needle 3 is parallel to the Z-axis. The needle bar 12 is supported by the head 11D so as to be movable in the Z-axis direction.

[0018] The throat plate 13 supports the object to be sewn S. The throat plate 13 supports the holding member 15. The throat plate 13 is supported by the bed 11B (see Fig. 1). The throat plate 13 is arranged lower than the holding member 15.

[0019] The holding member 15 holds the object to be sewn S. The holding member 15 is capable of moving while holding the object to be sewn S, in the XY plane including a sewing position Ps directly below the sewing needle 3. The holding member 15 is capable of holding the object to be sewn S in the XY plane including a position directly below the imaging device 41. The holding member 15 moves in the XY plane based on sewing data while holding the object to be sewn S. As a result, a stitch CH (see Fig. 5) is formed on the object to be sewn S having passed through the sewing position Ps. The holding member 15 is supported by the horizontal arm 11A via the support member 14.

[0020] The holding member 15 includes a pressing member 15A and a lower plate 15B facing the pressing member 15A. The pressing member 15A is a frame-shaped member. The pressing member 15A is movable in the Z-axis direction. The lower plate 15B is arranged below the pressing member 15A. The holding member 15 holds the object to be sewn S by sandwiching the object to be sewn S with the pressing member 15A and the lower plate 15B.

[0021] When the pressing member 15A moves in the +Z direction, the pressing member 15A and the lower plate 15B are separated from each other. As a result, the operator can arrange the object to be sewn S between the pressing member 15A and the lower plate 15B. When the pressing member 15A moves in the -Z direction while the object to be sewn S is arranged between the pressing member 15A and the lower plate 15B, the object to be sewn S is sandwiched between the pressing member 15A and the lower plate 15B. As a result, the object to be sewn S is held by the holding member 15. In addition, when the pressing member 15A moves in the +Z direction, the object to be sewn S is released from being held by the holding member 15. As a result, the operator can take out the object to be sewn S from between the pressing member 15A and the lower plate 15B.

[0022] As illustrated in Fig. 1, the actuator 16 generates power to move the needle bar 12 in the Z-axis direction. The actuator 16 includes a pulse motor. The actuator 16 is arranged on the horizontal arm 11A.

[0023] Inside the horizontal arm 11A, a horizontal arm shaft extending in the Y-axis direction is arranged. The actuator 16 is connected to an end portion on the +Y side of the horizontal arm shaft. An end portion on the -Y side of the horizontal arm shaft is connected to the needle bar 12 via a power transmission mechanism arranged inside the head 11D. By an operation of the actuator 16, the horizontal arm shaft rotates. The power generated by the actuator 16 is transmitted to the needle bar 12 via the horizontal arm shaft and the power transmission mechanism. As a result, the sewing needle 3 held by the needle bar 12 reciprocates in the Z-axis direction.

[0024] Inside the vertical arm 11C, a timing belt extending in the Z-axis direction is arranged. In addition, inside the bed 11B, a bed shaft extending in the Y-axis direction is arranged. A pulley is arranged on each of the horizontal arm shaft and the bed shaft. The timing belt is looped around each of the pulley arranged on the horizontal arm shaft and the pulley arranged on the bed shaft. The horizontal arm shaft and the bed shaft are connected to each other via the power transmission mechanism including the timing belt.

[0025] Inside the bed 11B, a shuttle is arranged. In the shuttle, a bobbin contained in a bobbin case is accommodated. By an operation of the actuator 16, each of the horizontal arm shaft and the bed shaft rotates. The power generated by the actuator 16 is transmitted to the shuttle via the horizontal arm shaft, the timing belt, and the bed shaft. As a result, the shuttle rotates in synchronization with reciprocation of the needle bar 12 in the Z-axis direction.

[0026] The actuator 17 generates power to move the holding member 15 in the XY plane. The actuator 17 includes a pulse motor. The actuator 17 includes an X-axis motor 17X and a Y-axis motor 17Y. The X-axis motor 17X (see Fig. 8) generates power to move the holding member 15 in the X-axis direction. The Y-axis motor 17Y (see Fig. 8) generates power to move the holding member 15 in the Y-axis direction. The actuator 17 is arranged inside the bed 11B.

[0027] The power generated by the actuator 17 is transmitted to the holding member 15 via the support member 14. As a result, the holding member 15 can move in the X-axis direction and the Y-axis direction, respectively, between the sewing needle 3 and the throat plate 13. By an operation of the actuator 17, the holding member 15 can move while holding the object to be sewn S, in the XY plane including the sewing position Ps directly below the sewing needle 3.

[0028] The actuator 18 generates power to move the pressing member 15A of the holding member 15 in the Z-axis direction. The actuator 18 includes a pulse motor. When the pressing member 15A moves in the +Z direction, the pressing member 15A and the lower plate 15B are separated from each other. When the pressing member 15A moves in the -Z direction, the object to be sewn S is sandwiched between the pressing member 15A and the lower plate 15B.

[0029] As illustrated in Fig. 2, the sewing machine body 10 includes an intermediate pressing member 19 arranged around the sewing needle 3. The intermediate pressing member 19 restrains the object to be sewn S around the sewing needle 3. The intermediate pressing member 19 is supported by the head 11D so as to be movable in the Z-axis direction. Inside the head 11D, an intermediate pressing motor is arranged. The intermediate pressing motor generates power to move the intermediate pressing member 19 in the Z-axis direction. By an operation of the intermediate pressing motor, the intermediate pressing member 19 moves in the Z-axis direction in synchronization with the needle bar 12. The intermediate pressing member 19 suppresses lifting of the object to be sewn S caused by movement of the sewing needle 3.

[0030] As illustrated in Fig. 1, the operating device 20 receives an operating input from an operator. When the operation device 20 is operated, the sewing machine 1 is operated. The operation device 20 includes an operation panel 21 and an operation pedal 22. The operation panel 21 is mounted on the upper surface of the table 2. The operation pedal 22 is arranged below the table 2. The operator operates the operation pedal 22 with a foot. The sewing machine 1 operates when at least one of the operation panel 21 and the operation pedal 22 is operated by the operator.

[0031] Before the start of a sewing operation, the image processing device 40 performs processing for acquiring images of the object to be sewn S by the plurality of imaging devices 41, detecting a target position for sewing by the image processing unit 42, and calculating correction data for displacement caused by a created stitch CH or the like.

[0032] The plurality of imaging devices 41 capture images of the object to be sewn S held by the holding member 15. The imaging device 41 includes an optical system and an image sensor that receives light incident through the optical system. The image sensor includes a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor.

[0033] The plurality of imaging devices 41 are arranged higher than the throat plate 13 and the holding member 15. The plurality of imaging devices 41 each capture, from above, an image of at least a portion of the object to be sewn S held by the holding member 15. The number of imaging devices 41 is two or more, but is not particularly limited. Details such as an arrangement, imaging fields of view, and the like of the plurality of imaging devices 41 will be described below.

[0034] The image processing unit 42 includes a computer system. The image processing unit 42 processes an image relating to the object to be sewn S. The image processing unit 42 detects a recognition object of the object to be sewn S, based on the images of the object to be sewn S captured by the plurality of imaging devices 41.[Object To Be Sewn]

[0035] Fig. 3 is a cross-sectional view illustrating a part of the object to be sewn S according to the present embodiment. Fig. 4 is a plan view illustrating a part of the object to be sewn S according to the present embodiment. Figs. 3 and 4 illustrate the object to be sewn S before a sewing process. In the present embodiment, the object to be sewn S is a skin material used for a vehicle seat.

[0036] As illustrated in Fig. 3, the object to be sewn S includes a surface material 4, a padding material 5, and a back material 6. A hole 7 is provided in the surface material 4.

[0037] A surface of the surface material 4 is a seating surface that comes into contact with a passenger when the passenger sits on a vehicle seat. The surface material 4 includes at least one of woven fabric, nonwoven fabric, and leather. The padding material 5 has elasticity. The padding material 5 includes, for example, a urethane resin. The back material 6 includes at least one of woven fabric, nonwoven fabric, and leather.

[0038] As illustrated in Fig. 4, the hole 7 is provided in plurality in the surface material 4. The holes 7 are formed to be distributed over the object to be sewn S. The holes 7 are arranged in a regular pattern. That is, the object to be sewn S includes a plurality of reference patterns DPh. Each reference pattern DPh is formed by a regular array of a plurality of holes 7. In the example of Fig. 4, the reference pattern DPh is composed of 17 holes 7.

[0039] As illustrated in Fig. 4, the reference patterns DPh are arranged on the surface material 4 at intervals. The reference patterns DPh are arranged at equal intervals in each of the X-axis direction and the Y-axis direction. The reference patterns DPh located at different positions in the Y-axis direction are arranged between the reference patterns DPh adjacent in the X-axis direction. No hole 7 is formed between adjacent reference patterns DPh.

[0040] In an area between the reference patterns DPh, a target pattern RP of a stitch CH to be formed on the object to be sewn S is defined. Fig. 4 virtually illustrates the target pattern RP of the stitch CH on the object to be sewn S before sewing. The sewing machine 1 forms a stitch CH so as to follow each target pattern RP in accordance with a preset sewing pattern.

[0041] In addition, a plurality of recognition objects UP (UP1, UP2, UP3, UP4, UP5, UP6, UP7) are arranged on the object to be sewn S. In the present embodiment, the recognition object UP is an array of the plurality of holes 7 existing in a predetermined pattern on the object to be sewn S. Specifically, the recognition object UP is a part of the reference pattern DPh. In the example of Fig. 4, the recognition object UP (UP1, UP2, UP3, UP4, UP5, UP6, UP7) is a pattern (an array of the holes 7) including one acute-angled corner portion of the reference pattern DPh having a rhombic outer shape. The recognition object UP is a pattern that can be specified by a template matching method, which is a type of image processing method.

[0042] Referring to Fig. 5, displacement that occurs on the surface of the object to be sewn S when a stitch CH is formed on the object to be sewn S having thickness and elasticity will be described. Fig. 5 is a cross-sectional view illustrating an example of the object to be sewn S according to the present embodiment. Fig. 5 illustrates the object to be sewn S after sewing processing has been performed. The object to be sewn S has thickness and elasticity. When a stitch CH is formed on the object to be sewn S having thickness and elasticity, the object to be sewn S is highly likely to be contracted, as illustrated in Fig. 5. When the object to be sewn S is contracted, the surface of the object to be sewn S may be displaced. When the surface of the object to be sewn S is displaced, a target position of the stitch CH defined on the surface of the object to be sewn S is highly likely to be displaced in the XY plane. If the target position of the stitch CH is displaced in the XY plane, it becomes difficult to form the stitch CH at the target position when the holding member 15 is moved to the position coordinates of the target pattern RP defined in design.

[0043] Therefore, in the present embodiment, a displacement amount (a correction amount for correcting the displacement amount) is acquired by the image processing device 40. Even if the surface of the object to be sewn S is displaced due to the contraction of the object to be sewn S caused by the formation of the stitch CH, the holding member 15 is moved according to the displacement amount acquired by the image processing device 40 such that a next stitch CH is to be formed at the target position. The image processing device 40 (see Fig. 1) detects the recognition object UP from the captured image of the object to be sewn S and acquires a correction amount on the basis of the displacement of the recognition object UP.[Imaging Device]

[0044] Fig. 6 is a plan view illustrating attachment positions of the imaging devices 41 according to the present embodiment.

[0045] The plurality of imaging devices 41 are arranged at intervals in the horizontal plane (in the XY plane) so as to be aligned with the sewing machine body 10. The plurality of imaging devices 41 are arranged in a sewing area SA. The sewing area SA is a range in the XY plane in which the object to be sewn S can be moved by the holding member 15 and sewn by the head 11D. At a center of the sewing area SA, the head 11D (i.e., the sewing position Ps) is arranged. The plurality of imaging devices 41 are aligned to follow the head 11D of the sewing machine body 10.

[0046] In the example of Fig. 6, the sewing machine 1 includes imaging units 50A and 50B. The imaging units 50A and 50B are provided one on each of both sides of the head 11D in the X-axis direction, and are each arranged to be adjacent to the head 11D. The plurality of imaging devices 41 of the image processing device 40 can be attached to the imaging units 50A and 50B. The imaging units 50A and 50B are arranged above the sewing area SA. The imaging units 50A and 50B are each fixed to the sewing machine 1.

[0047] The imaging unit 50A on the -X direction side has three attachment portions 51A, 51B, and 51C. Each of the attachment portions 51A, 51B, and 51C can each have one imaging device 41 attached thereto. The attachment portion 51A and the attachment portion 51B are arranged at an interval in the X-axis direction. The attachment portion 51B and the attachment portion 51C are arranged at an interval in the Y-axis direction.

[0048] The imaging unit 50B on the +X direction side has three attachment portions 51D, 51E, and 51F. Each of the attachment portions 51D, 51E, and 51F can each have one imaging device 41 attached thereto. The attachment portion 51D and the attachment portion 51E are arranged at an interval in the X-axis direction. The attachment portion 51D and the attachment portion 51F are arranged at an interval in the Y-axis direction.

[0049] Each of the attachment portions 51A, 51B, 51C, 51D, 51E, and 51F is positioned on an inner side of the sewing area SA with respect to the XY direction. Each of the attachment portions 51A, 51B, 51C, 51D, 51E, 51F can hold the imaging device 41 above the sewing area SA. Therefore, whichever attachment portion the imaging device 41 is attached to, the sewing machine 1 can position the object to be sewn S at the imaging position of the imaging device 41 by the holding member 15. In the example of Fig. 6, the attachment portions 51A and 51B of the imaging unit 50A, and the attachment portions 51D and 51E of the imaging unit 50B are arranged on a straight line in the X-axis direction with respect to the sewing position Ps.

[0050] The sewing machine 1 is provided with a total of six attachment positions (attachment portions 51A, 51B, 51C, 51D, 51E, and 51F) by the imaging units 50A and 50B. In the present embodiment, the imaging devices 41 may be attached to all of the six attachment positions, or may be attached to some of the attachment positions (two or more adjacent positions). The sewing machine 1 can select an attachment position of the imaging device 41, as well as change the attachment position of the imaging device 41, according to the shape of the object to be sewn S and the shape of a stitch CH to be formed on the object to be sewn S.

[0051] In addition, the imaging units 50A and 50B hold a plurality of illumination devices 43A and 43B of the image processing device 40. The imaging units 50A and 50B hold the illumination devices 43A and 43B fixed above the sewing area SA. In the imaging unit 50A, the illumination devices 43A (three in total) are arranged one on each of both outer sides of the array of the attachment portion 51A and the attachment portion 51B aligned along the X-axis direction, and one between the attachment portion 51A and the attachment portion 51B. Each illumination device 43A extends linearly along the Y-axis. Each illumination device 43A irradiates light in the form of epi-illumination toward the imaging fields of view set below the attachment portions 51A, 51B, and 51C. In the imaging unit 50B, the illumination devices 43B (three in total) are arranged one on each of both outer sides of the array of the attachment portion 51D and the attachment portion 51F aligned along the Y-axis direction, and one between the attachment portion 51D and the attachment portion 51F. Each illumination device 43B extends linearly along the X-axis. Each illumination device 43B irradiates light in the form of epi-illumination toward the imaging fields of view set below the attachment portions 51D, 51E, and 51F.

[0052] In this manner, the image processing device 40 according to the present embodiment includes the plurality of imaging devices 41 so as to be aligned with the sewing machine body 10 (the head 11D). The plurality of imaging devices 41 are arranged at intervals in the horizontal plane (in the XY plane). In the example of Fig. 6, the plurality of imaging devices 41 may be arranged along at least one of a longitudinal direction (X-axis direction) and a width direction (Y-axis direction) of the sewing area SA.

[0053] Fig. 7 is a plan view illustrating an arrangement example of the imaging devices 41 according to the present embodiment. The plurality of imaging devices 41 may be arranged, for example, as illustrated in Fig. 7.

[0054] In the example of Fig. 7, two imaging devices 41 are attached to each of the attachment portion 51A and the attachment portion 51B of the imaging unit 50A. The two imaging devices 41 of the imaging unit 50A are arranged along the longitudinal direction (X-axis direction) of the sewing area SA. The imaging device 41 is attached to each of the attachment portion 51D and the attachment portion 51F of the imaging unit 50B. The imaging devices 41 of the imaging unit 50B are arranged along the width direction (Y-axis direction) of the sewing area SA.

[0055] The imaging device 41 acquires an image of at least a portion of the object to be sewn S arranged in the imaging field of view. The imaging device 41 captures, from above, an image of at least a portion of the object to be sewn S held by the holding member 15. Each of the plurality of imaging devices 41 is arranged to have an overlapping region JA. The overlapping region JA partially overlaps with the imaging field of view of another adjacent imaging device 41. Specifically, the imaging device 41 of the attachment portion 51A has an imaging field of view 60A. The imaging device 41 of the attachment 51B has an imaging field of view 60B. The imaging field of view 60A and the imaging field of view 60B have an overlapping region JA having a rectangular shape and extending in the Y-axis direction. By the imaging field of view 60A and the imaging field of view 60B, an enlarged imaging region 61A is formed. The enlarged imaging region 61A is a region formed by combining the imaging fields of view of the plurality of imaging devices 41 that share the overlapping region JA. The imaging field of view 60A and the imaging field of view 60B aligned in the X-axis direction form the wide enlarged imaging region 61A extending in the X-axis direction.

[0056] In addition, the imaging device 41 of the attachment portion 51D has an imaging field of view 60D. The imaging device 41 of the attachment portion 51F has an imaging field of view 60F. The imaging field of view 60D and the imaging field of view 60F have an overlapping region JA having a rectangular shape and extending in the X-axis direction. By the imaging field of view 60D and the imaging field of view 60F, an enlarged imaging region 61B is formed. The imaging field of view 60D and the imaging field of view 60F aligned in the Y-axis direction form the wide enlarged imaging region 61B extending in the Y-axis direction.

[0057] In the configuration of Fig. 7, a size of the imaging field of view of the imaging device 41, a size of the overlapping region JA (an interval between the attachment portions), and the number of imaging devices 41 are set according to a dimension of the target pattern RP of the object to be sewn S (see Fig. 4). That is, the enlarged imaging regions 61A and 61B formed by the plurality of imaging devices 41 are designed to have a shape that encompasses the entirety of one target pattern RP of the object to be sewn S. The object to be sewn S on which a stitch CH extending in the X-axis direction is formed can be arranged and imaged in the enlarged imaging region 61A of the imaging unit 50A. The object to be sewn S on which a stitch CH extending in the Y-axis direction is formed can be arranged and imaged in the enlarged imaging region 61B of the imaging unit 50B. Accordingly, even when forming a stitch CH extending in either direction, an image of the entire target pattern RP can be captured without changing attachment positions of the imaging devices 41.

[0058] Note that the position of each imaging device 41 is fixed at one of the six attachment positions. A relative position between the imaging device 41 and the sewing machine frame 11 is fixed. A relative position between an optical axis of the imaging device 41 and the sewing needle 3 (the sewing position Ps) in the XY plane is fixed. Relative position data representing the relative position between a center of each of the enlarged imaging regions 61A and 61B and the sewing needle 3 in the XY plane is known data that can be derived from design data of the sewing machine 1.

[0059] A position of an image acquired by the imaging device 41 is defined in the camera coordinate system. The position of the image defined in the camera coordinate system is converted into a position of the image defined in the sewing machine coordinate system by a predetermined transformation formula or transformation matrix.

[0060] In addition, in the example of Fig. 7, the head 11D is provided with a second imaging device 33. As illustrated in Fig. 6, the head 11D has attachment portions 33A and 33B provided at an interval on both sides of the sewing position Ps in the X-axis direction, respectively. The second imaging device 33 is attached to one or both of the attachment portions 33A and 33B. In Fig. 7, one second imaging device 33 is provided on the attachment portion 33A. The second imaging device 33 has a smaller field of view and a higher resolution (number of pixels per unit area) compared with the imaging device 41. Accordingly, as described below, the plurality of imaging devices 41 may be for use in highspeed recognition for collectively capturing images of a plurality of recognition objects UP, whereas the second imaging device 33 may be for use in capturing an image of one recognition object UP in a single imaging, particularly for use in precise recognition in which precise positioning is required.[Sewing Machine Control Configuration]

[0061] Fig. 8 is a functional block diagram illustrating the sewing machine 1 according to the present embodiment. Referring to Fig. 8, the configuration related to control of the sewing machine 1 will be described.(Control Device)

[0062] The control device 30 outputs a control signal for controlling the sewing machine 1. The control device 30 includes a computer system. The control device 30 includes an input / output interface, a storage device including a non-volatile memory and a volatile memory, and a computation processing device including a processor. The non-volatile memory is, for example, a read only memory (ROM) or a storage. The volatile memory is, for example, a random access memory (RAM) or the like. The processor is, for example, a central processing unit (CPU). The control device 30 controls the sewing machine 1 in accordance with a computer program stored in the storage device.

[0063] The control device 30 is connected to the actuator 16, the actuator 17, the actuator 18, the operation device 20, and the image processing unit 42. The actuator 16 moves the sewing needle 3 in the Z-axis direction. The actuator 17 moves the holding member 15 in the XY plane. The actuator 18 moves the pressing member 15A of the holding member 15 in the Z-axis direction.

[0064] Additionally, a drive amount sensor 31 and a drive amount sensor 32 are connected to the control device 30. The drive amount sensor 31 detects a drive amount of the actuator 16. The drive amount sensor 32 detects a drive amount of the actuator 17.

[0065] The control device 30 controls the actuator 16 on the basis of the detection data from the drive amount sensor 31. The control device 30 determines, for example, an operation timing of the actuator 16 on the basis of the detection data from the drive amount sensor 31.

[0066] The drive sensor 32 includes an X-axis sensor 32X and a Y-axis sensor 32Y. The X-axis sensor 32X detects a drive amount of the X-axis motor 17X of the actuator 17. The Y-axis sensor 32Y detects a drive amount of the Y-axis motor 17Y of the actuator 17. The X-axis sensor 32X includes an encoder that detects a rotation amount of the X-axis motor 17X. The Y-axis sensor 32Y includes an encoder that detects a rotation amount of the Y-axis motor 17Y. Detection data of the drive amount sensor 32 is output to the control device 30. The control device 30 controls the actuator 17 on the basis of the detection data from the drive amount sensor 32. The control device 30 feedback-controls the actuator 17 so that the holding member 15 moves to a target position, based on the detection data from the drive amount sensor 32.

[0067] The drive amount sensor 32 serves as a position sensor that detects a position of the holding member 15 in the XY plane. The drive amount of the actuator 17 and a movement amount of holding member 15 correspond to each other on a one-to-one basis.

[0068] The X-axis sensor 32X can detect a movement amount of the holding member 15 in the X-axis direction from the origin in the sewing machine coordinate system by detecting a rotation amount of the X-axis motor 17X. The Y-axis sensor 32Y can detect a movement amount of the holding member 15 in the Y-axis direction from the origin in the sewing machine coordinate system by detecting a rotation amount of the Y-axis motor 17Y.

[0069] The control device 30 calculates a position of the holding member 15 in the XY plane on the basis of the detection data from the drive amount sensor 32. Based on the detection data from the drive amount sensor 32, the movement amount of the holding member 15 from the origin in the XY plane is detected. The control device 30 calculates a position of the holding member 15 in the XY plane on the basis of the detected movement amount of the holding member 15.

[0070] The control device 30 stores sewing data by the storage device. The sewing data is known data that can be derived from design data of the object to be sewn S, such as CAD (Computer Aided Design) data. The sewing data is referenced by the control device 30 in the sewing processing. The sewing processing refers to processing for forming a stitch CH on the object to be sewn S. The sewing data includes the target pattern RP (see Fig. 4) of the stitch CH that is formed on the object to be sewn S, the position coordinates of the recognition object UP, and movement conditions of the holding member 15.

[0071] The target pattern RP defines a target shape of the stitch CH that is formed on the object to be sewn S and a target position of the stitch CH in the sewing machine coordinate system.

[0072] The movement conditions of the holding member 15 include a movement trajectory of the holding member 15 defined in the sewing machine coordinate system. The movement trajectory of the holding member 15 includes a movement trajectory of the holding member 15 in the XY plane. The movement conditions of the holding member 15 are determined based on the target pattern RP.

[0073] Therefore, as illustrated in Fig. 4, when performing a plurality of sewing processes to form a plurality of stitches CH, a first sewing process is performed, and following the first sewing process, a second sewing process is performed. The first sewing process is a process for forming a first stitch CH on the object to be sewn S on the basis of the first target pattern RP1. The second sewing process is a process for forming a second stitch CH on the object to be sewn S on the basis of the second target pattern RP2. Thereafter, the sewing processes from a third sewing process to an Nth sewing process are sequentially performed according to the number N of stitches CH that are to be formed. Fig. 4 illustrates an example in which N is 10, and the target patterns from the first target pattern RP1 to the tenth target pattern RP10 are defined.

[0074] The control device 30 is connected to the image processing unit 42 via the input / output interface. The control device 30 outputs a control command to control the actuator 17 that moves the holding member 15, based on a processing result of the image processing unit 42.(Image Processing Unit)

[0075] The image processing unit 42 includes a computer system. The image processing unit 42 includes a processor 44, a storage device 45 including a non-volatile memory and a volatile memory, and an input / output unit 46 including an input / output interface. The processor 44 is, for example, a central processing unit (CPU). The non-volatile memory is, for example, a read only memory (ROM) or a storage. The volatile memory is, for example, a random access memory (RAM) or the like. The image processing unit 42 performs image processing for capturing images of the object to be sewn S by the plurality of imaging devices 41 and for detecting the recognition objects UP from the images, in accordance with a computer program stored in the storage device 45.

[0076] The image processing unit 42 is connected to the plurality of imaging devices 41 and the plurality of illumination devices 43A and 43B via the input / output unit 46. In Fig. 8, the input / output unit 46 is connected to each imaging device 41 and the illumination device 43A of the imaging unit 50A via a relay device 47. In addition, the input / output unit 46 is connected to each imaging device 41 and the illumination device 43B of the imaging unit 50B via the relay device 47. The relay device 47 is a concentrator for connecting a plurality of devices, such as a USB (Universal Serial Bus) hub.

[0077] The image processing unit 42 includes an image acquisition unit 44A, a composition processing unit 44B, a detection processing unit 44C, and a correction amount calculation unit 44D. The image acquisition unit 44A, the composition processing unit 44B, the detection processing unit 44C, and the correction amount calculation unit 44D are functional blocks that are implemented when the processor 44 executes a computer program. The image acquisition unit 44A, the composition processing unit 44B, the detection processing unit 44C, and the correction amount calculation unit 44D may be configured separately with dedicated processors.

[0078] The image acquisition unit 44A acquires images relating to the object to be sewn S from the plurality of imaging devices 41. The imaging device 41 captures an image of the object to be sewn S and outputs the obtained image to the image processing unit 42. When the object to be sewn S is positioned at a predetermined imaging position by the control device 30, the image acquisition unit 44A controls an operation of the imaging device 41 corresponding to the imaging position, and acquires an image relating to the object to be sewn S.

[0079] In Fig. 7, the imaging position Pf is set at the center of each of the enlarged imaging regions 61A and 61B. Imaging is performed with the central portion of the holding member 15 (the object to be sewn S) arranged at one of the imaging positions Pf of the enlarged imaging regions 61A and 61B. For example, in the imaging unit 50A, when the object to be sewn S is arranged at the imaging position Pf of the enlarged imaging region 61A, the image acquisition unit 44A turns on the illumination device 43A and causes the imaging device 41 of the attachment portion 51A and the imaging device 41 of the attachment portion 51B to perform imaging substantially simultaneously. As a result, an image corresponding to the imaging field of view 60A and an image corresponding to the imaging field of view 60B are acquired. In the imaging unit 50B, when the object to be sewn S is arranged at the imaging position Pf of the enlarged imaging region 61B, the image acquisition unit 44A turns on the illumination device 43B and causes the imaging device 41 of the attachment portion 51D and the imaging device 41 of the attachment portion 51F to perform imaging substantially simultaneously. The imaging may be performed in sequence.

[0080] The compositing processing unit 44B performs image compositing processing for stitching together the images from the plurality of imaging devices 41. The composition processing unit 44B stitches together a plurality of images constituting the enlarged imaging region (61A or 61B) on the basis of image portions of the overlapping regions JA with each other, thereby generating one composite image 75 (see Fig. 13). Therefore, the composite image 75 is an image that captures a range corresponding to the enlarged imaging region 61A or the enlarged imaging region 61B. The enlarged imaging regions 61A and 61B have a size that encompasses a plurality of recognition objects UP of the object to be sewn S. Therefore, the composite image 75 is an image that captures the plurality of recognition objects UP. Preferably, the composite image 75 includes all of the recognition objects UP (UP1 to UP7) detected when forming one stitch CH.

[0081] The detection processing unit 44C detects a plurality of recognition objects UP from the composite image 75 obtained through the stitching by the composition processing unit 44B. The detection processing unit 44C detects the recognition object UP in the composite image 75 by using a pattern matching method, which is a type of image processing method. Here, the operator stores in advance a reference image (template) for recognizing the recognition object UP of the object to be sewn S in the storage device 45. The detection processing unit 44C acquires the reference image recorded in the storage device 45, and acquires the position coordinates of the recognition object UP in the composite image 75 through matching with the reference image. The detection processing unit 44C sequentially detects the plurality of recognition object UPs included in the composite image 75, and acquires the position coordinates of each of the plurality of recognition objects UP.

[0082] The correction amount calculation unit 44D calculates correction data for correcting displacement of the surface of the object to be sewn S, based on the detection result obtained by the detection processing unit 44C. The target pattern RP of the sewing data is generated on the assumption that the object to be sewn S is not contracted, and is stored in the control device 30. When the sewing processing is performed, and the object to be sewn S is contracted to cause the surface of the object to be sewn S to be displaced, the correction amount calculating unit 44D obtains a displacement amount of the recognition object UP detected by the detection processing unit 44C, and generates correction data for correcting the sewing data.

[0083] Specifically, the correction amount calculation unit 44D generates correction data so that a relative position in the sewing machine coordinate system between the target pattern RP and the recognition object UP before the object to be sewn S is contracted matches a relative position in the sewing machine coordinate system between the stitch CH actually formed on the object to be sewn S through the sewing processing and the recognition object UP after the object to be sewn S is contracted.

[0084] The image processing unit 42 transmits the correction data calculated by the correction amount calculation unit 44D to the control device 30. The control device 30 executes sewing processing on the basis of the correction data.[Sewing Operation]

[0085] Fig. 9 is a flow chart showing a sewing operation of the sewing machine 1 according to the present embodiment. The sewing operation of the sewing machine 1 includes an imaging process S1, an image stitching process S2, a recognition object detection process S3, a correction data calculation process S4, a sewing process S5, and an end determination process S6. Note that the image processing method according to the present embodiment is performed as part of the sewing operation of the sewing machine 1. The image processing method according to the present embodiment includes an image stitching process S2 and a recognition object detection process S3.(Imaging Process)

[0086] The imaging process S1 is a process for capturing images of the object to be sewn S by the plurality of imaging devices 41. Whether to acquire an image of the enlarged imaging region 61A extending in the X-axis direction by the imaging unit 50A or to acquire an image of the enlarged imaging region 61B extending in the Y-axis direction by the imaging unit 50B is set in advance by the operator, or by the control device 30 based on sewing data. The control device 30 moves the holding member 15 holding the object to be sewn S and arranges the holding member 15 at a preset imaging position. The image processing unit 42 performs imaging by the plurality of set imaging devices 41 when the holding member 15 is arranged at the corresponding imaging position Pf.

[0087] Hereinafter, as an example, a case in which an image of the enlarged imaging region 61A is acquired by each imaging device 41 of the imaging unit 50A of Fig. 7 will be described. In this case, the image acquisition unit 44A acquires an image corresponding to the imaging field of view 60A and an image corresponding to the imaging field of view 60B by the imaging device 41 of the attachment portion 51A and the imaging device 41 of the attachment portion 51B.(Image Stitching Process)

[0088] The image stitching process S2 is a process for stitching together the images of the object to be sewn S captured by the plurality of imaging devices 41.

[0089] Fig. 10 is a flowchart showing the image stitching process S2 according to the present embodiment. Fig. 11 is a schematic diagram illustrating an image positioning process S2A in the image stitching process S2. Fig. 12 is a schematic diagram illustrating a mask region setting process S2B in the image stitching process S2. Fig. 13 is a schematic diagram illustrating a composition process S2C in the image stitching process S2.

[0090] As illustrated in Fig. 11, the image obtained by the imaging device 41 of the attachment portion 51A is defined as a first image 71. The image obtained by the imaging device 41 of the attachment portion 51B is defined as a second image 72. In this case, the second image 72 includes a region on the +X direction side with respect to the first image 71. The first image 71 corresponds to the imaging field of view 60A, and the second image 72 corresponds to the imaging field of view 60B. The first image 71 and the second image 72 share the overlapping region JA.

[0091] The overlapping region JA of the first image 71 and the second image 72 includes image elements GE that are common to at least a portion of the recognition object UP. That is, as illustrated in Fig. 4, when the recognition object UP is an array of the plurality of holes 7 existing in a predetermined pattern in the object to be sewn S, the image element GE is an image of each hole 7. The hole 7 is a component that constitutes the recognition object UP or the reference pattern DPh. In Figs. 11 to 13, for convenience of description, an example is illustrated in which the array of the holes 7 is simplified such that the holes 7 are arranged in an array pattern vertically and horizontally.

[0092] In detecting the recognition object UP by pattern matching, it is important that no omission or error occurs, as image information, in the image elements GE of the holes 7 that constitute the recognition target UP. Therefore, in the image stitching process S2 shown in Fig. 10, in stitching together the first image 71 and the second image 72, composition of the first image 71 and the second image 72 is performed such that information of the image elements GE constituting the recognition object UP is preserved.

[0093] As shown in Fig. 10, the image stitching process S2 includes an image positioning process S2A, a mask region setting process S2B, and a composition process S2C. The composition processing unit 44B may, prior to the image stitching process S2, binarize each image obtained in the imaging process S1.<Image Positioning Process>

[0094] The image positioning process S2A is a process for positioning the plurality of images obtained in the imaging process S1 on the basis of the image elements GE in the overlapping region JA.

[0095] As illustrated in Fig. 11, the composition processing unit 44B detects the image elements GE included in the overlapping region JA of the first image 71 and the image elements GE included in the overlapping region JA of the second image 72, respectively. Since the position coordinates, in the sewing machine coordinates, of the imaging centers of the imaging devices 41 attached to the attachment portions 51A and 51B and the sizes of the imaging fields of view 60A and 60B are known, the range of the overlapping region JA in each image is known. The composition processing unit 44B selects three or more image elements GE included in the overlapping region JA. In the example of Fig. 11, the composition processing unit 44B divides the overlapping region JA into four regions vertically and horizontally, and selects one image element GE from each of the divided regions, for a total of four image elements GE, as attention image elements GE1.

[0096] The composition processing unit 44B transforms the second image 72 such that corresponding image elements GE in the overlapping region JA match each other. That is, the composition processing unit 44B transforms the second image 72 such that the four corresponding attention image elements GE1 of the second image 72 match the four attention image elements GE1 of the first image 71. The transformation of the second image 72 is, for example, performed by affine transformation. The transformation may include X-axis and Y-axis coordinates, rotation in the θZ direction, scale adjustment (enlargement or reduction), and distortion correction (shear transformation). The first image 71 and the second image 72 after transformation (hereinafter, referred to as the second image 72A) match each other in positions and shapes of the four attention image elements GE1. Accordingly, the first image 71 and the transformed second image 72A substantially match each other in the images of the overlapping regions JA. Through the image positioning process S2A, differences in image distortion caused by assembly errors in the installation positions of the captured first image 71 and the second image 72A, or by aberrations of the optical system, are eliminated. As described below, the composition processing unit 44B stitches together the transformed second image 72A and the first image 71.<Mask Region Setting Process>

[0097] The mask region setting process S2B (see Fig. 10) is a process for setting regions to be stitched together by the composition process S2C for the first image 71 and the second image 72A that have been positionally aligned by the image positioning process S2A. In the present embodiment, the composition processing unit 44B sets a stitching position in the overlapping region JA such that the entire image elements GE detected in the overlapping region JA are included in either the first image 71 or the second image 72A.

[0098] As illustrated in Fig. 12, the composition processing unit 44B extracts the image elements GE in contact with an end edge EL on the second image 72A side in the overlapping region JA, and sets a first mask region 73 excluding the extracted image elements GE, as the first image 71.

[0099] For example, the composition processing unit 44B adds a straight line SL to the end edge EL and extracts a contour having the largest area within the overlapping region JA. Adding the straight line SL to the end edge EL means setting a predetermined pixel value to the row of pixels constituting the end edge EL in the overlapping region JA. When there is an image element GE that is in contact with the end edge EL, the image element GE is connected to the straight line SL. As a result, when the contour having the largest area within the overlapping region JA is extracted, only the image elements GE connected to the straight line SL (i.e., the image elements in contact with the end edge EL) are extracted. A contour formed by the straight line SL and the image elements GE connected to the straight line SL is referred to as a contour region 76.

[0100] The composition processing unit 44B sets, as a first mask region 73, a region obtained by excluding the extracted contour region 76 from within the first image 71. The first mask region 73 is a region within the first image 71 that is adopted as pixels constituting the composite image 75. The contour region 76 of the first image 71 is not adopted in the composition process because it is excluded from the first mask region 73.

[0101] Next, the composition processing unit 44B sets a second mask region 74 including the entire image elements GE in contact with the end edge EL of the overlapping region JA and adjacent to the first mask region 73, as the second image 72A.

[0102] The composition processing unit 44B sets, from within the second image 72A, a portion of the extracted contour region 76 of the overlapping region JA and a unique region of the second image 72A not included in the overlapping region JA, as the second mask region 74. Since the portion of the contour region 76 has been excluded from the first mask region 73, the second mask region 74 is adjacent to the first mask region 73 without any gap at the portion of the contour region 76. The second mask region 74 is a region within the second image 72A that is adopted as pixels constituting the composite image 75.

[0103] Accordingly, the first mask region 73 includes the unique region of the first image 71 and the remaining part of the overlapping region JA excluding the contour region 76. The unique region of the first image 71 is a region that is included only in the first image 71 and not in the second image 72A. The second mask region 74 includes the unique region of the second image 72A and the contour region 76 of the overlapping region JA. The unique region of the second image 72A is a region that is included only in the second image 72A and not in the first image 71.

[0104] Here, the image element GE in contact with the end edge EL is described. The end edge EL on the second image 72A side of the overlapping region JA is an end edge on the +X direction side of the first image 71. Therefore, the image element GE in contact with the end edge EL of the overlapping region JA is in contact with the end edge of the first image 71. In the image element GE in contact with the end edge E of the first image 71, a portion not captured in the first image 71 (a portion P2 located on the +X side beyond the end edge EL) may exist. That is, only a portion P1 of the hole 7 located on the -X side beyond the end edge EL may be captured in the first image 71.

[0105] If the position of the end edge EL is set to a boundary between the first mask region 73 and the second mask region 74, the portion P1 on the -X side is acquired from the first image 71, and the portion P2 on the +X side is acquired from the second image 72A. Accordingly, if the position of the end edge EL is set to the boundary between the first mask region 73 and the second mask region 74, information loss or errors may occur. On the other hand, the end edge EL on the second image 72A side of the overlapping region JA is located on the center side of the second image 72A. For this reason, in the second image 72A, the entire image element GE (both the portions P1 and P2) in contact with the end edge EL of the overlapping region JA is reliably captured. Therefore, by including the portion of the contour region 76 in the second mask region 74 of the second image 72A, it becomes possible to generate the composite image 75 in which the information of the image element GE of the hole 7 existing across the end edge EL is completely preserved.

[0106] Note that there may also be a case where there is no image element GE in contact with the end edge EL of the overlapping region JA. In this case, the composition processing unit 44B may set the first mask region 73 and the second mask region 74 with the end edge EL as a boundary. That is, the first mask region 73 may be set as the entire first image 71 including the overlapping region JA. The second mask region 74 may be set as a unique region of the second image 72A excluding the overlapping region JA, with the end edge EL as a boundary. When the recognition object UP is an array of multiple image elements GE (i.e., holes 7), a region other than the image elements GE in the image corresponds to a "background", which is not a processing target in the image recognition processing. Therefore, when the position of the end edge EL corresponds to the background, even if image information loss or errors occur, the detection accuracy of the recognition object UP is not affected.<Composition Process>

[0107] The composition process S2C (see Fig. 10) is a process for stitching together the first image 71 and the second image 72A by using the boundary between the first mask region 73 of the first image 71 and the second mask region 74 of the second image 72A as a stitching position. As illustrated in Fig. 13, the composition processing unit 44B adopts each pixel belonging to the first mask region 73 in the first image 71 and each pixel belonging to the second mask region 74 in the second image 72A to generate a composite image 75 in which the first image 71 and the second image 72A are stitched together. The generated composite image 75 is an image having a size corresponding to the enlarged imaging region 61A. In the composite image 75, a region corresponding to the first mask region 73 is an image region acquired from the first image 71. In the composite image 75, a region corresponding to the second mask region 74 is an image region acquired from the second image 72. Each image element GE included in the composite image 75 has no image information loss or errors with respect to the image element GE because the image element GE is entirely acquired from either the first image 71 or the second image 72A. In this manner, the image stitching process S2 is performed.(Recognition Object Detection process)

[0108] Next, the recognition object detection process S3 of Fig. 9 is a process for detecting the recognition object UP from the composite image 75 generated by the image stitching process S2 and acquiring the position coordinates of the detected recognition object UP. Fig. 14 is a schematic diagram illustrating the recognition object detection process S3.

[0109] As illustrated in Fig. 14, the detection processing unit 44C extracts, from within the composite image 75, images of recognition regions AR in which detection processing for the recognition objects UP is performed. The recognition region AR is set to a range that takes into account an assumed positional deviation amount of the recognition object UP associated with stitch formation, with respect to the design position coordinates of the recognition object UP set in the sewing data. Fig. 14 schematically illustrates the setting of the recognition regions AR for the target pattern RP having an arbitrary shape. The detection processing unit 44C sets a recognition area AR for each of a plurality of recognition objects UP to be detected in one stitch formation, and acquires an extracted image 77 obtained by cutting out an image of the set recognition area AR. For example, in the example of Fig. 4, the recognition region AR is set for each of the seven recognition objects UP from the recognition object UP1 to the recognition object UP7. Note that, the composition process S2C of Figs. 10 and 13 (i.e., generation of the composite image 75) may be performed only when the boundary (the end edge EL) of the overlapping region JA is included in the recognition area AR.

[0110] The detection processing unit 44C queues (sets a processing order for) each of the acquired extracted images 77 and sequentially performs image recognition processing using a pattern matching method. The detection processing unit 44C reads a reference image from the storage device 45 and acquires position coordinates of the recognition object UP in the extracted image 77 through matching with the reference image. As illustrated in Fig. 14, the processor 44 that implements the functions of the image processing unit 42 may include a plurality of processor cores CR. The processor core CR is a unit of a computation processing structure provided in the processor 44. Each processor core CR is capable of executing computation processing in accordance with a computer program. When the processor 44 includes a plurality of processor cores CR, the image processing unit 42 executes the image recognition process of each queued extraction image 77 in parallel by the plurality of processor cores CR. When the number of available processor cores CR is K, the image recognition processing for K extracted images 77 can be performed in parallel at the same time, thereby shortening the processing time.

[0111] Through the recognition object detection process S3, the detection processing unit 44C acquires the position coordinates in the sewing machine coordinate system of each recognition object UP (UP1 to UP7, see Fig. 4) to be detected in one stitch formation.(Correction Data Calculation Process)

[0112] The correction data calculation process S4 is a process for calculating correction data for correcting displacement of the surface of the object to be sewn S, based on the detection result obtained by the recognition object detection process S3. The correction amount calculation unit 44D calculates a displacement amount of the recognition object UP from an initial position to a current position, based on the initial position and the current position of the recognition object UP related to the current sewing process. The initial position of the recognition object UP is design position coordinates of the recognition object UP set in the sewing data. The current position of the recognition object UP is position coordinates of the recognition object UP detected by the recognition object detection process S3. The correction amount calculation unit 44D calculates, based on the calculated displacement amount, correction position coordinates of each correction point CP (see Fig. 14) on the target pattern RP such that the stitch CH is formed at a target position in the sewing machine coordinate system, and calculates correction data for the target pattern RP passing through the correction point CP. Note that, in a first sewing process, the displacement of the surface of the object to be sewn S caused by stitch formation has not occurred, and therefore displacement of the recognition object UP also does not occur. Therefore, in the first sewing process, the target pattern RP in an initial state defined by the sewing data is used as it is, so no correction data is calculated.(Sewing Process)

[0113] The sewing process S5 is a process for forming a stitch CH on the basis of the target pattern RP. The sewing process S5 is performed as each unit of the sewing machine 1 is controlled by the control device 30. The sewing process includes sewing processes from a first sewing process to an Nth sewing process (N=10 in Fig. 4). The first sewing process is performed based on the target pattern RP in the initial state defined by the sewing data. The sewing processes from the second sewing process to the Nth sewing process are performed based on the correction data (the target pattern RP after correction) calculated by the correction data calculation process S4. In the first sewing process, the control device 30 outputs a control command to the actuator 17 so that a stitch CH is formed in accordance with the target pattern RP in the initial state defined by the sewing data. From the second sewing process onward, the control device 30 acquires the correction data from the image processing unit 42, and outputs a control command to the actuator 17 so that a stitch CH is formed in accordance with the target pattern RP of the acquired correction data.

[0114] The end determination process S6 is a process for determining whether the sewing process for the object to be sewn S has been completed. The control device 30 determines whether the sewing process for the object to be sewn S has been completed on the basis of the sewing data. In a state in which the sewing processes from the first sewing process to the (N-1) sewing process have been completed, the control device 30 determines in the end determination process S6 that the sewing process has not been completed. In a state in which the Nth sewing process has been completed, the control device 30 determines in the end determination process S6 that the sewing process has been completed.

[0115] Accordingly, during the sewing processes from the first sewing process to the Nth sewing process, each time one sewing process is completed, the imaging process S1, the image stitching process S2, the recognition object detection process S3, and the correction data calculation process S4 are performed, whereby the correction data for the next sewing process is calculated, and the next sewing process is performed based on the calculated correction data. Such a series of processes is repeated. In the present embodiment, the plurality of recognition objects UP to be detected in one sewing process can be detected from the composite image 75 obtained by a single imaging operation performed by the plurality of imaging devices 41. As a result, the processing time required for the imaging process S1, the image stitching process S2, the recognition object detection process S3, and the correction data calculation process S4, which are performed for each sewing process, is reduced.[Effects]

[0116] As described above, according to the present embodiment, the images of the object to be sewn S captured by the plurality of imaging devices 41 are stitched together, and the plurality of recognition objects UP of the object to be sewn S are detected from the composite image 75 obtained through the stitching. Accordingly, compared with a case where each of the plurality of recognition objects UP is individually captured and detected, the number of times the object to be sewn S is moved and the number of times the imaging operation is performed can be reduced, so the time required for detection of the recognition object UP associated with the sewing process can be shortened. As a result, the time required for detection of the recognition object UP performed for each sewing process is reduced, thereby shortening the total processing time for the sewing operation. In addition, since the overlapping region JA is provided in the imaging fields of view of the plurality of imaging devices 41 and the same image elements GE in the overlapping region JA are matched with each other, high positional accuracy is obtained even when images are stitched together. In addition, since the stitching position is set such that the entire image element GE detected within the overlapping region JA is included in either the first image 71 or the second image 72, it is possible to prevent information loss or errors of the image element GE constituting the recognition object UP. As a result, the detection accuracy of the recognition object UP can be improved. In addition, since the image element GE in contact with the end edge EL on the second image 72 side in the overlapping region JA is extracted, and the second mask region 74 including the entire extracted image element GE is set as the second image 72, the image element GE whose image information may be lost during stitching can be easily extracted, and stitching can be performed while preserving the entire image element GE. Therefore, the computation load associated with the stitching process can be reduced, and the processing time can be shortened.

[0117] In addition, in the present embodiment, the image elements GE are the holes 7 formed to be distributed over the object to be sewn S, and even when the recognition object UP is an array of the plurality of holes 7 existing in a predetermined pattern in the object to be sewn S, the recognition object UP can be detected with high accuracy in a short time. In addition, since the plurality of imaging devices 41 are arranged along at least one of the longitudinal direction (X-axis direction) and the width direction (Y-axis direction) of the sewing area SA, it is possible to generate a composite image 75 that allows collective detection of the plurality of recognition objects UP according to the extension direction of the stitch CH. In addition, the plurality of illumination devices 43 (43A, 43B) are provided on both outer sides of the array of the plurality of imaging devices 41 and between the plurality of imaging devices 41 of the array, respectively. Therefore, even when a composite image 75 covering a wide area is generated, it is possible to achieve uniformity in brightness (pixel values) within the image, thereby improving detection accuracy.[Other Embodiments]

[0118] In the above-described embodiment, the example has been illustrated in which four imaging devices 41 are provided. However, the number of imaging devices 41 may be two, three, or five or more. The number of attachment positions of the imaging device 41 may be six or more. The plurality of imaging devices 41 may be arranged along a direction other than the longitudinal direction (X-axis direction) and the width direction (Y-axis direction) of the sewing area SA. The number of illumination devices 43A and 43B is not limited, and may be one or two or more. Although the example has been illustrated in which the image element GE is the hole 7 formed in the object to be sewn S, the image element GE is not particularly limited and may be an image element other than the hole 7.

[0119] In the above embodiment, the second image 72 is transformed to match the first image 71. However, the first image 71 may be transformed to match the second image 72. In addition, either of two images to be stitched together may be set as the first image 71 or the second image 72. In the image positioning process S2A, only positional alignment may be performed without transforming the second image 72.

[0120] In the above embodiment, each pixel constituting the composite image 75 is adopted from either the first image 71 or the second image 72. However, for each pixel corresponding to the overlapping region JA of the composite image 75, both the first image 71 and the second image 72 may be used. For example, for the overlapping region JA, averaging processing may be performed using both the first image 71 and the second image 72. In addition, for the overlapping region JA, an image obtained by binarizing both the first image 71 and the second image 72 and then taking a logical AND or a logical OR of the two images may be adopted.

[0121] The present disclosure includes the following aspects. (1) An image processing device including: a plurality of imaging devices arranged at intervals in a horizontal plane so as to be aligned with a sewing machine body including a head configured to perform sewing and a holding member configured to hold and move an object to be sewn; and an image processing unit configured to detect a recognition object of the object to be sewn on the basis of images of the object to be sewn captured by the plurality of imaging devices, wherein the image processing unit includes a composition processing unit configured to stitch together the images captured by the plurality of imaging devices, and a detection processing unit configured to detect a plurality of the recognition objects from a composite image obtained through the stitching by the composition processing unit. (2) The image processing device according to the above (1), wherein each of the plurality of imaging devices is arranged to have an overlapping region in which its imaging field of view partially overlaps with that of another adjacent imaging device, and the image processing unit detects image elements common to at least a portion of the recognition object, within the overlapping region of a first image and a second image to be stitched together, transforms the second image such that corresponding image elements in the overlapping region match each other, and stitches together the transformed second image and the first image. (3) The image processing device according to the above (2), wherein the composition processing unit sets a stitching position in the overlapping region such that the entire image elements detected in the overlapping region are included in either the first image or the second image. (4) The image processing device according to the above (2) or (3), wherein the composition processing unit extracts the image elements in contact with an end edge E on the second image side in the overlapping region, and sets a first mask region excluding the extracted image elements, as the first image, sets a second mask region including the entire image elements in contact with the end edge E and adjacent to the first mask region, as the second image, and stitches together the first image and the second image by using a boundary between the first mask region of the first image and the second mask region of the second image as the stitching position. (5) The image processing device according to any one of the above (2) to (4), wherein the image elements are holes formed to be distributed over the object to be sewn, and the recognition object is an array of the plurality of holes existing in a predetermined pattern in the object to be sewn. (6) The image processing device according to any one of the above (1) to (5), wherein the plurality of imaging devices are arranged along at least one of a longitudinal direction and a width direction of a sewing area where the object to be sewn can be moved by the holding member and sewn by the head. (7) The image processing device according to any one of the above (1) to (4), further including a plurality of illumination devices arranged on both outer sides of an array of the plurality of imaging devices and between the plurality of imaging devices of the array, respectively. (8) A sewing machine including: a sewing machine body including a head configured to perform sewing and a holding member configured to hold and move an object to be sewn; the image processing device according to any one of the above (1) to (7); and a control device configured to control the sewing machine body on the basis of a processing result of the image processing device. (9) An image processing method including: stitching together images of an object to be sewn captured by a plurality of imaging devices arranged at intervals in a predetermined direction in a horizontal plane so as to be aligned with a sewing machine body including a head configured to perform sewing and a holding member configured to hold and move an object to be sewn; and detecting a plurality of recognition objects of the object to be sewn from a composite image obtained through the stitching.

[0122] The present application is based on Japanese Patent Application No. 2023-108671 filed on June 30, 2023, the contents of which are incorporated herein by reference.

Claims

1. An image processing device comprising: a plurality of imaging devices arranged at intervals in a horizontal plane so as to be aligned with a sewing machine body comprising a head configured to perform sewing and a holding member configured to hold and move an object to be sewn; and an image processing unit configured to detect a recognition object of the object to be sewn on the basis of images of the object to be sewn captured by the plurality of imaging devices, wherein the image processing unit comprises a composition processing unit configured to stitch together the images captured by the plurality of imaging devices, and a detection processing unit configured to detect a plurality of the recognition objects from a composite image obtained through the stitching by the composition processing unit.

2. The image processing device according to claim 1, wherein each of the plurality of imaging devices is arranged to have an overlapping region in which its imaging field of view partially overlaps with that of another adjacent imaging device, and the image processing unit detects image elements common to at least a portion of the recognition object, within the overlapping region of a first image and a second image to be stitched together, transforms the second image such that corresponding image elements in the overlapping region match each other, and stitches together the transformed second image and the first image.

3. The image processing device according to claim 2, wherein the composition processing unit sets a stitching position in the overlapping region such that the entire image elements detected in the overlapping region are included in either the first image or the second image.

4. The image processing device according to claim 3, wherein the composition processing unit extracts the image elements in contact with an end edge E on the second image side in the overlapping region, and sets a first mask region excluding the extracted image elements, as the first image, sets a second mask region comprising the entire image elements in contact with the end edge E and adjacent to the first mask region, as the second image, and stitches together the first image and the second image by using a boundary between the first mask region of the first image and the second mask region of the second image as the stitching position.

5. The image processing device according to any one of claims 2 to 4, wherein the image elements are holes formed to be distributed over the object to be sewn, and the recognition object is an array of the plurality of holes existing in a predetermined pattern in the object to be sewn.

6. The image processing device according to any one of claims 1 to 4, wherein the plurality of imaging devices are arranged along at least one of a longitudinal direction and a width direction of a sewing area where the object to be sewn can be moved by the holding member and sewn by the head.

7. The image processing device according to any one of claims 1 to 4, further comprising a plurality of illumination devices arranged on both outer sides of an array of the plurality of imaging devices and between the plurality of imaging devices of the array, respectively.

8. A sewing machine comprising: a sewing machine body comprising a head configured to perform sewing and a holding member configured to hold and move an object to be sewn; the image processing device according to any one of claims 1 to 4; and a control device configured to control the sewing machine body on the basis of a processing result of the image processing device.

9. An image processing method comprising: stitching together images of an object to be sewn captured by a plurality of imaging devices arranged at intervals in a predetermined direction in a horizontal plane so as to be aligned with a sewing machine body comprising a head configured to perform sewing and a holding member configured to hold and move an object to be sewn; and detecting a plurality of recognition objects of the object to be sewn from a composite image obtained through the stitching.

Images