Image processing apparatus, sewing machine, and image processing method

Abstract

An image processing apparatus (40) comprises: multiple imaging devices (41) that are arranged at intervals within a horizontal plane so as to be lined up with respect to a sewing machine body (10) which is equipped with a head (11D) for performing sewing and a holding member (15) for holding and moving a sewing object (S); and an image processing unit (42) that detects a recognition target included in the sewing object (S), on the basis of images of the sewing object (S) which are captured by the multiple imaging devices (41). The image processing unit (42) comprises: a synthesis processing unit (44B) that joins together the images captured by the multiple imaging devices (41); and a detection processing unit (44C) that detects a plurality of recognition targets (UP) from a synthetic image (75) obtained by the joining together by the synthesis processing unit (44B).

Description

Image processing device, sewing machine, and image processing method

[0001] The present disclosure relates to an image processing device, a sewing machine, and an image processing method.

[0002] BACKGROUND ART In order to enhance the design of an object to be sewn, stitches may be formed on the object to be sewn. Patent Document 1 discloses a technique for forming stitches on an upholstery material used for a vehicle seat.

[0003] Japanese Patent Application Publication No. 2013-162957

[0004] Upholstery materials used for vehicle seats are thick and elastic. When stitches are formed on a thick and elastic sewing object, the sewing object may shrink and the surface of the sewing object may be displaced. For example, when a second stitch is formed after a first stitch is formed, it is desirable to form the second stitch at a target position on the sewing object in accordance with the displacement of the surface of the sewing object caused by the formation of the first stitch.

[0005] One way to form stitches at target positions on a sewing object is to photograph the surface of the sewing object before sewing and detect displacement on the surface of the sewing object. If multiple points along the stitch line are photographed one by one in sequence and displacement detection is performed for each point, it takes time for the photographing and image recognition processing. Since the displacement changes each time a stitch is formed, photographing and displacement detection are performed each time a stitch is formed, which increases the processing time for the entire sewing operation.

[0006] An aspect of the present disclosure aims to reduce the processing time of the entire sewing operation.

[0007] According to a first aspect of the present disclosure, there is provided an image processing device comprising: a plurality of imaging devices arranged at intervals in a horizontal plane in line with a sewing machine body having a head for sewing and a holding member for holding and moving an object to be sewn; and an image processing unit that detects a recognition target possessed by an object to be sewn based on images of the object to be sewn captured by the plurality of imaging devices, wherein the image processing unit comprises a synthesis processing unit that joins together images of the plurality of imaging devices, and a detection processing unit that detects the plurality of recognition targets from the composite image joined together by the synthesis processing unit.

[0008] According to a second aspect of the present disclosure, there is provided a sewing machine comprising a sewing machine body having a head for sewing and a holding member for holding and moving an object to be sewn, an image processing device according to the first aspect, and a control device for controlling the sewing machine body based on the processing results of the image processing device.

[0009] According to a third aspect of the present disclosure, there is provided an image processing method including stitching together images of an object to be sewn captured by a plurality of imaging devices arranged at intervals in a predetermined direction in a horizontal plane in line with a sewing machine body having a head for sewing and a holding member for holding and moving the object to be sewn, and detecting a plurality of recognition targets possessed by the object to be sewn from the stitched composite image.

[0010] According to the aspects of the present disclosure, the processing time for the entire sewing operation can be reduced.

[0011] FIG. 1 is a perspective view of a sewing machine according to the present embodiment. FIG. 2 is a perspective view of a portion of the sewing machine according to the present embodiment. FIG. 3 is a cross-sectional view of a portion of a sewing object according to the present embodiment. FIG. 4 is a plan view of the sewing object according to the present embodiment. FIG. 5 is a cross-sectional view of a portion of the sewing object according to the present embodiment. FIG. 6 is a plan view illustrating an installation position of an imaging device according to the present embodiment. FIG. 7 is a plan view illustrating an example of the arrangement of an imaging device according to the present embodiment. FIG. 8 is a functional block diagram of a sewing machine according to the present embodiment. FIG. 9 is a flowchart illustrating a sewing operation of the sewing machine according to the present embodiment. FIG. 10 is a flowchart illustrating an image joining process according to the present embodiment. FIG. 11 is a schematic diagram illustrating an image positioning process in the image joining process. FIG. 12 is a schematic diagram illustrating a mask area setting process in the image joining process. FIG. 13 is a schematic diagram illustrating a synthesis process in the image joining process. FIG. 14 is a schematic diagram illustrating a recognition target detection process.

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings, but the present disclosure is not limited thereto. The components of the embodiments described below can be combined as appropriate. In addition, some components may not be used.

[0013] In this embodiment, a local coordinate system is defined for the sewing machine 1. In the following description, the local coordinate system defined for the sewing machine 1 will be referred to as the sewing machine coordinate system as appropriate. The sewing machine coordinate system is defined by an XYZ Cartesian coordinate system. In this embodiment, the positional relationship of each part will be described based on the sewing machine coordinate system. The direction parallel to the X axis in a predetermined plane is defined as the X-axis direction. The direction parallel to the Y axis in a predetermined plane that is perpendicular to the X axis is defined as the Y-axis direction. The direction parallel to the Z axis that is perpendicular to the predetermined plane is defined as the Z-axis direction. Furthermore, the direction of rotation or tilt around the X axis is defined as the θX direction. The direction of rotation or tilt around the Y axis is defined as the θY direction. The direction of rotation or tilt around the Z axis is defined as the θZ direction. Furthermore, in this embodiment, the plane including the X axis and the Y axis is defined as the XY plane as appropriate. The plane including the X axis and the Z axis is defined as the XZ plane as appropriate. The plane including the Y axis and the Z axis is defined as the YZ plane as appropriate. The XY plane is parallel to a predetermined plane. The XY plane, XZ plane, and YZ plane are orthogonal to each other. In this embodiment, the XY plane and the horizontal plane are parallel. The Z axis direction is the up-down direction. The +Z direction is the upward direction, and the -Z direction is the downward direction. The XY plane may be inclined with respect to the horizontal plane.

[0014] [Sewing Machine] Fig. 1 is a perspective view showing a sewing machine 1 according to this embodiment. Fig. 2 is a perspective view showing a portion of the sewing machine 1 according to this embodiment. In this embodiment, the sewing machine 1 is an electronic cycle sewing machine. The sewing machine 1 includes a sewing machine main body 10, an operation device 20, a control device 30, and an image processing device 40. The operation device 20 is operated by an operator. The control device 30 controls the sewing machine main body 10. The image processing device 40 captures an image of the sewing object S and performs image recognition. The image processing device 40 includes a plurality of image capturing devices 41 and an image processing unit 42.

[0015] The sewing machine body 10 is mounted on the upper surface of the table 2. As shown in FIG. 2, the sewing machine body 10 has a sewing machine frame 11, a needle bar 12, a needle plate 13, a holding member 15, an actuator 16, an actuator 17, and an actuator 18. The needle bar 12 and the needle plate 13 are each supported by the sewing machine frame 11. The holding member 15 is supported by the sewing machine frame 11 via a support member 14. The actuator 16 (see FIG. 1) generates power to move the needle bar 12. The actuator 17 (see FIG. 1) generates power to move the holding member 15. The actuator 18 (see FIG. 1) generates power to move at least a portion of the holding member 15.

[0016] The sewing machine frame 11 has a horizontal arm 11A, a bed 11B, a vertical arm 11C, and a head 11D. The horizontal arm 11A extends in the Y-axis direction. The bed 11B (see FIG. 1) is located below the horizontal arm 11A. The vertical arm 11C (see FIG. 1) is located so as to connect the end of the horizontal arm 11A on the +Y side to the bed 11B. The head 11D is located on the -Y side of the horizontal arm 11A.

[0017] Needle bar 12 holds sewing needle 3. Needle bar 12 holds sewing needle 3 so that sewing needle 3 is parallel to the Z axis. Needle bar 12 is supported by head 11D so as to be movable in the Z axis direction.

[0018] The needle plate 13 supports the sewing object S. The needle plate 13 supports a holding member 15. The needle plate 13 is supported by the bed 11B (see FIG. 1 ). The needle plate 13 is disposed below the holding member 15.

[0019] The holding member 15 holds the sewing object S. The holding member 15 is capable of moving the sewing object S within an XY plane that includes a sewing position Ps directly below the sewing needle 3. The holding member 15 is capable of holding the sewing object S within an XY plane that includes a position directly below the imaging device 41. The holding member 15 moves within the XY plane based on the sewing data while holding the sewing object S. As a result, a stitch CH (see FIG. 5) is formed on the sewing object S that has passed the sewing position Ps. The holding member 15 is supported by the horizontal arm 11A via the support member 14.

[0020] The holding member 15 has a presser member 15A and a lower plate 15B facing the presser member 15A. The presser member 15A is a frame-shaped member. The presser member 15A is movable in the Z-axis direction. The lower plate 15B is disposed below the presser member 15A. The holding member 15 holds the sewing object S by sandwiching the sewing object S between the presser member 15A and the lower plate 15B.

[0021] When the presser member 15A moves in the +Z direction, the presser member 15A and the lower plate 15B separate. This allows the operator to place the sewing object S between the presser member 15A and the lower plate 15B. When the presser member 15A moves in the -Z direction with the sewing object S placed between the presser member 15A and the lower plate 15B, the sewing object S is sandwiched between the presser member 15A and the lower plate 15B. This causes the sewing object S to be held by the holding member 15. Furthermore, when the presser member 15A moves in the +Z direction, the holding of the sewing object S by the holding member 15 is released. This allows the operator to remove the sewing object S from between the presser member 15A and the lower plate 15B.

[0022] 1, the actuator 16 generates power to move the needle bar 12 in the Z-axis direction. The actuator 16 includes a pulse motor. The actuator 16 is disposed on the horizontal arm 11A.

[0023] A horizontal arm shaft extending in the Y-axis direction is disposed inside horizontal arm 11A. Actuator 16 is connected to the +Y side end of the horizontal arm shaft. The -Y side end of the horizontal arm shaft is connected to needle bar 12 via a power transmission mechanism disposed inside head 11D. Operation of actuator 16 causes the horizontal arm shaft to rotate. Power generated by actuator 16 is transmitted to needle bar 12 via the horizontal arm shaft and the power transmission mechanism. As a result, sewing machine needle 3 held by needle bar 12 moves back and forth in the Z-axis direction.

[0024] A timing belt extending in the Z-axis direction is disposed inside the vertical arm 11C. A bed shaft extending in the Y-axis direction is disposed inside the bed 11B. Pulleys are disposed on both the horizontal arm shaft and the bed shaft. The timing belt is wound around the pulley disposed on the horizontal arm shaft and the pulley disposed on the bed shaft. The horizontal arm shaft and the bed shaft are connected via a power transmission mechanism including the timing belt.

[0025] A shuttle is disposed inside the bed 11B. A bobbin placed in a bobbin case is housed in the shuttle. The horizontal arm shaft and the bed shaft are rotated by the operation of the actuator 16. The power generated by the actuator 16 is transmitted to the shuttle via the horizontal arm shaft, timing belt, and bed shaft. As a result, the shuttle rotates in synchronization with the reciprocating movement of the needle bar 12 in the Z-axis direction.

[0026] The actuator 17 generates power to move the holding member 15 in the XY plane. The actuator 17 includes a pulse motor. The actuator 17 includes an X-axis motor 17X and a Y-axis motor 17Y. The X-axis motor 17X (see FIG. 8) generates power to move the holding member 15 in the X-axis direction. The Y-axis motor 17Y (see FIG. 8) generates power to move the holding member 15 in the Y-axis direction. The actuator 17 is disposed inside the bed 11B.

[0027] The power generated by the actuator 17 is transmitted to the holding member 15 via the support member 14. This allows the holding member 15 to move in both the X-axis and Y-axis directions between the sewing needle 3 and the needle plate 13. By operating the actuator 17, the holding member 15 can move while holding the sewing object S within an XY plane that includes the sewing position Ps directly below the sewing needle 3.

[0028] The actuator 18 generates power to move the presser member 15A of the holding member 15 in the Z-axis direction. The actuator 18 includes a pulse motor. When the presser member 15A moves in the +Z direction, the presser member 15A and the lower plate 15B are separated from each other. When the presser member 15A moves in the -Z direction, the sewing object S is sandwiched between the presser member 15A and the lower plate 15B.

[0029] As shown in FIG. 2 , the sewing machine body 10 has an intermediate presser member 19 that is arranged around the sewing needle 3. The intermediate presser member 19 presses down on the sewing object S around the sewing needle 3. The intermediate presser member 19 is supported by the head 11D in a state that allows it to move in the Z-axis direction. An intermediate presser motor is arranged inside the head 11D. The intermediate presser motor generates power that moves the intermediate presser member 19 in the Z-axis direction. Operation of the intermediate presser motor causes the intermediate presser member 19 to move in the Z-axis direction in synchronization with the needle bar 12. The intermediate presser member 19 prevents the sewing object S from lifting up due to movement of the sewing needle 3.

[0030] As shown in Fig. 1, the operating device 20 accepts operation inputs from an operator. The sewing machine 1 is operated by operating the operating device 20. The operating device 20 includes an operation panel 21 and an operation pedal 22. The operation panel 21 is mounted on the top surface of the table 2. The operation pedal 22 is disposed below the table 2. The operator operates the operation pedal 22 with his / her foot. The sewing machine 1 is operated by the operator operating at least one of the operation panel 21 and the operation pedal 22.

[0031] Before starting the sewing process, the image processing device 40 acquires images of the sewing object S using multiple imaging devices 41, detects the target sewing position using the image processing unit 42, and performs processing to calculate correction data for displacement caused by the stitch CH that has already been created, etc.

[0032] The plurality of imaging devices 41 capture images of the sewing object S held by the holding member 15. Each imaging device 41 includes an optical system and an image sensor that receives light incident thereon via the optical system. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0033] The multiple imaging devices 41 are disposed above the needle plate 13 and the holding member 15. Each of the multiple imaging devices 41 captures an image of at least a portion of the sewing object S held by the holding member 15 from above. The number of imaging devices 41 is two or more, but is not particularly limited. Details such as the arrangement and imaging field of view of the multiple imaging devices 41 will be described later.

[0034] The image processing unit 42 includes a computer system. The image processing unit 42 processes images related to the sewing object S. The image processing unit 42 detects a recognition target that the sewing object S has, based on the images of the sewing object S captured by the multiple imaging devices 41.

[0035] [Sewing Object] Fig. 3 is a cross-sectional view showing a part of the sewing object S according to this embodiment. Fig. 4 is a plan view showing the sewing object S according to this embodiment. Figs. 3 and 4 show the sewing object S before sewing processing. In this embodiment, the sewing object S is a cover material used for a vehicle seat.

[0036] 3, the sewing object S has a surface material 4, a padding material 5, and a backing material 6. A hole 7 is formed in the surface material 4.

[0037] The surface of the surface material 4 is the seating surface that comes into contact with the occupant when the occupant sits in the vehicle seat. The surface material 4 includes at least one of woven fabric, nonwoven fabric, and leather. The padding material 5 has elasticity. The padding material 5 includes, for example, urethane resin. The backing material 6 includes at least one of woven fabric, nonwoven fabric, and leather.

[0038] As shown in FIG. 4 , a plurality of holes 7 are provided in the surface material 4. The holes 7 are formed so as to be scattered throughout the sewing object S. The holes 7 are arranged in a regular pattern. That is, the sewing object S includes a plurality of reference patterns DPh. The reference patterns DPh are configured by a regular arrangement of a plurality of holes 7. In the example of FIG. 4 , the reference pattern DPh is configured by 17 holes 7.

[0039] As shown in Figure 4, the reference patterns DPh are arranged on the surface material 4 at intervals. The reference patterns DPh are arranged at equal intervals in both the X-axis direction and the Y-axis direction. Between adjacent reference patterns DPh in the X-axis direction, a reference pattern DPh at a different position in the Y-axis direction is arranged. No holes 7 are formed between adjacent reference patterns DPh.

[0040] A target pattern RP of the stitch CH to be formed on the sewing object S is defined in the area between the reference patterns DPh. Fig. 4 shows a virtual target pattern RP of the stitch CH on the sewing object S before sewing. The sewing machine 1 forms the stitch CH so as to follow each target pattern RP in accordance with a preset sewing pattern.

[0041] Furthermore, a plurality of recognition targets UP (UP1, UP2, UP3, UP4, UP5, UP6, and UP7) are arranged on the sewing object S. In this embodiment, the recognition targets UP are an array of a plurality of holes 7 that exist in a predetermined pattern on the sewing object S. Specifically, the recognition targets UP are part of the reference pattern DPh. In the example of FIG. 4 , the recognition targets UP (UP1, UP2, UP3, UP4, UP5, UP6, and UP7) are a pattern (an array of holes 7) that includes one acute corner of the reference pattern DPh that has a diamond-shaped outer shape. The recognition targets UP are patterns that can be identified by a pattern matching method, which is a type of image processing method.

[0042] With reference to FIG. 5 , displacement that occurs on the surface of the sewing object S when a stitch CH is formed on the sewing object S, which has thickness and elasticity, will be described. FIG. 5 is a cross-sectional view showing an example of the sewing object S according to this embodiment. FIG. 5 shows the sewing object S after a sewing process has been performed. The sewing object S is thick and elastic. When a stitch CH is formed on the sewing object S, which has thickness and elasticity, there is a high possibility that the sewing object S will shrink, as shown in FIG. 5 . When the sewing object S shrinks, there is a possibility that the surface of the sewing object S will displace. When the surface of the sewing object S displaces, there is a high possibility that the target position of the stitch CH defined on the surface of the sewing object S will displace in the XY plane. If the target position of the stitch CH displaces in the XY plane, it will be difficult to form the stitch CH at the target position when the holding member 15 is moved to the position coordinates of the design target pattern RP.

[0043] Therefore, in this embodiment, the displacement amount (correction amount for correcting the displacement amount) is acquired by the image processing device 40. Even if the sewing object S shrinks due to the formation of the stitch CH and the surface of the sewing object S is displaced, the holding member 15 is moved in accordance with the displacement amount acquired by the image processing device 40 so that the next stitch CH is formed at the target position. The image processing device 40 (see FIG. 1) detects the recognition target UP from an image of the sewing object S and acquires the correction amount based on the displacement of the recognition target UP.

[0044] [Imaging Device] FIG. 6 is a plan view illustrating the mounting position of the imaging device 41 according to this embodiment.

[0045] The multiple imaging devices 41 are arranged at intervals in a horizontal plane (XY plane) so as to be aligned with the sewing machine body 10. The multiple imaging devices 41 are arranged within the sewing area SA. The sewing area SA is the range in the XY plane within which the sewing object S can be moved by the holding member 15 and sewn by the head 11D. The head 11D (i.e., sewing position Ps) is located in the center of the sewing area SA. The multiple imaging devices 41 are arranged along the head 11D of the sewing machine body 10.

[0046] 6, the sewing machine 1 is equipped with imaging units 50A and 50B. The imaging units 50A and 50B are provided on both sides of the head 11D in the X-axis direction, and each is disposed adjacent to the head 11D. A plurality of imaging devices 41 provided in the image processing device 40 can be attached to the imaging units 50A and 50B. The imaging units 50A and 50B are disposed above the sewing area SA. The imaging units 50A and 50B are each fixed to the sewing machine 1.

[0047] The imaging unit 50A on the -X direction side has three mounting portions 51A, 51B, and 51C. Each of the mounting portions 51A, 51B, and 51C can mount one imaging device 41. The mounting portions 51A and 51B are arranged at an interval in the X-axis direction. The mounting portions 51B and 51C are arranged at an interval in the Y-axis direction.

[0048] The imaging unit 50B on the +X direction side has three mounting portions 51D, 51E, and 51F. Each of the mounting portions 51D, 51E, and 51F can mount one imaging device 41. The mounting portions 51D and 51E are arranged at an interval in the X-axis direction. The mounting portions 51D and 51F are arranged at an interval in the Y-axis direction.

[0049] Each of the mounting portions 51A, 51B, 51C, 51D, 51E, and 51F is located inside the sewing area SA in the X and Y directions. Each of the mounting portions 51A, 51B, 51C, 51D, 51E, and 51F can hold the imaging device 41 above the sewing area SA. Therefore, regardless of which mounting portion the imaging device 41 is attached to, the sewing machine 1 can position the sewing object S at the imaging position of the imaging device 41 using the holding member 15. In the example of Figure 6, the mounting portions 51A and 51B of the imaging unit 50A and the mounting portions 51D and 51E of the imaging unit 50B are arranged on a straight line in the X-axis direction with respect to the sewing position Ps.

[0050] The imaging units 50A and 50B provide a total of six mounting positions (mounting portions 51A, 51B, 51C, 51D, 51E, and 51F) on the sewing machine 1. In this embodiment, the imaging devices 41 may be mounted at all six mounting positions, or at some (two or more adjacent positions). The sewing machine 1 is capable of selecting and changing the mounting position of the imaging device 41 in accordance with the shape of the sewing object S and the shape of the stitches CH to be formed on the sewing object S.

[0051] The imaging units 50A and 50B also hold a plurality of lighting devices 43A and 43B included in the image processing device 40. The imaging units 50A and 50B hold the lighting devices 43A and 43B in a fixed state above the sewing area SA. The imaging unit 50A has three lighting devices 43A, one on each side of the arrangement of the mounting portions 51A and 51B along the X-axis direction and one between the arrangements. Each lighting device 43A extends linearly along the Y-axis direction. Each lighting device 43A irradiates an imaging field of view set below the mounting portions 51A, 51B, and 51C with light in the form of epi-illumination. The imaging unit 50B has three lighting devices 43B, one on each side of the arrangement of the mounting portions 51D and 51F along the Y-axis direction and one between the arrangements. Each lighting device 43B extends linearly along the X-axis direction. Each of the illumination devices 43B irradiates light in the form of epi-illumination onto an imaging field of view set below the attachment portions 51D, 51E, 51F.

[0052] As described above, the image processing device 40 according to the present embodiment includes a plurality of image capturing devices 41 arranged side by side with respect to the sewing machine body 10 (head 11D). The plurality of image capturing devices 41 are arranged at intervals in a horizontal plane (in the XY plane). In the example of Fig. 6, the plurality of image capturing devices 41 can be arranged along at least one of the longitudinal direction (X-axis direction) and the lateral direction (Y-axis direction) of the sewing area SA.

[0053] 7 is a plan view illustrating an example of the arrangement of the imaging devices 41 according to this embodiment. The imaging devices 41 can be arranged, for example, as shown in FIG.

[0054] 7, two imaging devices 41 are attached to each of attachment portions 51A and 51B of imaging unit 50A. The two imaging devices 41 of imaging unit 50A are arranged along the longitudinal direction (X-axis direction) of sewing area SA. Two imaging devices 41 are attached to each of attachment portions 51D and 51F of imaging unit 50B. The imaging device 41 of imaging unit 50B is arranged along the lateral direction (Y-axis direction) of sewing area SA.

[0055] The imaging device 41 captures an image of at least a portion of the sewing object S arranged within its imaging field of view. The imaging device 41 captures an image of at least a portion of the sewing object S held by the holding member 15 from above. Each of the multiple imaging devices 41 is arranged to have an overlapping area JA. The overlapping area JA is an area where the imaging field of view of an adjacent imaging device 41 partially overlaps. Specifically, the imaging device 41 of the mounting unit 51A has an imaging field of view 60A. The imaging device 41 of the mounting unit 51B has an imaging field of view 60B. The imaging field of view 60A and the imaging field of view 60B have a rectangular shape and an overlapping area JA extending in the Y-axis direction. The imaging field of view 60A and the imaging field of view 60B form an enlarged imaging area 61A. The enlarged imaging area 61A is an area where the imaging fields of view of the multiple imaging devices 41 that share the overlapping area JA are combined. The imaging fields 60A and 60B aligned in the X-axis direction form a wide enlarged imaging area 61A extending in the X-axis direction.

[0056] Furthermore, the imaging device 41 of the mounting portion 51D has an imaging field of view 60D. The imaging device 41 of the mounting portion 51F has an imaging field of view 60F. The imaging fields 60D and 60F have a rectangular shape and have an overlapping area JA extending in the X-axis direction. The imaging fields 60D and 60F form an enlarged imaging area 61B. The imaging fields 60D and 60F are aligned in the Y-axis direction, forming a wide enlarged imaging area 61B extending in the Y-axis direction.

[0057] 7 , the size of the imaging field of view of the imaging device 41, the size of the overlapping area JA (the spacing between the mounting portions), and the number of imaging devices 41 are set according to the dimensions of the target pattern RP (see FIG. 4 ) of the sewing object S. That is, the enlarged imaging areas 61A, 61B formed by the multiple imaging devices 41 are designed to have a shape that encompasses the entire target pattern RP of the sewing object S. The sewing object S on which a stitch CH extending in the X-axis direction is to be formed can be imaged by being positioned in the enlarged imaging area 61A of the imaging unit 50A. The sewing object S on which a stitch CH extending in the Y-axis direction is to be formed can be imaged by being positioned in the enlarged imaging area 61B of the imaging unit 50B. Therefore, regardless of the direction in which the stitch CH extending in the sewing machine 1 is to be formed, the entire target pattern RP can be imaged at once without changing the mounting position of the imaging device 41.

[0058] The position of each imaging device 41 is fixed at one of six mounting positions. The relative position of the imaging device 41 and the sewing machine frame 11 is fixed. The relative position of the optical axis of the imaging device 41 and the sewing needle 3 (sewing position Ps) in the XY plane is fixed. Relative position data indicating the relative position of the center of the enlarged imaging areas 61A, 61B and the sewing needle 3 in the XY plane is known data that can be derived from the design data of the sewing machine 1.

[0059] The position of the image captured by the imaging device 41 is defined in the camera coordinate system. A defined transformation formula or transformation matrix is ​​used to convert the position of the image defined in the camera coordinate system into the position of the image defined in the sewing machine coordinate system.

[0060] In the example shown in FIG. 7 , the head 11D is provided with a second imaging device 33. As shown in FIG. 6 , the head 11D has mounting portions 33A and 33B, each spaced apart from the sewing position Ps in the X-axis direction. The second imaging device 33 is attached to either one or both of the mounting portions 33A and 33B. In FIG. 7 , one second imaging device 33 is attached to the mounting portion 33A. The second imaging device 33 has a smaller imaging field of view and higher resolution (number of pixels per unit area) than the imaging device 41. Therefore, the multiple imaging devices 41 are used for high-speed recognition applications in which multiple recognition targets UP are simultaneously imaged, as described below. The second imaging device 33 images one recognition target UP per image, and can be used for precision recognition applications in which particularly precise positioning is required.

[0061] [Configuration Related to Control of Sewing Machine] Fig. 8 is a functional block diagram showing the sewing machine 1 according to this embodiment. The configuration related to control of the sewing machine 1 will be described with reference to Fig. 8 .

[0062] (Control Device) The control device 30 outputs control signals to control the sewing machine 1. The control device 30 includes a computer system. The control device 30 has an input / output interface, a storage device including a non-volatile memory and a volatile memory, and an arithmetic processing device including a processor. The non-volatile memory is, for example, a read-only memory (ROM) or storage. The volatile memory is, for example, a random access memory (RAM). The processor is, for example, a central processing unit (CPU). The control device 30 controls the sewing machine 1 in accordance with a computer program stored in the storage device.

[0063] The control device 30 is connected to the actuator 16, the actuator 17, the actuator 18, the operation device 20, and the image processing unit 42. The actuator 16 moves the sewing needle 3 in the Z-axis direction. The actuator 17 moves the holding member 15 within the XY plane. The actuator 18 moves the presser member 15A of the holding member 15 in the Z-axis direction.

[0064] Further, a drive amount sensor 31 and a drive amount sensor 32 are connected to the control device 30. The drive amount sensor 31 detects the drive amount of the actuator 16. The drive amount sensor 32 detects the drive amount of the actuator 17.

[0065] The control device 30 controls the actuator 16 based on the detection data of the drive amount sensor 31. The control device 30 determines, for example, the operation timing of the actuator 16 based on the detection data of the drive amount sensor 31.

[0066] The drive amount sensor 32 includes an X-axis sensor 32X and a Y-axis sensor 32Y. The X-axis sensor 32X detects the drive amount of the X-axis motor 17X of the actuator 17. The Y-axis sensor 32Y detects the drive amount of the Y-axis motor 17Y of the actuator 17. The X-axis sensor 32X includes an encoder that detects the rotation amount of the X-axis motor 17X. The Y-axis sensor 32Y includes an encoder that detects the rotation amount of the Y-axis motor 17Y. The detection data of the drive amount sensor 32 is output to the control device 30. The control device 30 controls the actuator 17 based on the detection data of the drive amount sensor 32. The control device 30 feedback-controls the actuator 17 based on the detection data of the drive amount sensor 32 so that the holding member 15 moves to a target position.

[0067] The drive amount sensor 32 functions as a position sensor that detects the position of the holding member 15 in the XY plane. The drive amount of the actuator 17 and the movement amount of the holding member 15 correspond one-to-one.

[0068] The X-axis sensor 32X detects the amount of rotation of the X-axis motor 17X, thereby detecting the amount of movement of the holding member 15 in the X-axis direction from the origin in the sewing machine coordinate system. The Y-axis sensor 32Y detects the amount of rotation of the Y-axis motor 17Y, thereby detecting the amount of movement of the holding member 15 in the Y-axis direction from the origin in the sewing machine coordinate system.

[0069] The control device 30 calculates the position of the holding member 15 in the XY plane based on the detection data of the drive amount sensor 32. The amount of movement of the holding member 15 from the origin in the XY plane is detected based on the detection data of the drive amount sensor 32. The control device 30 calculates the position of the holding member 15 in the XY plane based on the detected amount of movement of the holding member 15.

[0070] The control device 30 stores the sewing data in a storage device. The sewing data is known data that can be derived from design data of the sewing object S, such as CAD (Computer Aided Design) data. The sewing data is referenced by the control device 30 during sewing processing. The sewing processing refers to processing for forming stitches CH on the sewing object S. The sewing data includes a target pattern RP (see FIG. 4) of the stitches CH to be formed on the sewing object S, position coordinates of the recognition target UP, and movement conditions for the holding member 15.

[0071] The target pattern RP defines the target shape of the stitch CH to be formed on the sewing object S and the target position of the stitch CH in the sewing machine coordinate system.

[0072] The movement conditions of the holding member 15 include a movement trajectory of the holding member 15 defined in the sewing machine coordinate system. The movement trajectory of the holding member 15 includes a movement trajectory of the holding member 15 in the XY plane. The movement conditions of the holding member 15 are determined based on the target pattern RP.

[0073] Therefore, as shown in Figure 4, when performing multiple sewing processes to form multiple stitches CH, a first sewing process is performed, and then a second sewing process is performed after the first sewing process. The first sewing process is a process of forming a first stitch CH on the sewing object S based on a first target pattern RP1. The second sewing process is a process of forming a second stitch CH on the sewing object S based on a second target pattern RP2. Thereafter, the third sewing process to the Nth sewing process are performed sequentially depending on the number N of stitches CH to be formed. Figure 4 shows an example where N = 10, and the first target pattern RP1 to the tenth target pattern RP10 are defined.

[0074] The control device 30 is connected to the image processing unit 42 via an input / output interface. The control device 30 outputs a control command for controlling the actuator 17 that moves the holding member 15 based on the processing result of the image processing unit 42.

[0075] (Image Processing Unit) The image processing unit 42 includes a computer system. The image processing unit 42 has a processor 44, a storage device 45 including a non-volatile memory and a volatile memory, and an input / output unit 46 including an input / output interface. The processor 44 is, for example, a central processing unit (CPU). The non-volatile memory is, for example, a read-only memory (ROM) or storage. The volatile memory is, for example, a random access memory (RAM). The image processing unit 42 captures images of the sewing object S using the multiple image capture devices 41 in accordance with a computer program stored in the storage device 45, and performs image processing to detect the recognition target UP from the images.

[0076] The image processing unit 42 is connected to the multiple image capture devices 41 and the multiple lighting devices 43A and 43B via the input / output unit 46. In Fig. 8, the input / output unit 46 is connected to the image capture devices 41 and the lighting device 43A of the image capture unit 50A via a relay device 47. The input / output unit 46 is also connected to the image capture devices 41 and the lighting device 43B of the image capture unit 50B via the relay device 47. The relay device 47 is a line concentrator for connecting multiple devices, such as a USB (Universal Serial Bus) hub.

[0077] The image processing unit 42 has an image acquisition unit 44A, a synthesis processing unit 44B, a detection processing unit 44C, and a correction amount calculation unit 44D. The image acquisition unit 44A, synthesis processing unit 44B, detection processing unit 44C, and correction amount calculation unit 44D are functional blocks realized by the processor 44 executing a computer program. The image acquisition unit 44A, synthesis processing unit 44B, detection processing unit 44C, and correction amount calculation unit 44D may be configured separately by dedicated processors.

[0078] The image acquisition unit 44A acquires images of the sewing object S from the multiple imaging devices 41. The imaging devices 41 capture images of the sewing object S and output the acquired images to the image processing unit 42. When the sewing object S is positioned at a predetermined imaging position by the control device 30, the image acquisition unit 44A controls the operation of the imaging device 41 corresponding to the imaging position and acquires the images of the sewing object S.

[0079] In FIG. 7 , an imaging position Pf is set at the center of the enlarged imaging areas 61A and 61B. An imaging operation is performed with the center of the holding member 15 (the sewing object S) positioned at the imaging position Pf of either the enlarged imaging areas 61A or 61B. For example, in the imaging unit 50A, when the sewing object S is positioned at the imaging position Pf of the enlarged imaging area 61A, the image acquisition unit 44A turns on the lighting device 43A and causes the imaging device 41 of the attachment unit 51A and the imaging device 41 of the attachment unit 51B to capture images substantially simultaneously. As a result, an image corresponding to the imaging field of view 60A and an image corresponding to the imaging field of view 60B are acquired. In the imaging unit 50B, when the sewing object S is positioned at the imaging position Pf of the enlarged imaging area 61B, the image acquisition unit 44A turns on the lighting device 43B and causes the imaging device 41 of the attachment unit 51D and the imaging device 41 of the attachment unit 51F to capture images substantially simultaneously. The imaging may be performed in sequence.

[0080] The composition processing unit 44B performs image composition processing to stitch together images from the multiple image capture devices 41. The composition processing unit 44B stitches together the multiple images that make up the enlarged image capture area (61A or 61B) based on the image portions of the overlapping area JA to generate a single composite image 75 (see FIG. 13 ). Therefore, the composite image 75 is an image that captures the range corresponding to the enlarged image capture area 61A or 61B. The enlarged image capture areas 61A and 61B are large enough to include multiple recognition targets UP on the sewing object S. Therefore, the composite image 75 is an image that captures multiple recognition targets UP. The composite image 75 preferably includes all recognition targets UP (UP1 to UP7) that are detected when forming one stitch CH.

[0081] The detection processing unit 44C detects a plurality of recognition targets UP from the composite image 75 stitched together by the composition processing unit 44B. The detection processing unit 44C detects the recognition targets UP in the composite image 75 by a pattern matching method, which is a type of image processing method. Here, the operator stores in advance in the storage device 45 a reference image (template) for recognizing the recognition targets UP of the sewing object S. The detection processing unit 44C acquires the reference image recorded in the storage device 45 and acquires the position coordinates of the recognition targets UP in the composite image 75 by matching with the reference image. The detection processing unit 44C sequentially detects the plurality of recognition targets UP included in the composite image 75 and acquires the position coordinates of each of the plurality of recognition targets UP.

[0082] The correction amount calculation unit 44D calculates correction data for correcting the displacement of the surface of the sewing object S based on the detection result by the detection processing unit 44C. The target pattern RP of the sewing data is generated on the assumption that the sewing object S has not shrunk, and is stored in the control device 30. When the sewing process is performed and the sewing object S shrinks and the surface of the sewing object S is displaced, the correction amount calculation unit 44D determines the amount of displacement of the recognition object UP detected by the detection processing unit 44C, and generates correction data for correcting the sewing data.

[0083] Specifically, the correction amount calculation unit 44D generates correction data so that the relative positions in the sewing machine coordinate system between the target pattern RP and the recognition target UP before the sewing object S shrinks coincide with the relative positions in the sewing machine coordinate system between the stitch CH actually formed on the sewing object S by the sewing process and the recognition target UP after the sewing object S shrinks.

[0084] The image processing unit 42 transmits the correction data calculated by the correction amount calculation unit 44D to the control device 30. The control device 30 executes the sewing process based on the correction data.

[0085] 9 is a flowchart for explaining the sewing operation of the sewing machine 1 according to this embodiment. The sewing operation of the sewing machine 1 includes an image capturing process S1, an image joining process S2, a recognition target detection process S3, a correction data calculation process S4, a sewing process S5, and an end determination process S6. The image processing method according to this embodiment is performed as part of the sewing operation of the sewing machine 1. The image processing method according to this embodiment includes the image joining process S2 and the recognition target detection process S3.

[0086] (Image Capture Process) The image capture process S1 is a process of capturing images of the sewing object S using the multiple image capture devices 41. Whether the image capture unit 50A captures an image of the enlarged image capture area 61A extending in the X-axis direction or the image capture unit 50B captures an image of the enlarged image capture area 61B extending in the Y-axis direction is set in advance by the operator or by the control device 30 based on sewing data. The control device 30 moves the holding member 15 that holds the sewing object S and positions the holding member 15 at a preset image capture position. When the holding member 15 is positioned at the corresponding image capture position Pf, the image processing unit 42 performs image capture using the multiple image capture devices 41 that have been set.

[0087] 7 will be described as an example in which images of the enlarged imaging area 61A are acquired by the imaging devices 41 of the imaging unit 50A. In this case, the image acquisition unit 44A acquires an image corresponding to the imaging field of view 60A and an image corresponding to the imaging field of view 60B by the imaging device 41 of the attachment part 51A and the imaging device 41 of the attachment part 51B.

[0088] (Image Joining Process) The image joining process S2 is a process of joining together images of the sewing object S captured by the multiple image capturing devices 41.

[0089] Fig. 10 is a flowchart showing the image joining process S2 according to this embodiment. Fig. 11 is a schematic diagram illustrating the image positioning process S2A in the image joining process S2. Fig. 12 is a schematic diagram illustrating the mask area setting process S2B in the image joining process S2. Fig. 13 is a schematic diagram illustrating the synthesis process S2C in the image joining process S2.

[0090] 11 , the image obtained by the imaging device 41 of the mounting unit 51A is referred to as a first image 71. The image obtained by the imaging device 41 of the mounting unit 51B is referred to as a second image 72. In this case, the second image 72 includes an area on the +X direction side of the first image 71. The first image 71 corresponds to the imaging field of view 60A, and the second image 72 corresponds to the imaging field of view 60B. The first image 71 and the second image 72 share an overlapping area JA.

[0091] The overlapping area JA between the first image 71 and the second image 72 includes image elements GE that are common to at least a portion of the recognition target UP. That is, as shown in Fig. 4, when the recognition target UP is an array of multiple holes 7 that exist in a predetermined pattern in the sewing object S, the image elements GE are images of the individual holes 7. The holes 7 are components that make up the recognition target UP and the reference pattern DPh. For ease of explanation, Figs. 11 to 13 show an example in which the arrangement of the holes 7 is simplified and the holes 7 are arranged vertically and horizontally in an array.

[0092] In detecting the recognition target UP by pattern matching, it is important that there are no missing or error image information in the image elements GE of the holes 7 that make up the recognition target UP. Therefore, in the image joining process S2 shown in Fig. 10, the first image 71 and the second image 72 are combined together so that the information of the image elements GE that make up the recognition target UP is preserved.

[0093] 10, the image stitching process S2 includes an image positioning process S2A, a mask area setting process S2B, and a synthesis process S2C. Prior to the image stitching process S2, the synthesis processing unit 44B may binarize each image obtained in the imaging process S1.

[0094] <Image Positioning Process> The image positioning process S2A is a process for positioning the multiple images obtained in the imaging process S1 based on the image elements GE in the overlapping area JA.

[0095] As shown in FIG. 11 , the composition processing unit 44B detects image elements GE included in the overlapping area JA of the first image 71 and image elements GE included in the overlapping area JA of the second image 72. Since the position coordinates of the imaging center of the imaging device 41 attached to the attachment parts 51A and 51B and the size of the imaging fields of view 60A and 60B in the sewing machine coordinate system are known, the extent of the overlapping area JA in each image is known. The composition processing unit 44B selects three or more image elements GE included in the overlapping area JA. In the example of FIG. 11 , the composition processing unit 44B divides the overlapping area JA into four parts vertically and horizontally, and selects one image element GE from each divided area, for a total of four image elements GE1 of interest.

[0096] The synthesis processing unit 44B deforms the second image 72 so that identical image elements GE in the overlapping area JA match each other. That is, the synthesis processing unit 44B deforms the second image 72 so that four target image elements GE1 in the first image 71 match four corresponding target image elements GE1 in the second image 72. The deformation of the second image 72 is performed, for example, by affine transformation. The deformation may include X-axis coordinates, Y-axis coordinates, θZ rotation, scale adjustment (enlargement / reduction), and distortion correction (shear deformation). The positions and shapes of the four target image elements GE1 in the first image 71 and the deformed second image 72 (hereinafter referred to as second image 72A) match. Therefore, the images in the overlapping area JA of the first image 71 and the deformed second image 72A substantially match each other. The image positioning process S2A eliminates differences in image distortion due to assembly errors in the installation positions of the captured first image 71 and second image 72A, aberrations in the optical system, etc. As will be described later, the synthesis processing unit 44B joins the transformed second image 72A with the first image 71.

[0097] <Mask Area Setting Process> The mask area setting process S2B (see FIG. 10 ) is a process for setting an area to be joined by the compositing process S2C for the first image 71 and the second image 72A that have been aligned by the image positioning process S2A. In this embodiment, the compositing processor 44B sets the joining position in the overlapping area JA so that the entire image element GE detected in the overlapping area JA is included in either the first image 71 or the second image 72A.

[0098] As shown in Figure 12, the synthesis processing unit 44B extracts the image element GE that is adjacent to the edge EL on the second image 72A side in the overlapping area JA, and sets a first mask area 73 in the first image 71 excluding the extracted image element GE.

[0099] For example, the synthesis processing unit 44B adds a straight line SL to the end edge EL and extracts the contour with the largest area within the overlapping area JA. Adding a straight line SL to the end edge EL means setting a predetermined pixel value to the pixel row that makes up the end edge EL in the overlapping area JA. If an image element GE is present that is in contact with the end edge EL, that image element GE is connected to the straight line SL. As a result, when the contour with the largest area within the overlapping area JA is extracted, only the image element GE connected to the straight line SL (i.e., in contact with the end edge EL) is extracted. The straight line SL and the contour of the image element GE connected by the straight line SL are referred to as the contour area 76.

[0100] The synthesis processing unit 44B sets the area of ​​the first image 71 excluding the extracted outline area 76 as a first mask area 73. The first mask area 73 is an area of ​​the first image 71 that is used as pixels that form the synthesized image 75. The outline area 76 of the first image 71 is excluded from the first mask area 73, and is therefore not used in the synthesis process.

[0101] Next, the synthesis processing unit 44B sets a second mask area 74 in the second image 72A, which is adjacent to the first mask area 73 and includes the entire image element GE that contacts the edge EL of the overlap area JA.

[0102] The synthesis processing unit 44B sets the extracted outline region 76 from the overlapping region JA of the second image 72A and the region unique to the second image 72A that is not included in the overlapping region JA as a second mask region 74. Because the outline region 76 is excluded from the first mask region 73, the second mask region 74 is adjacent to the first mask region 73 without any gaps at the outline region 76. The second mask region 74 is a region of the second image 72A that is used as pixels that constitute the synthesized image 75.

[0103] Therefore, the first mask region 73 includes the unique region of the first image 71 and the entire overlapping region JA excluding the outline region 76. The unique region of the first image 71 is the region that is included only in the first image 71 and is not included in the second image 72A. The second mask region 74 includes the unique region of the second image 72A and the outline region 76 of the overlapping region JA. The unique region of the second image 72A is the region that is included only in the second image 72A and is not included in the first image 71.

[0104] Here, we will explain the image element GE that contacts the edge EL. The edge EL of the overlapping area JA on the second image 72A side is the edge on the +X direction side of the first image 71. Therefore, the image element GE that contacts the edge EL of the overlapping area JA contacts the edge of the first image 71. There is a possibility that the image element GE that contacts the edge of the first image 71 has a portion that is not shown in the first image 71 (a portion P2 on the +X side of the edge EL). In other words, there is a possibility that only the portion P1 of the hole 7 that is on the -X side of the edge EL is shown in the first image 71.

[0105] If the position of the edge EL were set at the boundary between the first mask region 73 and the second mask region 74, the -X side portion P1 would be obtained from the first image 71, and the +X side portion P2 would be obtained from the second image 72A. Therefore, if the position of the edge EL were set at the boundary between the first mask region 73 and the second mask region 74, information loss or errors could occur. On the other hand, the edge EL on the second image 72A side of the overlap region JA is located toward the center of the second image 72A. Therefore, the entire image element GE (all of portions P1 and P2) that abuts the edge EL of the overlap region JA is reliably captured in the second image 72A. Therefore, by including the outline region 76 in the second mask region 74 of the second image 72A, a composite image 75 can be generated that completely preserves the information of the image element GE of the hole 7 that straddles the edge EL.

[0106] It is possible that there is no image element GE adjacent to the edge EL of the overlapping area JA. In this case, the synthesis processing unit 44B simply sets the first mask area 73 and the second mask area 74, using the edge EL as the boundary. That is, the first mask area 73 may be set as the entire first image 71, including the overlapping area JA. The second mask area 74 may be set as a unique area of ​​the second image 72A, excluding the overlapping area JA, using the edge EL as the boundary. When the recognition target UP is an array of multiple image elements GE (i.e., holes 7), the area in the image other than the image elements GE corresponds to the "background" that is not subject to processing in the image recognition process. Therefore, if the position of the edge EL corresponds to the background, even if missing or error image information occurs, it does not affect the detection accuracy of the recognition target UP.

[0107] <Composition Process> The composition process S2C (see FIG. 10 ) is a process of joining the first image 71 and the second image 72A using the boundary between the first mask region 73 of the first image 71 and the second mask region 74 of the second image 72A as the joining position. As shown in FIG. 13 , the composition processing unit 44B generates a composite image 75 by joining the first image 71 and the second image 72A using each pixel belonging to the first mask region 73 in the first image 71 and each pixel belonging to the second mask region 74 in the second image 72A. The generated composite image 75 is an image of a size corresponding to the enlarged imaging region 61A. In the composite image 75, the region corresponding to the first mask region 73 is an image region acquired from the first image 71. In the composite image 75, the region corresponding to the second mask region 74 is an image region acquired from the second image 72. Each image element GE included in the composite image 75 is obtained entirely from either the first image 71 or the second image 72A, so there is no loss or error in image information regarding the image element GE. In this manner, the image joining process S2 is performed.

[0108] (Recognition Target Detection Process) Next, the recognition target detection process S3 in Fig. 9 is a process for detecting a recognition target UP from the composite image 75 generated by the image stitching process S2 and acquiring the position coordinates of the detected recognition target UP. Fig. 14 is a schematic diagram illustrating the recognition target detection process S3.

[0109] As shown in FIG. 14 , the detection processing unit 44C extracts an image of a recognition area AR from the composite image 75, where the detection process for the recognition target UP is performed. The recognition area AR is set within a range that takes into account the expected positional deviation of the recognition target UP due to stitch formation relative to the design position coordinates of the recognition target UP set in the sewing data. FIG. 14 schematically illustrates the setting of the recognition area AR for a target pattern RP of an arbitrary shape. The detection processing unit 44C sets a recognition area AR for each of the multiple recognition targets UP to be detected in one stitch formation, and obtains an extracted image 77 by cutting out the image of the set recognition area AR. For example, in the example of FIG. 4 , a recognition area AR is set for each of the seven recognition targets UP, from recognition target UP1 to recognition target UP7. Note that the composition process S2C in FIGS. 10 and 13 (i.e., generation of the composite image 75) may be performed only when the recognition area AR includes the boundary (edge ​​EL) of the overlap area JA.

[0110] The detection processing unit 44C queues (sets the processing order) each acquired extracted image 77 and performs image recognition processing using pattern matching in order. The detection processing unit 44C reads a reference image from the storage device 45 and acquires the position coordinates of the recognition target UP in the extracted image 77 by matching it with the reference image. As shown in FIG. 14 , the processor 44 that realizes the functions of the image processing unit 42 may include multiple processor cores CR. The processor cores CR are units of the arithmetic processing structure provided in the processor 44. Each processor core CR can execute arithmetic processing according to a computer program. When the processor 44 includes multiple processor cores CR, the image processing unit 42 executes image recognition processing for each queued extracted image 77 in parallel using the multiple processor cores CR. If there are K available processor cores CR, image recognition processing for K extracted images 77 can be performed in parallel simultaneously, thereby reducing processing time.

[0111] Through the recognition target detection process S3, the detection processing section 44C acquires the position coordinates in the sewing machine coordinate system of each recognition target UP (UP1 to UP7, see FIG. 4) to be detected in one stitch formation.

[0112] (Correction Data Calculation Process) The correction data calculation process S4 is a process for calculating correction data for correcting displacement of the surface of the sewing object S based on the detection results of the recognition object detection process S3. The correction amount calculation unit 44D calculates the amount of displacement of the recognition object UP from its initial position to its current position based on the initial position and current position of the recognition object UP related to the current sewing process. The initial position of the recognition object UP is the design position coordinate of the recognition object UP set in the sewing data. The current position of the recognition object UP is the position coordinate of the recognition object UP detected by the recognition object detection process S3. Based on the calculated amount of displacement, the correction amount calculation unit 44D calculates the corrected position coordinates of each correction point CP (see FIG. 14) on the target pattern RP so that stitches CH are formed at target positions in the sewing machine coordinate system, and calculates correction data for the target pattern RP that passes through the correction points CP. Note that in the initial sewing process (first sewing process), no displacement of the surface of the sewing object S occurs due to stitch formation, and therefore no displacement of the recognition object UP occurs. Therefore, in the first sewing process (first sewing process), the target pattern RP in the initial state defined by the sewing data is used as is, and no correction data is calculated.

[0113] (Sewing Process) The sewing process S5 is a process for forming stitches CH based on the target pattern RP. The sewing process S5 is performed by the control device 30 controlling each part of the sewing machine 1. The sewing process includes a first sewing process to an N-th sewing process (N=10 in FIG. 4). The first sewing process is performed based on the target pattern RP in the initial state defined by the sewing data. The second sewing process to the N-th sewing process are performed based on the correction data calculated by the correction data calculation process S4 (the corrected target pattern RP). In the first sewing process, the control device 30 outputs a control command to the actuator 17 so that the stitches CH are formed in accordance with the target pattern RP in the initial state defined by the sewing data. From the second sewing process onwards, the control device 30 acquires correction data from the image processing unit 42 and outputs a control command to the actuator 17 so that the stitches CH are formed in accordance with the target pattern RP of the acquired correction data.

[0114] The end determination process S6 is a process for determining whether or not the sewing process of the sewing object S has been completed. The control device 30 determines whether or not the sewing process of the sewing object S has been completed based on the sewing data. When the first sewing process to the (N-1)th sewing process have been completed, the control device 30 determines in the end determination process S6 that the sewing process has not been completed. When the Nth sewing process has been completed, the control device 30 determines in the end determination process S6 that the sewing process has been completed.

[0115] In this way, in the first to Nth sewing processes, each time one sewing process is completed, the imaging process S1, the image joining process S2, the recognition target detection process S3, and the correction data calculation process S4 are performed, thereby calculating correction data for the next sewing process, and the next sewing process is performed based on the calculated correction data. This series of processes is repeated. In this embodiment, the multiple recognition targets UP detected in one sewing process can be detected from the composite image 75 obtained by capturing images once using the multiple image capture devices 41. As a result, the processing time required for the imaging process S1, the image joining process S2, the recognition target detection process S3, and the correction data calculation process S4, which are performed each time a sewing process is performed, is reduced.

[0116] [Effects] As described above, according to this embodiment, images of the sewing object S captured by the multiple image capture devices 41 are stitched together, and multiple recognition targets UP of the sewing object S are detected from the stitched composite image 75. This reduces the number of times the sewing object S is moved and captured compared to capturing and detecting multiple recognition targets UP one by one, thereby shortening the time required to detect the recognition targets UP during the sewing process. As a result, by shortening the time required to detect the recognition targets UP each time the sewing process is performed, the overall processing time for the sewing operation can be shortened. Furthermore, by providing an overlapping area JA in the imaging fields of the multiple image capture devices 41 and matching identical image elements GE in the overlapping area JA with each other, high positional accuracy can be achieved even when stitching images together. Furthermore, by setting the stitching position so that the entire image element GE detected in the overlapping area JA is included in either the first image 71 or the second image 72, missing or incorrect information about the image elements GE that make up the recognition target UP can be prevented. As a result, the detection accuracy of the recognition targets UP can be improved. Furthermore, image elements GE that are in contact with the edge EL of the second image 72 in the overlapping area JA are extracted, and a second mask area 74 that includes the entire extracted image elements GE is set in the second image 72. This makes it possible to easily extract image elements GE that may lose image information during joining, and to perform joining that preserves the entire image elements GE. This reduces the computational load associated with the joining process and shortens processing time.

[0117] Furthermore, in this embodiment, the image elements GE are holes 7 scattered throughout the sewing object S, and even if the recognition targets UP are an array of multiple holes 7 that exist in a predetermined pattern on the sewing object S, the recognition targets UP can be detected accurately and in a short time. Furthermore, because the multiple image capture devices 41 are arranged along at least one of the longitudinal direction (X-axis direction) and lateral direction (Y-axis direction) of the sewing area SA, a composite image 75 can be generated that can simultaneously detect multiple recognition targets UP according to the direction in which the stitches CH extend. Furthermore, because the multiple illumination devices 43 (43A, 43B) are provided on both sides of the array of the multiple image capture devices 41 and between the array of the multiple image capture devices 41, the brightness (pixel values) within the image can be made uniform even when generating a composite image 75 over a wide area, improving detection accuracy.

[0118] [Other Embodiments] In the above-described embodiment, an example in which four imaging devices 41 are provided has been shown, but the number of imaging devices 41 may be two, three, five or more. The imaging devices 41 may be attached at six or more locations. The multiple imaging devices 41 may be arranged along a direction other than the longitudinal direction (X-axis direction) and the lateral direction (Y-axis direction) of the sewing area SA. The number of lighting devices 43A, 43B is not limited and may be one, two or more. While an example has been shown in which the image element GE is a hole 7 formed in the sewing object S, the image element GE is not particularly limited and may be an image element other than a hole 7.

[0119] In the above embodiment, the second image 72 is deformed to fit the first image 71, but the first image 71 may also be deformed to fit the second image 72. Also, either of the two images to be joined may be the first image 71 or the second image 72. In the image positioning process S2A, only the positioning may be performed without deforming the second image 72.

[0120] In the above embodiment, each pixel constituting the composite image 75 is used from either the first image 71 or the second image 72, but both the first image 71 and the second image 72 may be used for each pixel corresponding to the overlapping region JA of the composite image 75. For example, for the overlapping region JA, averaging processing may be performed using both the first image 71 and the second image 72. Furthermore, for the overlapping region JA, both the first image 71 and the second image 72 may be binarized, and an image obtained by taking the logical AND or OR of both images may be used.

[0121] The present disclosure includes the following aspects: (1) An image processing device comprising: a plurality of image capture devices arranged at intervals in a horizontal plane so as to be aligned with a sewing machine body having a head for sewing and a holding member for holding and moving a sewing object; and an image processing unit that detects a recognition target on the sewing object based on images of the sewing object captured by the plurality of image capture devices, wherein the image processing unit comprises: a synthesis processing unit that stitches together images of the plurality of image capture devices; and a detection processing unit that detects the plurality of recognition targets from the stitched-together image. (2) The image processing device described in (1), wherein each of the plurality of image capture devices is arranged to have an overlapping area where an imaging field of view partially overlaps with that of an adjacent one of the image capture devices, and the synthesis processing unit detects an image element that is common to at least a part of the recognition target in the overlapping area of ​​a first image and a second image to be stitched together, deforms the second image so that the identical image elements in the overlapping area match each other, and stitches the deformed second image with the first image. (3) The image processing device according to (2), wherein the compositing processing unit sets a joining position in the overlapping area so that the entirety of the image elements detected in the overlapping area is included in either the first image or the second image. (4) The compositing processing unit extracts the image elements that contact an edge of the overlapping area on the second image side, sets a first mask area in the first image excluding the extracted image elements, sets a second mask area in the second image that includes the entirety of the image elements that contact the edge and is adjacent to the first mask area, and joins the first image and the second image using a boundary between the first mask area of ​​the first image and the second mask area of ​​the second image as the joining position. (5) The image processing device according to any one of (2) to (4), wherein the image elements are holes that are scattered throughout the sewing object, and the recognition target is an array of a plurality of the holes that exist in a predetermined pattern on the sewing object.(6) The image processing device according to any one of (1) to (5), wherein the plurality of image capturing devices are arranged along at least one of the longitudinal direction and the lateral direction of a sewing area in which the sewing object can be moved by the holding member and sewn by the head. (7) The image processing device according to any one of (1) to (4), further comprising a plurality of lighting devices respectively arranged on both outsides of the arrangement of the plurality of image capturing devices and between the arrangement of the plurality of image capturing devices. (8) A sewing machine comprising: a sewing machine main body including a head that performs sewing and a holding member that holds and moves the sewing object; the image processing device according to any one of (1) to (7); and a control device that controls the sewing machine main body based on processing results of the image processing device. (9) An image processing method including: stitching together images of an object to be sewn taken by a plurality of image capturing devices arranged at intervals in a predetermined direction in a horizontal plane so as to be aligned with a sewing machine body having a head for sewing and a holding member for holding and moving the object to be sewn; and detecting a plurality of recognition targets of the object to be sewn from the stitched composite image.

[0122] This application is based on Japanese Patent Application No. 2023-108671, filed on June 30, 2023, the contents of which are incorporated herein by reference.

Claims

1. An image processing device comprising: a plurality of imaging devices arranged at intervals in a horizontal plane in line with a sewing machine body having a head for sewing and a holding member for holding and moving an object to be sewn; and an image processing unit that detects a recognition target on the object to be sewn based on images of the object to be sewn captured by the plurality of imaging devices, the image processing unit comprising: a synthesis processing unit that joins together images of the plurality of imaging devices; and a detection processing unit that detects the plurality of recognition targets from the composite image joined together by the synthesis processing unit.

2. The image processing device of claim 1, wherein each of the multiple imaging devices is arranged to have an overlapping area in which the imaging field of view partially overlaps with that of the adjacent other imaging devices, and the synthesis processing unit detects image elements in the overlapping areas of the first and second images to be stitched together that are common to at least a portion of the recognition target, deforms the second image so that the identical image elements in the overlapping areas match each other, and stitches the transformed second image with the first image.

3. The image processing device according to claim 2, wherein the synthesis processing unit sets a stitching position in the overlapping area so that the entire image element detected in the overlapping area is included in either the first image or the second image.

4. The image processing device of claim 3, wherein the synthesis processing unit extracts the image elements that contact the edge of the overlapping area on the side of the second image, sets a first mask area in the first image excluding the extracted image elements, sets a second mask area in the second image that includes all of the image elements that contact the edge and is adjacent to the first mask area, and joins the first image and the second image together, using the boundary between the first mask area of ​​the first image and the second mask area of ​​the second image as the joining position.

5. An image processing device according to any one of claims 2 to 4, wherein the image elements are holes formed so as to be scattered throughout the sewing object, and the recognition object is an array of a plurality of the holes present in a predetermined pattern on the sewing object.

6. An image processing device as claimed in any one of claims 1 to 4, wherein the multiple imaging devices are arranged along at least one of the longitudinal and lateral directions of a sewing area in which the sewing object can be moved by the holding member and sewn by the head.

7. The image processing device according to any one of claims 1 to 4, further comprising a plurality of illumination devices respectively arranged on both outer sides of the array of the plurality of imaging devices and between the array of the plurality of imaging devices.

8. A sewing machine comprising: a sewing machine body having a head for sewing and a holding member for holding and moving an object to be sewn; an image processing device according to any one of claims 1 to 4; and a control device for controlling the sewing machine body based on the processing results of the image processing device.

9. An image processing method comprising: stitching together images of an object to be sewn captured by a plurality of imaging devices arranged at intervals in a predetermined direction in a horizontal plane in line with a sewing machine body having a head for sewing and a holding member for holding and moving the object to be sewn; and detecting a plurality of recognition targets possessed by the object to be sewn from the stitched composite image.

Images