Image processing device, sewing machine and image processing method

Abstract

To shorten processing time of entire sewing work.SOLUTION: An image processing device includes: a plurality of imaging devices arranged having an interval in a horizontal surface, so as to be side by side with respect to a sewing machine body which includes a head for performing sewing and a holding member for holding a sewing object and moving it; and an image processing part for detecting a recognition object which the sewing object has, based on the image of the sewing object imaged by the plurality of imaging devices. The image processing part includes: a composition processing part for combining the images from the plurality of imaging devices; and a detection processing part for detecting a plurality of recognition objects from a composite image combined by the composition processing part.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, a sewing machine, and an image processing method. [Background technology]

[0002] In order to improve the design of a sewn object, stitches may be formed on the sewn object. Patent Document 1 discloses a technique for forming stitches on a cover material used for a vehicle seat. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2013-162957 A Summary of the Invention [Problem to be solved by the invention]

[0004] The upholstery material used for vehicle seats is thick and elastic. When stitches are formed on a sewing object having thickness and elasticity, the sewing object may shrink and the surface of the sewing object may be displaced. For example, when a second stitch is formed after a first stitch is formed, it is desirable to form the second stitch at a target position of the sewing object in accordance with the displacement of the surface of the sewing object caused by the formation of the first stitch.

[0005] One way to form stitches at target positions on a sewing object is to photograph the surface of the sewing object before sewing and detect the displacement of the surface of the sewing object. If multiple points along the stitch line are photographed one by one in order and the displacement of each point is detected, it takes time for the photographing and image recognition processing. Since the displacement changes each time a stitch is formed due to that stitch, the overall processing time for the sewing operation increases if the photographing and displacement detection are performed each time a stitch is formed.

[0006] An object of the present invention is to reduce the processing time of the entire sewing operation. [Means for solving the problem]

[0007] According to a first aspect of the present invention, there is provided an image processing device comprising: a plurality of imaging devices arranged at intervals in a horizontal plane in line with a sewing machine body having a head for sewing and a holding member for holding and moving an object to be sewn; and an image processing unit which detects a recognition target possessed by the object to be sewn based on images of the object to be sewn captured by the plurality of imaging devices, wherein the image processing unit comprises a synthesis processing unit which joins together images of the plurality of imaging devices; and a detection processing unit which detects the plurality of recognition targets from the composite image joined together by the synthesis processing unit.

[0008] According to a second aspect of the present invention, there is provided a sewing machine comprising a sewing machine body having a head for sewing and a holding member for holding and moving an object to be sewn, an image processing device according to the first aspect, and a control device for controlling the sewing machine body based on the processing results of the image processing device.

[0009] According to a third aspect of the present invention, there is provided an image processing method including stitching together images of an object to be sewn captured by a plurality of imaging devices arranged at intervals in a predetermined direction in a horizontal plane in line with a sewing machine body having a head for sewing and a holding member for holding and moving the object to be sewn, and detecting a plurality of recognition targets possessed by the object to be sewn from the stitched composite image. Effect of the Invention

[0010] According to an aspect of the present invention, the processing time of the entire sewing operation can be shortened. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a perspective view showing a sewing machine according to the present embodiment. [Diagram 2]FIG. 2 is a perspective view showing a part of the sewing machine according to the present embodiment. [Diagram 3] FIG. 3 is a cross-sectional view showing a part of the sewing object according to the present embodiment. [Figure 4] FIG. 4 is a plan view showing the sewing object according to the present embodiment. [Diagram 5] FIG. 5 is a cross-sectional view showing a part of the sewing object according to the present embodiment. [Figure 6] FIG. 6 is a plan view illustrating the mounting position of the imaging device according to this embodiment. [Figure 7] FIG. 7 is a plan view illustrating an example of the arrangement of the imaging device according to this embodiment. [Figure 8] FIG. 8 is a functional block diagram showing the sewing machine according to the present embodiment. [Figure 9] FIG. 9 is a flowchart for explaining the sewing operation of the sewing machine according to the present embodiment. [Figure 10] FIG. 10 is a flowchart showing the image joining process according to this embodiment. [Figure 11] FIG. 11 is a schematic diagram for explaining the image positioning process in the image joining process. [Figure 12] FIG. 12 is a schematic diagram for explaining the mask area setting process in the image joining process. [Figure 13] FIG. 13 is a schematic diagram for explaining the synthesis process in the image joining process. [Figure 14] FIG. 14 is a schematic diagram for explaining the recognition target detection process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings, but the present invention is not limited thereto. The components of the embodiments described below can be appropriately combined. In addition, some components may not be used.

[0013] In this embodiment, a local coordinate system is defined for the sewing machine 1. In the following description, the local coordinate system defined for the sewing machine 1 is appropriately referred to as the sewing machine coordinate system. The sewing machine coordinate system is defined by an XYZ orthogonal coordinate system. In this embodiment, the positional relationship of each part will be described based on the sewing machine coordinate system. The direction parallel to the X axis in a predetermined plane is defined as the X axis direction. The direction parallel to the Y axis in a predetermined plane perpendicular to the X axis is defined as the Y axis direction. The direction parallel to the Z axis perpendicular to the predetermined plane is defined as the Z axis direction. In addition, the rotation direction or tilt direction around the X axis is defined as the θX direction. The rotation direction or tilt direction around the Y axis is defined as the θY direction. The rotation direction or tilt direction around the Z axis is defined as the θZ direction. In this embodiment, the plane including the X axis and the Y axis is appropriately referred to as the XY plane. The plane including the X axis and the Z axis is appropriately referred to as the XZ plane. The plane including the Y axis and the Z axis is appropriately referred to as the YZ plane. The XY plane is parallel to the predetermined plane. The XY plane, the XZ plane, and the YZ plane are orthogonal to each other. In this embodiment, the XY plane and the horizontal plane are parallel to each other. The Z axis direction is the up-down direction. The +Z direction is the upward direction, and the -Z direction is the downward direction. The XY plane may be inclined with respect to the horizontal plane.

[0014] [sewing machine] Fig. 1 is a perspective view showing a sewing machine 1 according to this embodiment. Fig. 2 is a perspective view showing a part of the sewing machine 1 according to this embodiment. In this embodiment, the sewing machine 1 is an electronic cycle sewing machine. The sewing machine 1 includes a sewing machine body 10, an operation device 20 operated by an operator, a control device 30 that controls the sewing machine body 10, and an image processing device 40 that captures an image of a sewing object S and performs image recognition. The image processing device 40 includes a plurality of imaging devices 41 and an image processing unit 42.

[0015] Sewing machine body 10 is mounted on the upper surface of table 2. As shown in Fig. 2, sewing machine body 10 has sewing machine frame 11, needle bar 12 supported by sewing machine frame 11, needle plate 13 supported by sewing machine frame 11, holding member 15 supported by sewing machine frame 11 via support member 14, actuator 16 (see Fig. 1) that generates power to move needle bar 12, actuator 17 (see Fig. 1) that generates power to move holding member 15, and actuator 18 (see Fig. 1) that generates power to move at least a portion of holding member 15.

[0016] The sewing machine frame 11 has a horizontal arm 11A extending in the Y-axis direction, a bed 11B (see Figure 1) arranged below the horizontal arm 11A, a vertical arm 11C (see Figure 1) connecting the +Y side end of the horizontal arm 11A to the bed 11B, and a head 11D arranged on the -Y side of the horizontal arm 11A.

[0017] Needle bar 12 holds sewing needle 3. Needle bar 12 holds sewing needle 3 so that sewing needle 3 is parallel to the Z-axis. Needle bar 12 is supported by head 11D so as to be movable in the Z-axis direction.

[0018] The needle plate 13 supports a sewing object S. The needle plate 13 supports a holding member 15. The needle plate 13 is supported by a bed 11B (see FIG. 1). The needle plate 13 is disposed below the holding member 15.

[0019] The holding member 15 holds the sewing object S. The holding member 15 is capable of moving while holding the sewing object S within an XY plane including a sewing position Ps directly below the sewing needle 3. The holding member 15 is capable of holding the sewing object S within an XY plane including a position directly below the imaging device 41. With the holding member 15 holding the sewing object S, it moves within the XY plane based on the sewing data. As a result, a stitch CH (see FIG. 5) is formed on the sewing object S that has passed the sewing position Ps. The holding member 15 is supported by the horizontal arm 11A via the support member 14.

[0020] The holding member 15 has a presser member 15A and a lower plate 15B facing the presser member 15A. The presser member 15A is a frame-shaped member. The presser member 15A is movable in the Z-axis direction. The lower plate 15B is disposed below the presser member 15A. The holding member 15 holds the sewing object S by sandwiching the sewing object S between the presser member 15A and the lower plate 15B.

[0021] When the presser member 15A moves in the +Z direction, the presser member 15A and the lower plate 15B are separated. This allows the operator to place the sewing object S between the presser member 15A and the lower plate 15B. When the presser member 15A moves in the -Z direction with the sewing object S placed between the presser member 15A and the lower plate 15B, the sewing object S is sandwiched between the presser member 15A and the lower plate 15B. This allows the sewing object S to be held by the holding member 15. Furthermore, when the presser member 15A moves in the +Z direction, the holding of the sewing object S by the holding member 15 is released. This allows the operator to remove the sewing object S from between the presser member 15A and the lower plate 15B.

[0022] 1, actuator 16 generates power to move needle bar 12 in the Z-axis direction. Actuator 16 includes a pulse motor. Actuator 16 is disposed on horizontal arm 11A.

[0023] A horizontal arm shaft extending in the Y-axis direction is disposed inside horizontal arm 11A. Actuator 16 is connected to the +Y side end of the horizontal arm shaft. The -Y side end of the horizontal arm shaft is connected to needle bar 12 via a power transmission mechanism disposed inside head 11D. Operation of actuator 16 rotates the horizontal arm shaft. Power generated by actuator 16 is transmitted to needle bar 12 via the horizontal arm shaft and the power transmission mechanism. As a result, sewing machine needle 3 held by needle bar 12 moves back and forth in the Z-axis direction.

[0024] A timing belt extending in the Z-axis direction is disposed inside the vertical arm 11C. A bed shaft extending in the Y-axis direction is disposed inside the bed 11B. Pulleys are disposed on each of the horizontal arm shaft and the bed shaft. The timing belt is hung on each of the pulleys disposed on the horizontal arm shaft and the pulleys disposed on the bed shaft. The horizontal arm shaft and the bed shaft are connected via a power transmission mechanism including the timing belt.

[0025] A shuttle is disposed inside bed 11B. A bobbin placed in a bobbin case is housed in the shuttle. The horizontal arm shaft and the bed shaft are each rotated by the operation of actuator 16. Power generated by actuator 16 is transmitted to the shuttle via the horizontal arm shaft, a timing belt, and the bed shaft. As a result, the shuttle rotates in synchronization with the reciprocating movement of needle bar 12 in the Z-axis direction.

[0026] Actuator 17 generates power to move holding member 15 in the XY plane. Actuator 17 includes a pulse motor. Actuator 17 includes an X-axis motor 17X (see FIG. 8) that generates power to move holding member 15 in the X-axis direction, and a Y-axis motor 17Y (see FIG. 8) that generates power to move holding member 15 in the Y-axis direction. Actuator 17 is disposed inside bed 11B.

[0027] The power generated by the actuator 17 is transmitted to the holding member 15 via the support member 14. This allows the holding member 15 to move in both the X-axis and Y-axis directions between the sewing needle 3 and the needle plate 13. By operating the actuator 17, the holding member 15 can move while holding the sewing object S within an XY plane including the sewing position Ps directly below the sewing needle 3.

[0028] The actuator 18 generates power to move the presser member 15A of the holding member 15 in the Z-axis direction. The actuator 18 includes a pulse motor. When the presser member 15A moves in the +Z direction, the presser member 15A and the lower plate 15B are separated from each other. When the presser member 15A moves in the -Z direction, the sewing object S is sandwiched between the presser member 15A and the lower plate 15B.

[0029] As shown in FIG. 2, the sewing machine body 10 has an intermediate presser member 19 arranged around the sewing needle 3. The intermediate presser member 19 holds down the sewing workpiece S around the sewing needle 3. The intermediate presser member 19 is supported by the head 11D so as to be movable in the Z-axis direction. An intermediate presser motor is arranged inside the head 11D, which generates power to move the intermediate presser member 19 in the Z-axis direction. Operation of the intermediate presser motor causes the intermediate presser member 19 to move in the Z-axis direction in synchronization with the needle bar 12. The intermediate presser member 19 prevents the sewing workpiece S from floating up due to movement of the sewing needle 3.

[0030] As shown in FIG. 1, the operation device 20 accepts operation input by an operator. The sewing machine 1 is operated by operating the operation device 20. The operation device 20 includes an operation panel 21 and an operation pedal 22. The operation panel 21 is mounted on the upper surface of the table 2. The operation pedal 22 is disposed below the table 2. The operator operates the operation pedal 22 with his / her foot. The sewing machine 1 is operated by the operator operating at least one of the operation panel 21 and the operation pedal 22.

[0031] Before starting the sewing process, the image processing device 40 acquires images of the sewing object S using multiple imaging devices 41, detects the target position for sewing using the image processing unit 42, and performs processing to calculate correction data for displacement caused by the already created stitches CH, etc.

[0032] The imaging devices 41 capture images of the sewing object S held by the holding member 15. The imaging devices 41 each include an optical system and an image sensor that receives light incident via the optical system. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0033] The multiple imaging devices 41 are disposed above the needle plate 13 and the holding member 15. Each of the multiple imaging devices 41 captures an image of at least a portion of the sewing object S held by the holding member 15 from above. The number of imaging devices 41 is two or more, but is not particularly limited. Details of the arrangement, imaging field of view, etc. of the multiple imaging devices 41 will be described later.

[0034] The image processing unit 42 includes a computer system. The image processing unit 42 processes images related to the sewing object S. The image processing unit 42 detects a recognition target of the sewing object S based on images of the sewing object S captured by the multiple imaging devices 41.

[0035] [Items to be sewn] Fig. 3 is a cross-sectional view showing a part of the sewing object S according to this embodiment. Fig. 4 is a plan view showing the sewing object S according to this embodiment. Figs. 3 and 4 show the sewing object S before sewing processing. In this embodiment, the sewing object S is a cover material used for a vehicle seat.

[0036] 3, the sewing object S has a surface material 4, a padding material 5, and a backing material 6. A hole 7 is provided in the surface material 4.

[0037] The surface of the cover material 4 is a seating surface that comes into contact with an occupant when the occupant sits on the vehicle seat. The cover material 4 includes at least one of a woven fabric, a nonwoven fabric, and leather. The pad material 5 has elasticity. The pad material 5 includes, for example, a urethane resin. The backing material 6 includes at least one of a woven fabric, a nonwoven fabric, and leather.

[0038] As shown in FIG. 4, a plurality of holes 7 are provided in the surface material 4. The holes 7 are formed so as to be scattered in the sewing object S. The holes 7 are arranged in a regular pattern. In other words, the sewing object S includes a plurality of reference patterns DPh. The reference pattern DPh is composed of a regular arrangement of a plurality of holes 7. In the example of FIG. 4, the reference pattern DPh is composed of 17 holes 7.

[0039] As shown in Fig. 4, the reference patterns DPh are arranged on the surface material 4 at intervals. The reference patterns DPh are arranged at equal intervals in both the X-axis direction and the Y-axis direction. A reference pattern DPh at a different position in the Y-axis direction is arranged between adjacent reference patterns DPh in the X-axis direction. No holes 7 are formed between adjacent reference patterns DPh.

[0040] A target pattern RP of a stitch CH to be formed on the sewing object S is defined in the area between the reference patterns DPh. Fig. 4 shows a virtual target pattern RP of a stitch CH on the sewing object S before sewing. The sewing machine 1 forms the stitch CH so as to follow each target pattern RP in accordance with a preset sewing pattern.

[0041] Further, a plurality of recognition objects UP (UP1, UP2, UP3, UP4, UP5, UP6, UP7) are arranged on the sewing object S. In this embodiment, the recognition objects UP are an array of a plurality of holes 7 that exist in a predetermined pattern on the sewing object S. Specifically, the recognition objects UP are part of a reference pattern DPh. In the example of FIG. 4, the recognition objects UP (UP1, UP2, UP3, UP4, UP5, UP6, UP7) are a pattern (arrangement of holes 7) that includes one acute corner of the reference pattern DPh having a diamond-shaped outer shape. The recognition objects UP are patterns that can be specified by a pattern matching method, which is a type of image processing method.

[0042] With reference to FIG. 5, a description will be given of the displacement that occurs on the surface of the sewing object S when a stitch CH is formed on the sewing object S, which has thickness and elasticity. FIG. 5 is a cross-sectional view showing an example of the sewing object S according to the present embodiment. FIG. 5 shows the sewing object S after a sewing process has been performed. The sewing object S is thick and elastic. When a stitch CH is formed on the sewing object S, which has thickness and elasticity, as shown in FIG. 5, there is a high possibility that the sewing object S will shrink. When the sewing object S shrinks, there is a possibility that the surface of the sewing object S will be displaced. When the surface of the sewing object S is displaced, there is a high possibility that the target position of the stitch CH defined on the surface of the sewing object S will be displaced in the XY plane. When the target position of the stitch CH is displaced in the XY plane, it becomes difficult to form the stitch CH at the target position when the holding member 15 is moved to the position coordinates of the target pattern RP in the design.

[0043] Therefore, in this embodiment, the image processing device 40 acquires the amount of displacement (a correction amount for correcting the amount of displacement), and the holding member 15 is moved according to the amount of displacement so that the next stitch CH is formed at the target position even if the sewing object S shrinks due to the formation of the stitch CH and the surface of the sewing object S is displaced. The image processing device 40 (see FIG. 1) detects the recognition object UP from an image captured of the sewing object S, and acquires the correction amount based on the displacement of the recognition object UP.

[0044] [Imaging device] FIG. 6 is a plan view illustrating the mounting position of the imaging device 41 according to this embodiment.

[0045] The multiple imaging devices 41 are arranged at intervals in a horizontal plane (XY plane) so as to be aligned with the sewing machine body 10. The multiple imaging devices 41 are arranged within a sewing area SA. The sewing area SA is a range in the XY plane in which the sewing object S can be moved by the holding member 15 and sewn by the head 11D. The head 11D (i.e., the sewing position Ps) is arranged in the center of the sewing area SA. The multiple imaging devices 41 are arranged along the head 11D of the sewing machine body 10.

[0046] 6, the sewing machine 1 includes imaging units 50A and 50B, one adjacent to each side of the head 11D in the X-axis direction. A plurality of imaging devices 41 included in the image processing device 40 can be attached to the imaging units 50A and 50B. The imaging units 50A and 50B are disposed above the sewing area SA. The imaging units 50A and 50B are fixed to the sewing machine 1.

[0047] The imaging unit 50A on the -X direction side has three attachment parts 51A, 51B, and 51C. Each of the attachment parts 51A, 51B, and 51C can mount one imaging device 41. The attachment parts 51A and 51B are disposed at an interval in the X-axis direction. The attachment parts 51B and 51C are disposed at an interval in the Y-axis direction.

[0048] The imaging unit 50B on the +X direction side has three attachment parts 51D, 51E, and 51F. Each of the attachment parts 51D, 51E, and 51F can mount one imaging device 41. The attachment parts 51D and 51E are disposed at an interval in the X-axis direction. The attachment parts 51D and 51F are disposed at an interval in the Y-axis direction.

[0049] Each of the mounting portions 51A, 51B, 51C, 51D, 51E, and 51F is located inside the sewing area SA in the XY directions. Each of the mounting portions 51A, 51B, 51C, 51D, 51E, and 51F can hold the imaging device 41 above the sewing area SA. Therefore, regardless of which mounting portion the imaging device 41 is attached to, the sewing machine 1 can position the sewing object S at the imaging position of the imaging device 41 by the holding member 15. In the example of FIG. 6, the mounting portions 51A and 51B of the imaging unit 50A and the mounting portions 51D and 51E of the imaging unit 50B are arranged on a straight line in the X-axis direction with respect to the sewing position Ps.

[0050] The imaging units 50A and 50B provide a total of six mounting positions (mounting portions 51A, 51B, 51C, 51D, 51E, 51F) on the sewing machine 1. In this embodiment, the imaging device 41 may be mounted at all of the six mounting positions, or at some (two or more adjacent positions). The sewing machine 1 is capable of selecting the mounting position of the imaging device 41 according to the shape of the sewing object S and the shape of the stitch CH to be formed on the sewing object S, and is also capable of changing the mounting position of the imaging device 41.

[0051] The imaging units 50A and 50B hold a plurality of lighting devices 43A and 43B provided in the image processing device 40. The imaging units 50A and 50B hold the lighting devices 43A and 43B in a fixed state above the sewing area SA. The imaging unit 50A has one lighting device 43A (total of three) arranged on both outsides of the arrangement of the mounting parts 51A and 51B along the X-axis direction and between the arrangements. Each lighting device 43A extends linearly along the Y-axis direction. Each lighting device 43A irradiates light in the form of epi-illumination to an imaging field of view set below the mounting parts 51A, 51B, and 51C. The imaging unit 50B has one lighting device 43B (total of three) arranged on both outsides of the arrangement of the mounting parts 51D and 51F along the Y-axis direction and between the arrangements. Each lighting device 43B extends linearly along the X-axis direction. Each of the illumination devices 43B irradiates light in the form of epi-illumination onto an imaging field of view set below the attachment portions 51D, 51E, and 51F.

[0052] Thus, the image processing device 40 according to this embodiment includes a plurality of imaging devices 41 arranged at intervals in a horizontal plane (XY plane) so as to be aligned with the sewing machine body 10 (head 11D). In the example of Fig. 6, the plurality of imaging devices 41 can be arranged along at least one of the longitudinal direction (X-axis direction) and the lateral direction (Y-axis direction) of the sewing area SA.

[0053] 7 is a plan view illustrating an example of the arrangement of the imaging devices 41 according to this embodiment. The imaging devices 41 can be arranged, for example, as shown in FIG.

[0054] 7, two imaging devices 41 are attached to each of mounting portions 51A and 51B of imaging unit 50A. The two imaging devices 41 of imaging unit 50A are arranged along the longitudinal direction (X-axis direction) of sewing area SA. Two imaging devices 41 are attached to each of mounting portions 51D and 51F of imaging unit 50B. The imaging device 41 of imaging unit 50B is arranged along the lateral direction (Y-axis direction) of sewing area SA.

[0055] The imaging device 41 captures an image of at least a part of the sewing object S arranged within the imaging field of view. The imaging device 41 captures an image of at least a part of the sewing object S held by the holding member 15 from above. Each of the imaging devices 41 is arranged to have an overlapping area JA where the imaging field of view of the adjacent imaging device 41 partially overlaps with that of the adjacent imaging device 41. Specifically, the imaging device 41 of the mounting portion 51A has an imaging field of view 60A. The imaging device 41 of the mounting portion 51B has an imaging field of view 60B. The imaging field of view 60A and the imaging field of view 60B have a rectangular shape and have an overlapping area JA extending in the Y-axis direction. The imaging field of view 60A and the imaging field of view 60B form an enlarged imaging area 61A. The enlarged imaging area is an area where the imaging fields of view of the multiple imaging devices 41 that share the overlapping area JA are combined. The imaging field of view 60A and the imaging field of view 60B arranged in the X-axis direction form a wide enlarged imaging area 61A extending in the X-axis direction.

[0056] Moreover, the imaging device 41 of the mounting part 51D has an imaging field of view 60D. The imaging device 41 of the mounting part 51F has an imaging field of view 60F. The imaging field of view 60D and the imaging field of view 60F have a rectangular shape and have an overlapping area JA extending in the X-axis direction. The imaging field of view 60D and the imaging field of view 60F form an enlarged imaging area 61B. The imaging field of view 60D and the imaging field of view 60F arranged in the Y-axis direction form a wide enlarged imaging area 61B extending in the Y-axis direction.

[0057] In the configuration of FIG. 7, the imaging field size of the imaging device 41, the size of the overlapping area JA (the interval between the mounting parts), and the number of imaging devices 41 are set according to the dimensions of the target pattern RP (see FIG. 4) of the sewing object S. That is, the enlarged imaging areas 61A and 61B formed by the multiple imaging devices 41 are designed to have a shape that encompasses the entire target pattern RP of the sewing object S. The sewing object S on which a stitch CH extending in the X-axis direction is formed can be imaged by being placed in the enlarged imaging area 61A of the imaging unit 50A, and the sewing object S on which a stitch CH extending in the Y-axis direction is formed can be imaged by being placed in the enlarged imaging area 61B of the imaging unit 50B. Therefore, the sewing machine 1 can image the entire target pattern RP at once without changing the mounting position of the imaging device 41, regardless of the direction in which the stitch CH extending is formed.

[0058] The position of each imaging device 41 is fixed at one of six mounting positions. The relative position of the imaging device 41 and the sewing machine frame 11 is fixed. The relative position of the optical axis of the imaging device 41 and the sewing needle 3 (sewing position Ps) in the XY plane is fixed. Relative position data indicating the relative position of the center of the enlarged imaging areas 61A, 61B and the sewing needle 3 in the XY plane is known data that can be derived from the design data of the sewing machine 1.

[0059] The position of the image captured by the imaging device 41 is defined in the camera coordinate system. The position of the image defined in the camera coordinate system is transformed into the position of the image defined in the sewing machine coordinate system by a defined transformation formula or transformation matrix.

[0060] In the example of FIG. 7, the head 11D is provided with a second imaging device 33. As shown in FIG. 6, the head 11D is provided with mounting parts 33A and 33B spaced apart from the sewing position Ps on both sides in the X-axis direction. The second imaging device 33 is attached to either one or both of the mounting parts 33A and 33B. In FIG. 7, one second imaging device 33 is provided on the mounting part 33A. The second imaging device 33 has a smaller imaging field of view and a higher resolution (number of pixels per unit area) than the imaging device 41. For this reason, the multiple imaging devices 41 are used for high-speed recognition applications in which multiple recognition targets UP are imaged at once, as described later, and the second imaging device 33 is used for precision recognition applications in which one recognition target UP is imaged at a time, and particularly precise positioning is required.

[0061] [Sewing machine control configuration] Fig. 8 is a functional block diagram showing the sewing machine 1 according to this embodiment. A configuration relating to the control of the sewing machine 1 will be described with reference to Fig. 8.

[0062] (Control device) The control device 30 outputs a control signal for controlling the sewing machine 1. The control device 30 includes a computer system. The control device 30 has an input / output interface, a storage device including a non-volatile memory such as a read only memory (ROM) or storage and a volatile memory such as a random access memory (RAM), and an arithmetic processing device including a processor such as a central processing unit (CPU). The control device 30 controls the sewing machine 1 according to a computer program stored in the storage device.

[0063] Connected to the control device 30 are an actuator 16 that moves the sewing needle 3 in the Z-axis direction, an actuator 17 that moves the holding member 15 within the XY plane, an actuator 18 that moves the pressing member 15A of the holding member 15 in the Z-axis direction, an operating device 20, and an image processing unit 42.

[0064] In addition, a drive amount sensor 31 that detects the drive amount of the actuator 16 and a drive amount sensor 32 that detects the drive amount of the actuator 17 are connected to the control device 30.

[0065] The control device 30 controls the actuator 16 based on the detection data of the drive amount sensor 31. The control device 30 determines, for example, the operation timing of the actuator 16 based on the detection data of the drive amount sensor 31.

[0066] The drive amount sensor 32 includes an X-axis sensor 32X that detects the drive amount of the X-axis motor 17X of the actuator 17, and a Y-axis sensor 32Y that detects the drive amount of the Y-axis motor 17Y of the actuator 17. The X-axis sensor 32X includes an encoder that detects the rotation amount of the X-axis motor 17X. The Y-axis sensor 32Y includes an encoder that detects the rotation amount of the Y-axis motor 17Y. The detection data of the drive amount sensor 32 is output to the control device 30. The control device 30 controls the actuator 17 based on the detection data of the drive amount sensor 32. The control device 30 feedback-controls the actuator 17 based on the detection data of the drive amount sensor 32 so that the holding member 15 moves to a target position.

[0067] The drive amount sensor 32 functions as a position sensor that detects the position in the XY plane of the holding member 15. The drive amount of the actuator 17 and the movement amount of the holding member 15 correspond one-to-one.

[0068] The X-axis sensor 32X detects the amount of rotation of the X-axis motor 17X, thereby detecting the amount of movement of the holding member 15 in the X-axis direction from the origin in the sewing machine coordinate system. The Y-axis sensor 32Y detects the amount of rotation of the Y-axis motor 17Y, thereby detecting the amount of movement of the holding member 15 in the Y-axis direction from the origin in the sewing machine coordinate system.

[0069] The control device 30 calculates the position of the holding member 15 in the XY plane based on the detection data of the drive amount sensor 32. The amount of movement of the holding member 15 from the origin in the XY plane is detected based on the detection data of the drive amount sensor 32. The control device 30 calculates the position of the holding member 15 in the XY plane based on the detected amount of movement of the holding member 15.

[0070] The control device 30 stores the sewing data in the storage device. The sewing data is known data that can be derived from design data of the sewing object S, such as CAD (Computer Aided Design) data. The sewing data is referred to by the control device 30 in the sewing process. The sewing process refers to a process of forming a stitch CH in the sewing object S. The sewing data includes a target pattern RP (see FIG. 4) of the stitch CH to be formed in the sewing object S, position coordinates of the recognition target UP, and movement conditions of the holding member 15.

[0071] The target pattern RP defines the target shape of the stitch CH to be formed on the sewing workpiece S and the target position of the stitch CH in the sewing machine coordinate system.

[0072] The movement conditions of the holding member 15 include a movement trajectory of the holding member 15 defined in the sewing machine coordinate system. The movement trajectory of the holding member 15 includes a movement trajectory of the holding member 15 in the XY plane. The movement conditions of the holding member 15 are determined based on the target pattern RP.

[0073] Therefore, as shown in Fig. 4, when performing multiple sewing processes to form multiple stitches CH, a first sewing process is performed to form a first stitch CH on the sewing object S based on a first target pattern RP1, and after the first sewing process, a second sewing process is performed to form a second stitch CH on the sewing object S based on a second target pattern RP2. Thereafter, the third sewing process to the Nth sewing process are performed sequentially according to the number N of stitches CH to be formed. Fig. 4 shows an example where N=10, and the first target pattern RP1 to the tenth target pattern RP10 are defined.

[0074] The control device 30 is connected to the image processing unit 42 via an input / output interface. The control device 30 outputs a control command for controlling the actuator 17 that moves the holding member 15 based on the processing result of the image processing unit 42.

[0075] (Image processing unit) The image processing unit 42 includes a computer system. The image processing unit 42 has a processor 44 such as a CPU (Central Processing Unit), a storage device 45 including a non-volatile memory such as a ROM (Read Only Memory) or storage and a volatile memory such as a RAM (Random Access Memory), and an input / output unit 46 including an input / output interface. The image processing unit 42 performs image processing to capture images of the sewing object S using the multiple imaging devices 41 and detect the recognition target UP from the images in accordance with a computer program stored in the storage device 45.

[0076] The image processing unit 42 is connected to a plurality of imaging devices 41 and a plurality of lighting devices 43A, 43B via an input / output unit 46. In Fig. 8, the input / output unit 46 is connected to each of the imaging devices 41 and the lighting devices 43A, 43B of the imaging unit 50A and the imaging unit 50B via a relay device 47. The relay device 47 is a line concentrator for connecting a plurality of devices, such as a USB (Universal Serial Bus) hub.

[0077] The image processing unit 42 has an image acquisition unit 44A, a synthesis processing unit 44B, a detection processing unit 44C, and a correction amount calculation unit 44D. The image acquisition unit 44A, the synthesis processing unit 44B, the detection processing unit 44C, and the correction amount calculation unit 44D are functional blocks realized by the processor 44 executing a computer program. The image acquisition unit 44A, the synthesis processing unit 44B, the detection processing unit 44C, and the correction amount calculation unit 44D may be configured separately by dedicated processors.

[0078] The image acquisition unit 44A acquires images of the sewing object S from a plurality of imaging devices 41. The imaging devices 41 capture images of the sewing object S and output the obtained images to the image processing unit 42. When the sewing object S is positioned at a predetermined imaging position by the control device 30, the image acquisition unit 44A controls the operation of the imaging device 41 corresponding to the imaging position and acquires the images of the sewing object S.

[0079] In FIG. 7, the imaging position Pf is set at the center of the enlarged imaging areas 61A and 61B. Imaging is performed with the center of the holding member 15 (sewing object S) being placed at the imaging position Pf of either of the enlarged imaging areas 61A and 61B. For example, in the imaging unit 50A, when the sewing object S is placed at the imaging position Pf of the enlarged imaging area 61A, the image acquisition section 44A turns on the illumination device 43A and causes the imaging device 41 of the mounting part 51A and the imaging device 41 of the mounting part 51B to perform imaging substantially simultaneously. As a result, an image corresponding to the imaging field of view 60A and an image corresponding to the imaging field of view 60B are acquired. In the imaging unit 50B, when the sewing object S is placed at the imaging position Pf of the enlarged imaging area 61B, the image acquisition section 44A turns on the illumination device 43B and causes the imaging device 41 of the mounting part 51D and the imaging device 41 of the mounting part 51F to perform imaging substantially simultaneously. The imaging may be performed in sequence.

[0080] The synthesis processing unit 44B performs image synthesis processing to join images from a plurality of imaging devices 41 together. The synthesis processing unit 44B joins a plurality of images constituting the enlarged imaging area (61A or 61B) based on the image portions of the overlapping areas JA to generate one synthetic image 75 (see FIG. 13). Therefore, the synthetic image 75 is an image that captures a range equivalent to the enlarged imaging area 61A or 61B. The enlarged imaging areas 61A and 61B have a size that encompasses a plurality of recognition targets UP of the sewing object S. Therefore, the synthetic image 75 is an image that captures a plurality of recognition targets UP. The synthetic image 75 preferably includes all recognition targets UP (UP1 to UP7) that are detected when forming one stitch CH.

[0081] The detection processing unit 44C detects a plurality of recognition targets UP from the composite image 75 stitched together by the synthesis processing unit 44B. The detection processing unit 44C detects the recognition targets UP in the composite image 75 by a pattern matching method, which is a type of image processing method. Here, the operator stores in advance in the storage device 45 a reference image (template) for recognizing the recognition targets UP of the sewing object S. The detection processing unit 44C acquires the reference image recorded in the storage device 45, and acquires the position coordinates of the recognition targets UP in the composite image 75 by matching with the reference image. The detection processing unit 44C sequentially detects a plurality of recognition targets UP included in the composite image 75, and acquires the position coordinates of each of the plurality of recognition targets UP.

[0082] The correction amount calculation unit 44D calculates correction data for correcting the displacement of the surface of the sewing object S based on the detection result by the detection processing unit 44C. The target pattern RP of the sewing data is generated on the premise that the sewing object S is not shrunk, and is stored in the control device 30. When the sewing process is performed and the sewing object S shrinks and the surface of the sewing object S is displaced, the correction amount calculation unit 44D determines the amount of displacement of the recognition object UP detected by the detection processing unit 44C, and generates correction data for correcting the sewing data.

[0083] Specifically, the correction amount calculation unit 44D generates correction data so that the relative positions in the sewing machine coordinate system between the target pattern RP and the recognition object UP before the sewing object S shrinks coincide with the relative positions in the sewing machine coordinate system between the stitch CH actually formed on the sewing object S by the sewing process and the recognition object UP after the sewing object S shrinks.

[0084] The image processing unit 42 transmits the correction data calculated by the correction amount calculation unit 44D to the control device 30. The control device 30 executes the sewing process based on the correction data.

[0085] [Sewing operation] 9 is a flowchart for explaining the sewing operation of the sewing machine 1 according to this embodiment. The sewing operation of the sewing machine 1 includes an image capturing process S1, an image joining process S2, a recognition target detection process S3, a correction data calculation process S4, a sewing process S5, and an end determination process S6. The image processing method according to this embodiment is performed as a part of the sewing operation of the sewing machine 1. The image processing method according to this embodiment includes the image joining process S2 and the recognition target detection process S3.

[0086] (Image capture processing) The imaging process S1 is a process of imaging the sewing object S by the multiple imaging devices 41. Whether to capture an image of the enlarged imaging area 61A extending in the X-axis direction by the imaging unit 50A or to capture an image of the enlarged imaging area 61B extending in the Y-axis direction by the imaging unit 50B is set in advance by the operator or by the control device 30 based on sewing data. The control device 30 moves the holding member 15 that holds the sewing object S to place the holding member 15 at a preset imaging position. When the holding member 15 is placed at the corresponding imaging position Pf, the image processing unit 42 performs imaging by the set multiple imaging devices 41.

[0087] In the following, as an example, a case where an image of the enlarged imaging area 61A is acquired by each imaging device 41 of the imaging unit 50A in Fig. 7 will be described. In this case, the image acquisition section 44A acquires an image corresponding to the imaging field of view 60A and an image corresponding to the imaging field of view 60B by the imaging device 41 of the mounting part 51A and the imaging device 41 of the mounting part 51B.

[0088] (Image stitching process) The image joining process S2 is a process of joining together images of the sewing object S captured by the multiple imaging devices 41.

[0089] Fig. 10 is a flowchart showing the image joining process S2 according to this embodiment. Fig. 11 is a schematic diagram for explaining the image positioning process S2A in the image joining process S2. Fig. 12 is a schematic diagram for explaining the mask area setting process S2B in the image joining process S2. Fig. 13 is a schematic diagram for explaining the synthesis process S2C in the image joining process S2.

[0090] 11, an image obtained by the imaging device 41 of the mounting part 51A is defined as a first image 71. An image obtained by the imaging device 41 of the mounting part 51B is defined as a second image 72. In this case, the second image 72 includes an area on the +X direction side with respect to the first image 71. The first image 71 corresponds to the imaging field of view 60A, and the second image 72 corresponds to the imaging field of view 60B. The first image 71 and the second image 72 share an overlapping area JA.

[0091] The overlapping area JA between the first image 71 and the second image 72 includes image elements GE that are common to at least a portion of the recognition target UP. That is, as shown in Fig. 4, when the recognition target UP is an array of a plurality of holes 7 that exist in a predetermined pattern in the sewing object S, the image elements GE are images of the individual holes 7. The holes 7 are components that make up the recognition target UP and the reference pattern DPh. For ease of explanation, Figs. 11 to 13 show an example in which the arrangement of the holes 7 is simplified and the holes 7 are arranged vertically and horizontally in an array.

[0092] In detecting the recognition target UP by pattern matching, it is important that there are no missing or error image information in the image elements GE of the holes 7 that constitute the recognition target UP. Therefore, in the image joining process S2 shown in Fig. 10, the first image 71 and the second image 72 are combined so that the information of the image elements GE that constitute the recognition target UP is preserved when joining the first image 71 and the second image 72.

[0093] 10, the image stitching process S2 includes an image positioning process S2A, a mask area setting process S2B, and a synthesis process S2C. Prior to the image stitching process S2, the synthesis processing unit 44B may binarize each image obtained in the imaging process S1.

[0094] Image Positioning Processing The image positioning process S2A is a process for positioning the multiple images obtained in the imaging process S1 based on the image elements GE in the overlapping area JA.

[0095] As shown in FIG. 11, the synthesis processing unit 44B detects image elements GE included in the overlapping area JA of the first image 71 and image elements GE included in the overlapping area JA of the second image 72. Since the position coordinates of the imaging center of the imaging device 41 attached to the attachment parts 51A and 51B and the size of the imaging fields of view 60A and 60B in the sewing machine coordinate system are known, the range of the overlapping area JA in each image is known. The synthesis processing unit 44B selects three or more image elements GE included in the overlapping area JA. In the example of FIG. 11, the synthesis processing unit 44B divides the overlapping area JA vertically and horizontally into four, and selects one image element GE from each divided area, a total of four image elements GE, as the image element of interest GE1.

[0096] The synthesis processing unit 44B deforms the second image 72 so that the same image elements GE in the overlapping area JA match each other. That is, the synthesis processing unit 44B deforms the second image 72 so that the four target image elements GE1 in the second image 72 match the four target image elements GE1 in the first image 71. The deformation of the second image 72 is, for example, by affine transformation. The deformation may include X-axis coordinates, Y-axis coordinates, θZ direction rotation, scale adjustment (enlargement / reduction), and distortion correction (shear deformation). The positions and shapes of the four target image elements GE1 in the first image 71 and the deformed second image 72 (hereinafter referred to as the second image 72A) match each other. Therefore, the images in the overlapping area JA of the first image 71 and the deformed second image 72A substantially match each other. The image positioning process S2A eliminates differences in image distortion caused by an assembly error in the installation positions of the captured first image 71 and second image 72A, an aberration of the optical system, etc. As described later, the synthesis process unit 44B joins the transformed second image 72A with the first image 71.

[0097] Mask area setting process The mask area setting process S2B (see FIG. 10) is a process for setting an area to be joined by the synthesis process S2C for the first image 71 and the second image 72A aligned by the image positioning process S2A. In this embodiment, the synthesis processing unit 44B sets a joining position in the overlap area JA so that the entire image element GE detected in the overlap area JA is included in either the first image 71 or the second image 72A.

[0098] As shown in Figure 12, the synthesis processing unit 44B extracts an image element GE that is adjacent to the edge EL on the second image 72A side in the overlap area JA, and sets a first mask area 73 in the first image 71 excluding the extracted image element GE.

[0099] For example, the synthesis processing unit 44B adds a straight line SL to the end side EL and extracts the contour with the largest area within the overlapping area JA. Adding a straight line SL to the end side EL means setting a predetermined pixel value to the pixel row that constitutes the end side EL in the overlapping area JA. If there is an image element GE that is in contact with the end side EL, that image element GE is connected to the straight line SL. As a result, when the contour with the largest area within the overlapping area JA is extracted, only the image element GE connected to the straight line SL (i.e., in contact with the end side EL) is extracted. The straight line SL and the contour of the image element GE connected by the straight line SL are referred to as the contour area 76.

[0100] The synthesis processing unit 44B sets the area of ​​the first image 71 excluding the extracted contour area 76 as a first mask area 73. The first mask area 73 is an area of ​​the first image 71 that is used as pixels that constitute the synthesized image 75. Since the contour area 76 of the first image 71 is excluded from the first mask area 73, it is not used in the synthesis process.

[0101] Next, the synthesis processing unit 44B sets a second mask area 74 adjacent to the first mask area 73 and including the entire image element GE that contacts the end side EL of the overlap area JA in the second image 72A.

[0102] The synthesis processing unit 44B sets the portion of the second image 72A that is extracted from the overlapping area JA as the second mask area 74, together with the area unique to the second image 72A that is not included in the overlapping area JA. Since the portion of the outline area 76 is excluded from the first mask area 73, the second mask area 74 is adjacent to the first mask area 73 without any gaps at the portion of the outline area 76. The second mask area 74 is an area of ​​the second image 72A that is used as pixels that constitute the synthetic image 75.

[0103] Therefore, the first mask region 73 includes the unique region of the first image 71 (the region included only in the first image 71 and not included in the second image 72A) and the entire region of the overlapping region JA excluding the contour region 76. The second mask region 74 includes the unique region of the second image 72A (the region included only in the second image 72A and not included in the first image 71) and the contour region 76 of the overlapping region JA.

[0104] Here, the image element GE that is in contact with the edge EL will be described. The edge EL of the overlapping area JA on the second image 72A side is the edge on the +X direction side of the first image 71. Therefore, the image element GE that is in contact with the edge EL of the overlapping area JA is in contact with the edge of the first image 71. The image element GE that is in contact with the edge of the first image 71 may have a portion that is not shown in the first image 71 (portion P2 on the +X side of the edge EL). In other words, only the portion P1 of the hole 7 that is on the -X side of the edge EL may be shown in the first image 71.

[0105] If the position of the edge EL is set at the boundary between the first mask area 73 and the second mask area 74, the -X side portion P1 is obtained from the first image 71, and the +X side portion P2 is obtained from the second image 72A, which may cause information loss or errors. On the other hand, the edge EL of the overlap area JA on the second image 72A side is located at the center of the second image 72A. Therefore, the entire image element GE (all of the portions P1 and P2) that contacts the edge EL of the overlap area JA is reliably captured in the second image 72A. Therefore, by including the portion of the outline area 76 in the second mask area 74 of the second image 72A, a composite image 75 that completely preserves the information of the image element GE of the hole 7 that exists at a position straddling the edge EL can be generated.

[0106] In addition, there may be a case where there is no image element GE that contacts the edge EL of the overlapping area JA. In this case, the synthesis processing unit 44B may set the first mask area 73 and the second mask area 74 with the edge EL as the boundary. That is, the first mask area 73 may be set as the entire first image 71 including the overlapping area JA. The second mask area 74 may be set as a unique area of ​​the second image 72A excluding the overlapping area JA with the edge EL as the boundary. When the recognition target UP is an array of multiple image elements GE (i.e., holes 7), the area other than the image elements GE in the image corresponds to the "background" that is not subject to processing in the image recognition process. Therefore, when the position of the edge EL corresponds to the background, even if a loss or error in image information occurs, it does not affect the detection accuracy of the recognition target UP.

[0107] <Composition Processing> The synthesis process S2C (see FIG. 10) is a process of joining the first image 71 and the second image 72A using the boundary between the first mask region 73 of the first image 71 and the second mask region 74 of the second image 72A as a joining position. As shown in FIG. 13, the synthesis processing unit 44B employs each pixel belonging to the first mask region 73 in the first image 71 and each pixel belonging to the second mask region 74 in the second image 72A to generate a synthetic image 75 by joining the first image 71 and the second image 72A. The generated synthetic image 75 is an image of a size corresponding to the enlarged imaging region 61A. In the synthetic image 75, the region corresponding to the first mask region 73 is an image region acquired from the first image 71, and the region corresponding to the second mask region 74 is an image region acquired from the second image 72. Since each image element GE included in the composite image 75 is entirely acquired from either the first image 71 or the second image 72A, there is no loss or error in image information regarding the image element GE. In this manner, the image stitching process S2 is performed.

[0108] (Recognition target detection process) Next, the recognition target detection process S3 in Fig. 9 is a process for detecting the recognition target UP from the composite image 75 generated by the image stitching process S2 and acquiring the position coordinates of the detected recognition target UP. Fig. 14 is a schematic diagram for explaining the recognition target detection process S3.

[0109] As shown in FIG. 14, the detection processing unit 44C extracts an image of the recognition area AR in which the detection process of the recognition target UP is performed from the composite image 75. The recognition area AR is set in a range that takes into consideration the expected positional deviation amount of the recognition target UP due to stitch formation with respect to the design position coordinates of the recognition target UP set in the sewing data. FIG. 14 shows a schematic diagram of the setting of the recognition area AR for a target pattern RP of an arbitrary shape. The detection processing unit 44C sets the recognition area AR for each of the multiple recognition targets UP to be detected in one stitch formation, and obtains an extracted image 77 in which the image of the set recognition area AR is cut out. For example, in the example of FIG. 4, the recognition area AR is set for each of the seven recognition targets UP from the recognition target UP1 to the recognition target UP7. Note that the composition process S2C in FIG. 10 and FIG. 13 (i.e., the generation of the composite image 75) may be performed only when the recognition area AR includes the boundary (edge ​​EL) of the overlapping area JA.

[0110] The detection processing unit 44C queues (sets the processing order) each of the acquired extracted images 77, and performs image recognition processing by the pattern matching method in order. The detection processing unit 44C reads out a reference image from the storage device 45, and acquires the position coordinates of the recognition target UP in the extracted image 77 by matching with the reference image. As shown in FIG. 14, the processor 44 that realizes the function of the image processing unit 42 may include multiple processor cores CR. The processor cores CR are units of the arithmetic processing structure provided in the processor 44, and each can execute arithmetic processing according to a computer program. When the processor 44 includes multiple processor cores CR, the image processing unit 42 executes image recognition processing of each of the queued extracted images 77 in parallel using the multiple processor cores CR. If the number of available processor cores CR is K, image recognition processing of K extracted images 77 can be simultaneously processed in parallel, thereby reducing the processing time.

[0111] Through the recognition target detection process S3, the detection processing section 44C acquires the position coordinates in the sewing machine coordinate system of each recognition target UP (UP1 to UP7, see FIG. 4) to be detected in one stitch formation.

[0112] (Correction data calculation process) The correction data calculation process S4 is a process for calculating correction data for correcting the displacement of the surface of the sewing object S based on the detection result by the recognition object detection process S3. The correction amount calculation unit 44D calculates the displacement amount of the recognition object UP from the initial position to the current position based on the initial position and the current position of the recognition object UP related to the current sewing process. The initial position of the recognition object UP is the design position coordinate of the recognition object UP set in the sewing data. The current position of the recognition object UP is the position coordinate of the recognition object UP detected by the recognition object detection process S3. The correction amount calculation unit 44D calculates the correction position coordinate of each correction point CP (see FIG. 14) on the target pattern RP based on the calculated displacement amount so that the stitch CH is formed at the target position in the sewing machine coordinate system, and calculates the correction data of the target pattern RP passing through the correction point CP. Note that in the first sewing process (first sewing process), since no displacement of the surface of the sewing object S occurs due to stitch formation, no displacement of the recognition object UP occurs either. Therefore, in the initial sewing process (first sewing process), the target pattern RP in the initial state defined by the sewing data is used as is, and no correction data is calculated.

[0113] (Sewing process) The sewing process S5 is a process for forming a stitch CH based on a target pattern RP. The sewing process S5 is performed by controlling each part of the sewing machine 1 by the control device 30. The sewing process includes a first sewing process to an N-th sewing process (N=10 in FIG. 4). The first sewing process is performed based on the target pattern RP in the initial state defined by the sewing data. The second sewing process to the N-th sewing process are performed based on the correction data (the corrected target pattern RP) calculated by the correction data calculation process S4. In the first sewing process, the control device 30 outputs a control command to the actuator 17 so that the stitch CH is formed according to the target pattern RP in the initial state defined by the sewing data. After the second sewing process, the control device 30 acquires the correction data from the image processing unit 42, and outputs a control command to the actuator 17 so that the stitch CH is formed according to the target pattern RP of the acquired correction data.

[0114] The end determination process S6 is a process for determining whether or not the sewing process of the sewing object S has been completed. The control device 30 determines whether or not the sewing process of the sewing object S has been completed based on the sewing data. When the first sewing process to the (N-1)th sewing process have been completed, the control device 30 determines in the end determination process S6 that the sewing process has not been completed. When the Nth sewing process has been completed, the control device 30 determines in the end determination process S6 that the sewing process has been completed.

[0115] In this manner, in the first to Nth sewing processes, the imaging process S1, the image joining process S2, the recognition target detection process S3, and the correction data calculation process S4 are performed each time one sewing process is completed, and the correction data for the next sewing process is calculated, and the next sewing process is performed based on the calculated correction data. This series of processes is repeated. In this embodiment, the multiple recognition targets UP detected in one sewing process can be detected from the composite image 75 obtained by one imaging process using the multiple imaging devices 41. As a result, the processing time required for the imaging process S1, the image joining process S2, the recognition target detection process S3, and the correction data calculation process S4, which are performed each time a sewing process is performed, is reduced.

[0116] [effect] As described above, according to this embodiment, images of the sewing object S captured by the multiple imaging devices 41 are joined together, and multiple recognition targets UP of the sewing object S are detected from the joined composite image 75. This reduces the number of times the sewing object S is moved and the number of times images are captured, compared to the case where multiple recognition targets UP are captured and detected one by one, so that the time required to detect the recognition target UP associated with the sewing process can be shortened. As a result, the detection time of the recognition target UP performed for each sewing process can be shortened, so that the processing time of the entire sewing work can be shortened. In addition, an overlapping area JA is provided in the imaging field of view of the multiple imaging devices 41, and the same image elements GE in the overlapping area JA are matched with each other, so that high positional accuracy can be obtained even when images are joined together. In addition, the joining position is set so that the entire image element GE detected in the overlapping area JA is included in either the first image 71 or the second image 72, so that missing or error in information of the image element GE constituting the recognition target UP can be prevented. As a result, the detection accuracy of the recognition target UP can be improved. In addition, image elements GE that contact the edge EL on the second image 72 side in the overlapping area JA are extracted, and a second mask area 74 including the entire extracted image elements GE is set in the second image 72, so image elements GE that may lose image information during joining can be easily extracted, and joining can be performed while preserving the entire image elements GE. This reduces the calculation load associated with the joining process and shortens the processing time.

[0117] In this embodiment, the image elements GE are holes 7 formed so as to be scattered in the sewing object S, and even if the recognition objects UP are an array of a plurality of holes 7 that exist in a predetermined pattern in the sewing object S, the recognition objects UP can be detected accurately and in a short time. Since the plurality of imaging devices 41 are arranged along at least one of the longitudinal direction (X-axis direction) and the lateral direction (Y-axis direction) of the sewing area SA, a composite image 75 can be generated that can collectively detect a plurality of recognition objects UP according to the extending direction of the stitch CH. Furthermore, since the plurality of illumination devices 43 (43A, 43B) are provided on both outsides of the array of the plurality of imaging devices 41 and between the array of the plurality of imaging devices 41, the luminance (pixel value) in the image can be made uniform even when a wide range of composite image 75 is generated, and the detection accuracy can be improved.

[0118] [Other embodiments] In the above embodiment, an example in which four imaging devices 41 are provided has been shown, but the number of imaging devices 41 may be two, three, five or more. The imaging devices 41 may be attached at six or more locations. The imaging devices 41 may be arranged along a direction other than the longitudinal direction (X-axis direction) and the lateral direction (Y-axis direction) of the sewing area SA. The number of lighting devices 43A, 43B is not limited, and may be one or two or more. An example in which the image element GE is a hole 7 formed in the sewing object S has been shown, but the image element GE is not particularly limited, and may be an image element other than the hole 7.

[0119] In the above embodiment, the second image 72 is deformed to fit the first image 71, but the first image 71 may be deformed to fit the second image 72. In addition, either of the two images to be joined may be the first image 71 or the second image 72. In the image positioning process S2A, only the positioning may be performed without deforming the second image 72.

[0120] In the above embodiment, each pixel constituting the composite image 75 is adopted from either the first image 71 or the second image 72, but both the first image 71 and the second image 72 may be used for each pixel corresponding to the overlapping region JA of the composite image 75. For example, for the overlapping region JA, an averaging process may be performed using both the first image 71 and the second image 72, or an image obtained by binarizing both images and then taking a logical AND or OR of both images may be adopted. [Explanation of symbols]

[0121] 1... sewing machine, 2... table, 3... sewing machine needle, 4... surface material, 5... padding material, 6... backing material, 7... hole, 10... sewing machine body, 11A... horizontal arm, 11B... bed, 11C... vertical arm, 11D... head, 12... needle bar, 13... needle plate, 14... support member, 11... sewing machine frame, 15... holding member, 15A... presser member, 15B... lower plate, 16... actuator, 17... actuator, 17X... X-axis motor, 17Y... Y-axis motor, 18... actuator, 19... intermediate presser member, 20... operation device, 21... operation panel, 22... operation pedal, 30... control device, 31... drive amount sensor, 32... drive amount sensor, 32X... X-axis sensor, 32Y... Y-axis sensor, 33... second imaging device, 33A, 33B... mounting portion, 40... image processing device, 41... imaging device, 42... image processing section, 43A, 43B... lighting device, 44... processor, 44A... image acquisition section, 44B... synthesis processing unit, 44C...detection processing unit, 44D...correction amount calculation unit, 45...storage device, 46...input / output unit, 47...relay device, 50A...imaging unit, 50B...imaging unit, 51A, 51B, 51C, 51D, 51E, 51F...mounting unit, 60A, 60B, 60D, 60F...imaging field of view, 61A, 61B...enlarged imaging area, 71...first image, 72, 72A...second image, 73...first mask area, 74...second mask area, 75...composite image, 7 6...contour area, 77...extracted image, AR...recognition area, CH (CH1 to CH10)...stitch, CP...correction point, CR...processor core, DPh...reference pattern, EL...edge, GE...image element, GE1...target image element, JA...overlapping area, P1, P2...portion of image element, Pf...imaging position, Ps...sewing position, RP (RP1 to RP10)...target pattern, S...sewing object, SA...sewing area, SL...straight line, UP...recognition object.

Claims

1. A sewing machine body includes a head for sewing and a holding member for holding and moving an object to be sewn. A plurality of image capturing devices are arranged at intervals in a horizontal plane so as to be aligned with the sewing machine body. an image processing unit that detects a recognition target of the sewing object based on images of the sewing object captured by the multiple imaging devices, The image processing unit includes: A synthesis processing unit that stitches together images from the plurality of imaging devices; a detection processing unit that detects the plurality of recognition targets from the composite image stitched together by the synthesis processing unit, Image processing device.

2. each of the plurality of imaging devices is arranged to have an overlapping region in which an imaging field of view partially overlaps with an adjacent imaging device; The synthesis processing unit includes: detecting an image element common to at least a portion of the recognition target in the overlapping region of the first image and the second image to be joined; transforming the second image so that the identical image elements in the overlapping region coincide with each other, and stitching the transformed second image with the first image; The image processing device according to claim 1 .

3. the synthesis processing unit sets a joining position in the overlapping region such that the entirety of the image element detected in the overlapping region is included in either the first image or the second image. The image processing device according to claim 2 .

4. The synthesis processing unit includes: extracting the image element that is in contact with an edge of the overlapping region on the side of the second image, and setting a first mask region, which excludes the extracted image element, on the first image; A second mask area is set in the second image, the second mask area being adjacent to the first mask area and including the entire image element that is in contact with the edge; the first image and the second image are joined together, with a boundary between the first mask region of the first image and the second mask region of the second image being set as the joining position; The image processing device according to claim 3 .

5. the image elements are holes formed so as to be scattered on the sewing object, the recognition target is an arrangement of a plurality of the holes that are present in a predetermined pattern on the sewing object, The image processing device according to claim 2 .

6. the plurality of imaging devices are arranged along at least one of a longitudinal direction and a lateral direction of a sewing area in which the sewing object can be moved by the holding member and sewn by the head; The image processing device according to claim 1 .

7. The system further includes a plurality of illumination devices respectively disposed on both outer sides of the array of the plurality of image capture devices and between the array of the plurality of image capture devices. The image processing device according to claim 1 .

8. A sewing machine body including a head for sewing and a holding member for holding and moving an object to be sewn; An image processing device according to any one of claims 1 to 4, and a control device that controls the sewing machine body based on a processing result of the image processing device. sewing machine.

9. stitching together images of the sewing object captured by a plurality of image capturing devices arranged at intervals in a predetermined direction in a horizontal plane so as to be aligned with a sewing machine body including a head for sewing and a holding member for holding and moving the sewing object; Detecting a plurality of recognition targets of the sewing object from the joined composite image. Image processing methods.

Images