Image processing apparatus, sewing machine, and image processing method

Abstract

An image processing device (40) is provided with: a plurality of imaging devices (41) arranged at intervals in a horizontal plane so as to be aligned with respect to a sewing machine main body (10) having a head part (11D) for sewing and a holding member (15) for holding and moving an object (S) to be sewn; and an image processing unit (42) that detects an identification object included in the sewing object (S) on the basis of the images of the sewing object (S) captured by the plurality of image capturing devices (41). The image processing unit (42) has: a combining processing unit (44B) that combines the images of the plurality of imaging devices (41); and a detection processing unit (44C) that detects the plurality of recognition objects (UP) on the basis of the combined image (75) joined by the combination processing unit (44B).

Description

Technical Field

[0001] This invention relates to an image processing apparatus, a sewing machine, and an image processing method. Background Technology

[0002] Sometimes, stitches are formed on the sewn object to improve its appearance. Patent Document 1 discloses a technique for forming stitches on the surface material used in vehicle seats.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2013-162957 Summary of the Invention

[0004] The cover material used for vehicle seats has thickness and elasticity. If a stitch is formed on a thick and elastic sewn object, the sewn object may shrink and its surface may shift. For example, it is preferable to form a second stitch after forming a first stitch, in which case the second stitch is formed at a target position on the sewn object in accordance with the displacement of the sewn object's surface caused by forming the first stitch.

[0005] As a countermeasure to ensure that the stitches are formed at the target location on the object being sewn, one approach is to photograph the surface of the object before sewing and detect any displacement on the surface. However, if multiple points along the stitch line are photographed sequentially at each location, and displacement is detected at each point individually, the photographing and image recognition processing takes time. Since the displacement changes each time a stitch is formed, photographing and displacement detection are performed each time a stitch is formed, thus increasing the overall processing time of the sewing operation.

[0006] The purpose of this invention is to shorten the overall processing time of the sewing industry.

[0007] According to a first aspect of the present invention, an image processing apparatus is provided, comprising: a plurality of imaging devices arranged at intervals in a horizontal plane relative to a sewing machine body, the sewing machine body having a sewing head and a holding member for holding and moving a sewing object; and an image processing unit that detects identification objects of the sewing object based on images of the sewing object captured by the plurality of imaging devices, the image processing unit comprising: a synthesis processing unit that combines the images of the plurality of imaging devices; and a detection processing unit that detects the plurality of identification objects based on the synthesized image formed by the synthesis processing unit.

[0008] According to a second aspect of the present invention, a sewing machine is provided, comprising: a sewing machine body having a sewing head and a holding member for holding and moving the sewing object; an image processing device according to the first aspect; and a control device that controls the sewing machine body based on the processing result of the image processing device.

[0009] According to a third aspect of the present invention, an image processing method is provided, comprising the steps of: combining images of a sewn object captured by a plurality of imaging devices, the plurality of imaging devices being arranged at intervals in a predetermined direction in a horizontal plane relative to a sewing machine body, the sewing machine body having a sewing head and a holding member for holding the sewn object and moving the sewn object; and detecting a plurality of identifying objects present in the sewn object based on the combined image.

[0010] The effects of the invention

[0011] According to the present invention, the overall processing time of the sewing operation can be shortened. Attached Figure Description

[0012] Figure 1 This is a perspective view of the sewing machine according to this embodiment.

[0013] Figure 2 This is a perspective view showing a portion of the sewing machine involved in this embodiment.

[0014] Figure 3 This is a cross-sectional view showing a portion of the sewing object involved in this embodiment.

[0015] Figure 4 This is a top view showing the sewing object involved in this embodiment.

[0016] Figure 5 This is a cross-sectional view showing a portion of the sewing object involved in this embodiment.

[0017] Figure 6 This is a top view illustrating the mounting position of the imaging device involved in this embodiment.

[0018] Figure 7 This is a top view illustrating an example configuration of the imaging device according to this embodiment.

[0019] Figure 8 This is a functional block diagram illustrating the sewing machine involved in this embodiment.

[0020] Figure 9 This is a flowchart used to explain the sewing operation of the sewing machine involved in this embodiment.

[0021] Figure 10 This is a flowchart illustrating the image joining process involved in this embodiment.

[0022] Figure 11 This is a schematic diagram illustrating the image positioning process in image stitching.

[0023] Figure 12 This is a schematic diagram illustrating the mask region setting process in image stitching.

[0024] Figure 13 This is a schematic diagram illustrating the compositing process in image joining.

[0025] Figure 14 This is a schematic diagram illustrating the object detection process. Detailed Implementation

[0026] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings, but the present invention is not limited thereto. The structural elements of the embodiments described below can be appropriately combined. In addition, sometimes some structural elements are not used.

[0027] In this embodiment, a local coordinate system is defined for the sewing machine 1. In the following description, the local coordinate system defined for the sewing machine 1 will be appropriately referred to as the sewing machine coordinate system. The sewing machine coordinate system is defined using an XYZ orthogonal coordinate system. In this embodiment, the positional relationships of each part will be explained based on the sewing machine coordinate system. The direction parallel to the X-axis within the defined plane is defined as the X-axis direction. The direction parallel to the Y-axis within the defined plane and orthogonal to the X-axis is defined as the Y-axis direction. The direction parallel to the Z-axis and orthogonal to the defined plane is defined as the Z-axis direction. Furthermore, the rotational or tilting direction centered on the X-axis is defined as the θX direction. The rotational or tilting direction centered on the Y-axis is defined as the θY direction. The rotational or tilting direction centered on the Z-axis is defined as the θZ direction. Additionally, in this embodiment, the plane including the X-axis and Y-axis is appropriately referred to as the XY plane. The plane including the X-axis and Z-axis is appropriately referred to as the XZ plane. The plane including the Y-axis and Z-axis is appropriately referred to as the YZ plane. The XY plane is parallel to the defined plane. The XY plane, XZ plane, and YZ plane are orthogonal. Furthermore, in this embodiment, the XY plane is parallel to the horizontal plane. The Z-axis direction is the vertical direction. The +Z direction is the upward direction, and the -Z direction is the downward direction. Additionally, the XY plane may also be inclined relative to the horizontal plane.

[0028] [Sewing machine]

[0029] Figure 1 This is a perspective view of the sewing machine 1 according to this embodiment. Figure 2This is a perspective view showing a portion of the sewing machine 1 according to this embodiment. In this embodiment, the sewing machine 1 is an electronic circulating sewing machine. The sewing machine 1 includes a sewing machine body 10, an operating device 20, a control device 30, and an image processing device 40. The operating device 20 is operated by the operator. The control device 30 controls the sewing machine body 10. The image processing device 40 captures an image of the object being sewn, S, and performs image recognition. The image processing device 40 includes multiple capturing devices 41 and an image processing unit 42.

[0030] The sewing machine body 10 is mounted on the upper surface of the worktable 2. For example... Figure 2 As shown, the sewing machine body 10 includes a sewing machine frame 11, a needle bar 12, a needle plate 13, a holding member 15, an actuator 16, an actuator 17, and an actuator 18. The needle bar 12 and needle plate 13 are respectively supported on the sewing machine frame 11. The holding member 15 is supported on the sewing machine frame 11 via a supporting member 14. The actuator 16 (see reference...) Figure 1 This generates the power to move the needle bar 12. Actuator 17 (see reference) Figure 1 This generates the power to move the retaining member 15. Actuator 18 (see reference) Figure 1 This generates power to move at least a portion of the retaining member 15.

[0031] The sewing machine frame 11 has a horizontal arm 11A, a base 11B, a vertical arm 11C, and a head 11D. The horizontal arm 11A extends in the Y-axis direction. The base 11B (see reference) Figure 1 The vertical arm 11C is positioned below the horizontal arm 11A. Figure 1 The horizontal arm 11A is configured to connect the +Y side end to the base 11B. The head 11D is configured on the -Y side of the horizontal arm 11A.

[0032] The needle bar 12 holds the sewing machine needle 3. The needle bar 12 holds the sewing machine needle 3 in a manner that makes the sewing machine needle 3 parallel to the Z-axis. The needle bar 12 is movably supported on the head 11D in the Z-axis direction.

[0033] The needle plate 13 supports the object S being sewn. The needle plate 13 also supports the holding member 15. The needle plate 13 is supported on the base 11B (see reference). Figure 1 The needle plate 13 is positioned below the retaining member 15.

[0034] The holding member 15 holds the sewing object S. The holding member 15 holds the sewing object S in the XY plane, including the sewing position Ps directly below the sewing machine needle 3, and is movable. The holding member 15 can hold the sewing object S in the XY plane, including the position directly below the imaging device 41. While holding the sewing object S, the holding member 15 moves in the XY plane based on sewing data. As a result, a stitch CH (see reference) is formed on the sewing object S after passing the sewing position Ps. Figure 5 The retaining member 15 is supported on the horizontal arm 11A via the supporting member 14.

[0035] The holding member 15 has a presser foot member 15A and a lower plate 15B opposite to the presser foot member 15A. The presser foot member 15A is a frame-shaped member. The presser foot member 15A is movable in the Z-axis direction. The lower plate 15B is disposed below the presser foot member 15A. The holding member 15 holds the sewing object S by clamping it with the presser foot member 15A and the lower plate 15B.

[0036] If the presser foot component 15A moves in the +Z direction, the presser foot component 15A and the lower plate 15B separate. This allows the operator to place the workpiece S between the presser foot component 15A and the lower plate 15B. If the presser foot component 15A moves in the -Z direction with the workpiece S placed between the presser foot component 15A and the lower plate 15B, the workpiece S is clamped between the presser foot component 15A and the lower plate 15B. Thus, the workpiece S is held by the holding member 15. Furthermore, if the presser foot component 15A moves in the +Z direction, the holding of the workpiece S by the holding member 15 is released. This allows the operator to remove the workpiece S from between the presser foot component 15A and the lower plate 15B.

[0037] like Figure 1 As shown, actuator 16 generates power to move needle bar 12 in the Z-axis direction. Actuator 16 includes a pulse motor. Actuator 16 is disposed on horizontal arm 11A.

[0038] A horizontal arm shaft extending in the Y-axis direction is configured inside the horizontal arm 11A. An actuator 16 is connected to the +Y side end of the horizontal arm shaft. The -Y side end of the horizontal arm shaft is connected to the needle bar 12 via a power transmission mechanism configured inside the head 11D. The horizontal arm shaft rotates due to the operation of the actuator 16. The power generated by the actuator 16 is transmitted to the needle bar 12 via the horizontal arm shaft and the power transmission mechanism. Consequently, the sewing machine needle 3 held by the needle bar 12 reciprocates in the Z-axis direction.

[0039] A timing belt extending in the Z-axis direction is arranged inside the vertical arm 11C. Additionally, a base shaft extending in the Y-axis direction is arranged inside the base 11B. Pulleys are respectively arranged on the horizontal arm shaft and the base shaft. The timing belt is mounted on the pulleys on both the horizontal arm shaft and the base shaft. The horizontal arm shaft and the base shaft are connected via a power transmission mechanism including the timing belt.

[0040] A vessel is disposed inside the base 11B. The vessel houses the spool that has been placed into the spool housing. The horizontal arm shaft and the base shaft rotate independently due to the operation of the actuator 16. The power generated by the actuator 16 is transmitted to the vessel via the horizontal arm shaft, the timing belt, and the base shaft. Thus, the vessel and the needle bar 12 rotate synchronously in the Z-axis reciprocating motion.

[0041] Actuator 17 generates power to move holding member 15 in the XY plane. Actuator 17 includes a pulse motor. Actuator 17 includes an X-axis motor 17X and a Y-axis motor 17Y. X-axis motor 17X (see reference) Figure 8 This generates power to move the holding member 15 in the X-axis direction. The Y-axis motor 17Y (see reference) Figure 8 This generates power to move the retaining member 15 in the Y-axis direction. The actuator 17 is disposed inside the base 11B.

[0042] The power generated by the actuator 17 is transmitted to the holding member 15 via the support member 14. Thus, the holding member 15 can move between the sewing machine needle 3 and the needle plate 13 in the X-axis and Y-axis directions, respectively. Through the operation of the actuator 17, the holding member 15 can hold and move the sewing object S within the XY plane, including the sewing position Ps directly below the sewing machine needle 3.

[0043] Actuator 18 generates power to move presser foot member 15A of holding member 15 in the Z-axis direction. Actuator 18 includes a pulse motor. Presser foot member 15A moves in the +Z direction, thereby separating presser foot member 15A from lower plate 15B. Presser foot member 15A moves in the -Z direction, thereby clamping the sewing object S by presser foot member 15A and lower plate 15B.

[0044] like Figure 2As shown, the sewing machine body 10 has a presser foot component 19 arranged around the sewing machine needle 3. The presser foot component 19 presses down on the workpiece S to be sewn around the sewing machine needle 3. The presser foot component 19 is supported on the head 11D in a state that allows it to move in the Z-axis direction. A presser foot motor is arranged inside the head 11D. The presser foot motor generates power to move the presser foot component 19 in the Z-axis direction. Through the operation of the presser foot motor, the presser foot component 19 moves synchronously with the needle bar 12 in the Z-axis direction. The presser foot component 19 suppresses the lifting of the workpiece S caused by the movement of the sewing machine needle 3.

[0045] like Figure 1 As shown, the operating device 20 receives input from the operator. The sewing machine 1 operates by operating the operating device 20. The operating device 20 includes an operating panel 21 and an operating pedal 22. The operating panel 21 is mounted on the upper surface of the worktable 2. The operating pedal 22 is located below the worktable 2. The operator operates the operating pedal 22 using their foot. The sewing machine 1 operates by the operator operating at least one of the operating panel 21 and the operating pedal 22.

[0046] Before the sewing process begins, the image processing device 40 performs the following processing: it acquires images of the object to be sewn, S, through multiple imaging devices 41; it detects the target position of the sewing, through the image processing unit 42; and it calculates correction data for the displacement caused by the completed stitch CH, etc.

[0047] Multiple imaging devices 41 capture images of the sewn object S held by the holding member 15. Each imaging device 41 has an optical system and an image sensor that receives light incident through the optical system. The image sensor may be a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0048] Multiple shooting devices 41 are positioned above the needle plate 13 and the holding member 15. Each of the multiple shooting devices 41 takes a picture from above of at least a portion of the sewing object S held by the holding member 15. The number of shooting devices 41 is two or more, but is not particularly limited thereto. Details regarding the arrangement of the multiple shooting devices 41 or the field of view, etc., will be described later.

[0049] The image processing unit 42 includes a computer system. The image processing unit 42 processes images relating to the sewing object S. Based on images of the sewing object S captured by multiple imaging devices 41, the image processing unit 42 detects objects that can be identified in the sewing object S.

[0050] [Object to be sewn]

[0051] Figure 3 This is a cross-sectional view showing a portion of the sewing object S involved in this embodiment. Figure 4 This is a top view showing the sewing object S involved in this embodiment. Figure 3 and Figure 4 The object to be sewn, S, is shown before the sewing process. In this embodiment, the object to be sewn, S, is the cover material used for a vehicle seat.

[0052] like Figure 3 As shown, the object S to be sewn has a surface material 4, a padding material 5, and a backing material 6. A hole 7 is provided in the surface material 4.

[0053] The surface material 4 is the seating surface that comes into contact with the passenger when the passenger is seated in the vehicle seat. The surface material 4 includes at least one of woven fabric, nonwoven fabric, and leather. The padding material 5 is elastic. The padding material 5 includes, for example, polyurethane resin. The backing material 6 includes at least one of woven fabric, nonwoven fabric, and leather.

[0054] like Figure 4 As shown, a plurality of holes 7 are provided in the surface material 4. The holes 7 are formed to be scattered throughout the sewn object S. The holes 7 are arranged in a regular pattern. That is, the sewn object S contains a plurality of reference patterns DPh. The reference patterns DPh are constituted by the regular arrangement of the plurality of holes 7. Figure 4 In the example, the reference pattern DPh consists of 17 holes 7.

[0055] like Figure 4 As shown, reference patterns DPh are arranged at intervals on surface material 4. The reference patterns DPh are arranged at equal intervals in both the X-axis and Y-axis directions. Reference patterns DPh at different positions in the Y-axis direction are arranged between adjacent reference patterns DPh in the X-axis direction. No holes 7 are formed between adjacent reference patterns DPh.

[0056] In the area between the reference patterns DPh, the target pattern RP for the stitch CH to be formed on the sewn object S is defined. Figure 4 In the object S to be sewn before sewing, the target pattern RP of the stitch CH is virtually shown. The sewing machine 1 forms the stitch CH along each target pattern RP according to the pre-set sewing pattern.

[0057] Furthermore, multiple identification objects UP (UP1, UP2, UP3, UP4, UP5, UP6, UP7) are arranged on the sewn object S. In this embodiment, the identification objects UP are arranged in a predetermined pattern among multiple holes 7 in the sewn object S. Specifically, the identification objects UP are part of a reference pattern DPh. Figure 4 In the example, the identified objects UP (UP1, UP2, UP3, UP4, UP5, UP6, UP7) are patterns (the arrangement of holes 7) that include the corner of one acute angle of the reference pattern DPh with a rhomboid shape. The identified objects UP are patterns that can be determined by template matching, which is an image processing method.

[0058] Reference Figure 5 The displacement that occurs on the surface of a sewn object S when a stitch CH is formed on the object S with thickness and elasticity is explained. Figure 5 This is a cross-sectional view showing an example of the sewing object S involved in this embodiment. Figure 5 The image shows a sewn object S after the sewing process has been performed. The sewn object S has thickness and elasticity. By forming a stitch CH on the thick, elastic sewn object S, as shown... Figure 5 As shown, the sewn object S is highly likely to shrink. If the sewn object S shrinks, its surface may shift. If the surface of the sewn object S shifts, the target position of the stitch CH defined on the surface of the sewn object S is highly likely to shift in the XY plane. If the target position of the stitch CH shifts in the XY plane, it will be difficult to form the stitch CH at the target position if the holding member 15 is moved to the position coordinates of the designed target pattern RP.

[0059] Therefore, in this embodiment, the displacement amount (a correction amount for correcting the displacement amount) is obtained by the image processing device 40. The holding member 15 moves accordingly to the displacement amount obtained by the image processing device 40, so that even if the surface of the sewn object S is displaced due to the shrinkage of the sewn object S caused by the formation of the stitch CH, the next stitch CH will be formed at the target position. Image processing device 40 (see reference) Figure 1 The object to be identified, UP, is detected based on the image obtained from the sewing object S, and the correction amount is obtained based on the displacement of the object to be identified, UP.

[0060] [Filming device]

[0061] Figure 6 This is a top view illustrating the mounting position of the imaging device 41 according to this embodiment.

[0062] Multiple camera devices 41 are arranged at intervals in a horizontal plane (XY plane) relative to the sewing machine body 10. The multiple camera devices 41 are positioned within the sewing area SA. The sewing area SA is the region in the XY plane where sewing can be performed via the head 11D by moving the object S to be sewn using the holding member 15. The head 11D (i.e., the sewing position Ps) is located at the center of the sewing area SA. The multiple camera devices 41 are arranged along the head 11D of the sewing machine body 10.

[0063] exist Figure 6 In this example, the sewing machine 1 has imaging units 50A and 50B. One imaging unit 50A and one 50B are located on each side of the head 11D in the X-axis direction, and each is configured adjacent to the head 11D. Multiple imaging devices 41 of the image processing device 40 can be mounted in the imaging units 50A and 50B. The imaging units 50A and 50B are positioned above the sewing area SA. Each of the imaging units 50A and 50B is fixed to the sewing machine 1.

[0064] The imaging unit 50A on the X-axis side has three mounting portions 51A, 51B, and 51C. Each mounting portion 51A, 51B, and 51C can mount one imaging device 41. Mounting portions 51A and 51B are arranged at a distance from each other in the X-axis direction. Mounting portions 51B and 51C are arranged at a distance from each other in the Y-axis direction.

[0065] The imaging unit 50B on the +X direction side has three mounting portions 51D, 51E, and 51F. Each mounting portion 51D, 51E, and 51F can mount one imaging device 41. Mounting portions 51D and 51E are arranged at a distance from each other in the X-axis direction. Mounting portions 51D and 51F are arranged at a distance from each other in the Y-axis direction.

[0066] Each mounting part 51A, 51B, 51C, 51D, 51E, and 51F is arranged in the XY direction inside the sewing area SA. Each mounting part 51A, 51B, 51C, 51D, 51E, and 51F can hold the shooting device 41 above the sewing area SA. Therefore, regardless of which mounting part the shooting device 41 is installed in, the sewing machine 1 can position the sewing object S towards the shooting position of the shooting device 41 via the holding member 15. Figure 6 In the example, the mounting portions 51A and 51B of the shooting unit 50A and the mounting portions 51D and 51E of the shooting unit 50B are arranged on a straight line in the X-axis direction relative to the sewing position Ps.

[0067] The sewing machine 1 has a total of six mounting positions (mounting parts 51A, 51B, 51C, 51D, 51E, and 51F) provided by the shooting units 50A and 50B. In this embodiment, the shooting device 41 can be installed at all six mounting positions, or it can be installed at a portion (two or more adjacent positions). The sewing machine 1 can select the mounting position of the shooting device 41 according to the shape of the sewing object S and the shape of the stitch CH formed on the sewing object S, and the mounting position of the shooting device 41 can be changed.

[0068] In addition, the imaging units 50A and 50B hold the multiple lighting devices 43A and 43B included in the image processing device 40. The imaging units 50A and 50B hold the lighting devices 43A and 43B in a fixed state above the sewing area SA. In the imaging unit 50A, one lighting device 43A is arranged on each of the outer sides and between the mounting portions 51A and 51B along the X-axis direction (a total of three). Each lighting device 43A extends linearly along the Y-axis direction. Each lighting device 43A illuminates the shooting field of view set below the mounting portions 51A, 51B, and 51C using incident illumination. In the imaging unit 50B, one lighting device 43B is arranged on each of the outer sides and between the mounting portions 51D and 51F along the Y-axis direction (a total of three). Each lighting device 43B extends linearly along the X-axis direction. Each lighting device 43B illuminates the shooting field of view set below the mounting parts 51D, 51E, and 51F with incident light.

[0069] As described above, the image processing apparatus 40 according to this embodiment has a plurality of shooting devices 41 arranged relative to the sewing machine body 10 (head 11D). The plurality of shooting devices 41 are arranged at intervals in the horizontal plane (XY plane). Figure 6 In the example, multiple shooting devices 41 can be configured along at least one of the length direction (X-axis direction) and width direction (Y-axis direction) of the sewn area SA.

[0070] Figure 7 This is a top view illustrating an example configuration of the imaging device 41 according to this embodiment. Multiple imaging devices 41 can, for example, be configured as follows: Figure 7 Configure as shown.

[0071] exist Figure 7In the example, two shooting devices 41 are respectively installed in the mounting portions 51A and 51B of the shooting unit 50A. The two shooting devices 41 of the shooting unit 50A are arranged along the length direction (X-axis direction) of the sewing area SA. Shooting devices 41 are respectively installed in the mounting portions 51D and 51F of the shooting unit 50B. The shooting devices 41 of the shooting unit 50B are arranged along the width direction (Y-axis direction) of the sewing area SA.

[0072] The imaging device 41 acquires an image of at least a portion of the sewn object S disposed within the imaging field of view. The imaging device 41 captures an image of at least a portion of the sewn object S held by the holding member 15 from above. Multiple imaging devices 41 are each configured to have an overlapping region JA. The overlapping region JA partially overlaps with the imaging fields of adjacent other imaging devices 41. Specifically, the imaging device 41 of the mounting part 51A has an imaging field of view 60A. The imaging device 41 of the mounting part 51B has an imaging field of view 60B. The imaging fields 60A and 60B have an overlapping region JA, which has a rectangular shape and extends in the Y-axis direction. An enlarged imaging region 61A is formed by the imaging fields 60A and 60B. The enlarged imaging region 61A is a region formed by merging the imaging fields of multiple imaging devices 41 sharing the overlapping region JA. A wide enlarged imaging region 61A extending in the X-axis direction is formed by the imaging fields 60A and 60B arranged in the X-axis direction.

[0073] Furthermore, the imaging device 41 of the mounting unit 51D has an imaging field of view 60D. The imaging device 41 of the mounting unit 51F has an imaging field of view 60F. The imaging fields of view 60D and 60F have an overlapping area JA, which has a rectangular shape and extends in the X-axis direction. An enlarged imaging area 61B is formed by the imaging fields of view 60D and 60F. By arranging the imaging fields of view 60D and 60F in the Y-axis direction, a wide enlarged imaging area 61B extending in the Y-axis direction is formed.

[0074] exist Figure 7 In the structure, the shooting field size of the shooting device 41, the size of the overlapping area JA (the interval of the mounting parts), and the number of shooting devices 41 are related to the target pattern RP of the sewing object S (refer to...). Figure 4The dimensions of the images are set accordingly. That is, the enlarged shooting areas 61A and 61B formed by the multiple shooting devices 41 are designed to include the entire target pattern RP of the sewing object S. The sewing object S with stitch CH extending along the X-axis can be positioned in the enlarged shooting area 61A of the shooting unit 50A and photographed, and the sewing object S with stitch CH extending along the Y-axis can be positioned in the enlarged shooting area 61B of the shooting unit 50B and photographed. Therefore, regardless of which direction the stitch CH extends in the sewing machine 1, the entire target pattern RP can be photographed at once without changing the installation position of the shooting device 41.

[0075] Furthermore, the positions of each imaging device 41 are fixed at any of the six mounting locations. The relative positions of the imaging device 41 and the sewing machine frame 11 are fixed. The relative positions of the optical axis of the imaging device 41 and the sewing machine needle 3 (sewing position Ps) in the XY plane are fixed. The relative position data representing the relative positions of the centers of the enlarged imaging areas 61A and 61B in the XY plane and the sewing machine needle 3 are known data that can be derived from the design data of the sewing machine 1.

[0076] The position of the image acquired by the imaging device 41 is defined in the camera coordinate system. By using a defined transformation formula or transformation matrix, the position of the image defined in the camera coordinate system is transformed into the position of the image defined in the sewing machine coordinate system.

[0077] In addition, Figure 7 In the example, a second imaging device 33 is installed in the head 11D. For example... Figure 6 As shown, in the head 11D, mounting portions 33A and 33B are respectively positioned on both sides of the X-axis, spaced apart from the sewing position Ps. The second imaging device 33 is mounted on either or both of the mounting portions 33A and 33B. Figure 7 In the mounting section 33A, a second imaging device 33 is provided. Compared with the imaging device 41, the second imaging device 33 has a smaller field of view and a higher resolution (number of pixels per unit area). Therefore, as described later, while multiple imaging devices 41 are used for high-speed recognition applications that simultaneously image multiple recognition objects UP, the second imaging device 33 can image one recognition object UP in a single shot and is used for precision recognition applications that require particularly precise positioning.

[0078] [Structures related to the control of the sewing machine]

[0079] Figure 8 This is a functional block diagram illustrating the sewing machine 1 according to this embodiment. (Using...) Figure 8 The structure related to the control of sewing machine 1 will be described.

[0080] (Control device)

[0081] The control device 30 outputs control signals to control the sewing machine 1. The control device 30 includes a computer system. The control device 30 has an input / output interface, a storage device including non-volatile memory and volatile memory, and an arithmetic processing unit including a processor. The non-volatile memory is, for example, ROM (Read Only Memory) or a storage device. The volatile memory is, for example, RAM (Random Access Memory). The processor is, for example, a CPU (Central Processing Unit). The control device 30 controls the sewing machine 1 according to a computer program stored in the storage device.

[0082] Actuators 16, 17, and 18, the operating device 20, and the image processing unit 42 are connected to the control device 30. Actuator 16 moves the sewing machine needle 3 in the Z-axis direction. Actuator 17 moves the holding member 15 in the XY plane. Actuator 18 moves the presser foot member 15A of the holding member 15 in the Z-axis direction.

[0083] Additionally, drive quantity sensor 31 and drive quantity sensor 32 are connected to control device 30. Drive quantity sensor 31 detects the drive quantity of actuator 16. Drive quantity sensor 32 detects the drive quantity of actuator 17.

[0084] The control device 30 controls the actuator 16 based on the detection data from the drive quantity sensor 31. The control device 30, for example, determines the operating timing of the actuator 16 based on the detection data from the drive quantity sensor 31.

[0085] The drive quantity sensor 32 includes an X-axis sensor 32X and a Y-axis sensor 32Y. The X-axis sensor 32X detects the drive quantity of the X-axis motor 17X of the actuator 17. The Y-axis sensor 32Y detects the drive quantity of the Y-axis motor 17Y of the actuator 17. The X-axis sensor 32X includes an encoder for detecting the rotational amount of the X-axis motor 17X. The Y-axis sensor 32Y includes an encoder for detecting the rotational amount of the Y-axis motor 17Y. The detection data from the drive quantity sensor 32 is output to the control device 30. The control device 30 controls the actuator 17 based on the detection data from the drive quantity sensor 32. The control device 30 performs feedback control on the actuator 17 based on the detection data from the drive quantity sensor 32 to move the holding member 15 to the target position.

[0086] The drive quantity sensor 32 functions as a position sensor to detect the position of the holding member 15 in the XY plane. The drive quantity of the actuator 17 and the movement quantity of the holding member 15 are in a one-to-one correspondence.

[0087] X-axis sensor 32X detects the rotation of X-axis motor 17X, thereby detecting the movement of holding member 15 in the X-axis direction from the origin of the sewing machine coordinate system. Y-axis sensor 32Y detects the rotation of Y-axis motor 17Y, thereby detecting the movement of holding member 15 in the Y-axis direction from the origin of the sewing machine coordinate system.

[0088] The control device 30 calculates the position of the holding member 15 in the XY plane based on the detection data from the drive quantity sensor 32. Based on the detection data from the drive quantity sensor 32, it detects the amount of movement of the holding member 15 from the origin in the XY plane. Based on the detected amount of movement of the holding member 15, the control device 30 calculates the position of the holding member 15 in the XY plane.

[0089] The control device 30 stores sewing data via a storage device. The sewing data is known data that can be derived from design data of the sewing object S, such as CAD (Computer Aided Design) data. The sewing data is referenced by the control device 30 during the sewing process. The sewing process refers to the process of forming stitch CH on the sewing object S. The sewing data package contains the target pattern RP (reference) of the stitch CH formed on the sewing object S. Figure 4 ), identify the position coordinates of the object UP and the movement conditions of the holding component 15.

[0090] The target pattern RP defines the target shape of the stitch CH formed on the sewing object S and the target position of the stitch CH in the sewing machine coordinate system.

[0091] The movement conditions of the holding member 15 include the movement trajectory of the holding member 15 as defined in the sewing machine coordinate system. The movement trajectory of the holding member 15 includes the movement trajectory of the holding member 15 in the XY plane. The movement conditions of the holding member 15 are determined based on the target pattern RP.

[0092] Therefore, as Figure 4 As shown, in the case of performing multiple sewing processes to form multiple stitches CH, a first sewing process is performed, followed by a second sewing process. The first sewing process is the process of forming a first stitch CH on the sewing object S based on a first target pattern RP1. The second sewing process is the process of forming a second stitch CH on the sewing object S based on a second target pattern RP2. Then, corresponding to the number N of stitches CH that should be formed, a third to an Nth sewing process are performed sequentially. Figure 4 An example with N=10 is shown, specifying the first target pattern RP1 to the tenth target pattern RP10.

[0093] The control device 30 is connected to the image processing unit 42 via an input / output interface. Based on the processing results of the image processing unit 42, the control device 30 outputs control commands to control the actuator 17 that moves the holding member 15.

[0094] (Image Processing Department)

[0095] The image processing unit 42 includes a computer system. The image processing unit 42 has a processor 44, a storage device 45 including non-volatile memory and volatile memory, and an input / output unit 46 including an input / output interface. The processor 44 is, for example, a CPU (Central Processing Unit). The non-volatile memory is, for example, ROM (Read Only Memory) or storage. The volatile memory is, for example, RAM (Random Access Memory). The image processing unit 42 performs image processing as follows: according to a computer program stored in the storage device 45, it takes pictures of the sewing object S using multiple imaging devices 41, and detects the identification object UP based on the images.

[0096] The image processing unit 42 is connected to multiple imaging devices 41 and multiple lighting devices 43A, 43B via the input / output unit 46. Figure 8 In this unit, the input / output unit 46 is connected to each of the shooting devices 41 and the lighting device 43A of the shooting unit 50A via the relay device 47. Similarly, the input / output unit 46 is connected to each of the shooting devices 41 and the lighting device 43B of the shooting unit 50B via the relay device 47. The relay device 47 is, for example, a hub device used to connect multiple devices, such as a USB (Universal Serial Bus) hub.

[0097] The image processing unit 42 includes an image acquisition unit 44A, a compositing unit 44B, a detection unit 44C, and a correction calculation unit 44D. The image acquisition unit 44A, compositing unit 44B, detection unit 44C, and correction calculation unit 44D are functional blocks implemented by a computer program executed by the processor 44. Alternatively, the image acquisition unit 44A, compositing unit 44B, detection unit 44C, and correction calculation unit 44D can be configured independently using a dedicated processor.

[0098] The image acquisition unit 44A acquires images of the sewn object S from multiple imaging devices 41. The imaging devices 41 capture images of the sewn object S and output the obtained images to the image processing unit 42. If the sewn object S is positioned at a predetermined shooting position by the control device 30, the image acquisition unit 44A controls the operation of the imaging devices 41 corresponding to the shooting position to acquire images of the sewn object S.

[0099] exist Figure 7 In the imaging unit 50A, a shooting position Pf is set at the center of the enlarged shooting areas 61A and 61B. Shooting is performed with the holding member 15 (the sewing object S) positioned at any shooting position Pf within the enlarged shooting areas 61A and 61B. For example, in the imaging unit 50A, if the sewing object S is positioned at shooting position Pf in the enlarged shooting area 61A, the image acquisition unit 44A illuminates the lighting device 43A, and the shooting devices 41 of the mounting unit 51A and 51B substantially simultaneously perform shooting. As a result, an image corresponding to the shooting field of view 60A and an image corresponding to the shooting field of view 60B are acquired. In the imaging unit 50B, if the sewing object S is positioned at shooting position Pf in the enlarged shooting area 61B, the image acquisition unit 44A illuminates the lighting device 43B, and the shooting devices 41 of the mounting unit 51D and 51F substantially simultaneously perform shooting. Shooting can also be performed sequentially.

[0100] The compositing processing unit 44B performs image compositing processing, which combines images from multiple imaging devices 41. The compositing processing unit 44B combines multiple images constituting the expanded imaging area (61A or 61B) based on the image portions of their overlapping areas JA, generating a single composite image 75 (see reference). Figure 13 Therefore, the composite image 75 is an image capturing a range equivalent to that of the enlarged capturing area 61A or the enlarged capturing area 61B. The enlarged capturing areas 61A and 61B have a size that includes multiple identification objects UP of the sewn object S. Therefore, the composite image 75 is an image capturing multiple identification objects UP. Preferably, the composite image 75 includes all identification objects UP (UP1 to UP7) detected when forming one stitch CH.

[0101] The detection processing unit 44C detects multiple identification objects UP based on the composite image 75 formed by the synthesis processing unit 44B. The detection processing unit 44C detects the identification objects UP in the composite image 75 using a pattern matching method, which is a type of image processing method. Here, the operator pre-stores a reference image (template) used to identify the identification objects UP of the sewing object S in the storage device 45. The detection processing unit 44C retrieves the reference image recorded in the storage device 45 and obtains the position coordinates of the identification objects UP in the composite image 75 by matching it with the reference image. The detection processing unit 44C sequentially detects the multiple identification objects UP included in the composite image 75 and obtains the position coordinates of each of the multiple identification objects UP.

[0102] The correction calculation unit 44D calculates correction data for correcting the displacement of the surface of the sewn object S based on the detection results obtained by the detection processing unit 44C. The target pattern RP of the sewing data is generated under the premise that the sewn object S has not shrunk and is stored in the control device 30. If the sewing process is performed and the sewn object S shrinks, causing displacement of the surface of the sewn object S, the correction calculation unit 44D calculates the displacement of the identification object UP detected by the detection processing unit 44C and generates correction data for correcting the sewing data.

[0103] Specifically, the correction calculation unit 44D generates correction data so that the relative position in the sewing machine coordinate system between the target pattern RP and the identification object UP before shrinkage of the sewing object S is consistent with the relative position in the sewing machine coordinate system between the stitch CH actually formed in the sewing object S through the sewing process and the identification object UP after shrinkage of the sewing object S.

[0104] The image processing unit 42 sends the correction data calculated by the correction amount calculation unit 44D to the control device 30. The control device 30 performs sewing processing based on the correction data.

[0105] [Sewing action]

[0106] Figure 9 This is a flowchart illustrating the sewing operation of the sewing machine 1 according to this embodiment. The sewing operation of the sewing machine 1 includes image capture processing S1, image stitching processing S2, object detection processing S3, correction data calculation processing S4, sewing processing S5, and end determination processing S6. Furthermore, the image processing method according to this embodiment is implemented as part of the sewing operation of the sewing machine 1. The image processing method according to this embodiment includes image stitching processing S2 and object detection processing S3.

[0107] (Photo processing)

[0108] The imaging process S1 involves photographing the sewn object S using multiple imaging devices 41. The operator or the control device 30 pre-sets whether to acquire an image of an enlarged imaging area 61A extending in the X-axis direction using imaging unit 50A, or an image of an enlarged imaging area 61B extending in the Y-axis direction using imaging unit 50B. The control device 30 moves the holding member 15 that holds the sewn object S, positioning it at a pre-set imaging position. If the holding member 15 is positioned at the corresponding imaging position Pf, the image processing unit 42 performs imaging using the pre-set multiple imaging devices 41.

[0109] Below, as an example, we will discuss the process of... Figure 7The following describes the case where each of the imaging devices 41 of the imaging unit 50A acquires an image of the expanded imaging area 61A. In this case, the image acquisition unit 44A acquires an image corresponding to the imaging field of view 60A and an image corresponding to the imaging field of view 60B via the imaging devices 41 of the mounting unit 51A and the mounting unit 51B.

[0110] (Image stitching processing)

[0111] Image stitching process S2 is a process of stitching together images of the sewn object S captured by multiple shooting devices 41.

[0112] Figure 10 This is a flowchart illustrating the image joining process S2 involved in this embodiment. Figure 11 This is a schematic diagram illustrating the image positioning process S2A in the image joining process S2. Figure 12 This is a schematic diagram illustrating the mask region setting process S2B in the image joining process S2. Figure 13 This is a schematic diagram illustrating the synthesis process S2C in the image joining process S2.

[0113] like Figure 11 As shown, the image obtained by the imaging device 41 via the mounting unit 51A is designated as the first image 71. The image obtained by the imaging device 41 via the mounting unit 51B is designated as the second image 72. In this case, the second image 72 includes a region relative to the first image 71 on the +X direction side. The first image 71 corresponds to the imaging field of view 60A, and the second image 72 corresponds to the imaging field of view 60B. The first image 71 and the second image 72 share an overlapping region JA.

[0114] The overlapping region JA of the first image 71 and the second image 72 contains image features GE that are common to at least a portion of the identified object UP. That is, as shown in the image JA. Figure 4 As shown, when the identification object UP is an arrangement of multiple holes 7 existing in a sewn object S in a prescribed pattern, the image element GE is an image of each hole 7. The holes 7 are structural elements constituting the identification object UP or the reference pattern DPh. Figures 11 to 13 In order to simplify the explanation, the arrangement of holes 7 is shown as an example of holes 7 arranged in an array along the longitudinal and transverse directions.

[0115] When detecting the object UP through pattern matching, it is crucial to ensure that the image elements GE constituting the hole 7 of the object UP do not suffer from image information loss or errors. Therefore, in Figure 10 In the image joining process S2 shown, when joining the first image 71 and the second image 72, the first image 71 and the second image 72 are synthesized so that information of the image elements GE constituting the recognition object UP is saved.

[0116] like Figure 10 As shown, the image stitching process S2 includes image positioning process S2A, mask region setting process S2B, and compositing process S2C. The compositing process unit 44B may also binarize each image obtained by the shooting process S1 before the image stitching process S2.

[0117] <Image Positioning Processing>

[0118] Image localization processing S2A is a process that locates multiple images obtained through shooting processing S1 based on image elements GE in the overlapping region JA.

[0119] like Figure 11 As shown, the compositing processing unit 44B detects the image elements GE contained in the overlapping region JA of the first image 71 and the overlapping region JA of the second image 72, respectively. The position coordinates of the shooting center of the shooting device 41 mounted on the mounting units 51A and 51B in the sewing machine coordinate system and the sizes of the shooting fields 60A and 60B are known; therefore, the range of the overlapping region JA in each image is known. The compositing processing unit 44B selects image elements GE containing 3 or more points within the overlapping region JA. Figure 11 In the example, the synthesis processing unit 44B divides the overlapping region JA into four parts along the longitudinal and transverse directions, and selects a total of four image elements GE from each segmented region, which are then used as the image elements of interest GE1.

[0120] The compositing processing unit 44B deforms the second image 72 so that the same image element GE in the overlapping region JA is consistent with each other. That is, the compositing processing unit 44B deforms the second image 72 so that the corresponding four image elements of interest GE1 in the second image 72 are consistent with the four image elements of interest GE1 in the first image 71. The deformation of the second image 72 is performed, for example, by affine transformation. The deformation may include rotation in the X-axis coordinate, Y-axis coordinate, θZ direction, scale adjustment (enlargement and reduction), and distortion correction (shearing deformation). The positions and shapes of the four image elements of interest GE1 in the first image 71 and the deformed second image 72 (hereinafter referred to as the second image 72A) are consistent. Therefore, the images in the overlapping region JA of the first image 71 and the deformed second image 72A are substantially consistent. The image positioning processing S2A eliminates the differences in image distortion caused by assembly errors in the setting positions of the captured first image 71 and second image 72A, aberrations of the optical system, etc. As described later, the composite processing unit 44B combines the deformed second image 72A with the first image 71.

[0121] <Mask Area Setting Processing>

[0122] Mask area setting processing S2B (refer to) Figure 10 The following process is used: for the first image 71 and the second image 72A after alignment by the image positioning process S2A, the area to be joined is set by the compositing process S2C. In this embodiment, the compositing process 44B sets the joining position in the overlapping area JA in such a way that the entire image element GE detected in the overlapping area JA is included in either the first image 71 or the second image 72A.

[0123] like Figure 12 As shown, the compositing processing unit 44B extracts the image element GE that is connected to the end edge EL on the side of the second image 72A in the overlapping region JA, and sets a first mask region 73 in the first image 71 to exclude the extracted image element GE.

[0124] For example, the compositing processing unit 44B attaches a straight line SL to the edge EL and extracts the contour with the largest area within the overlapping region JA. Attaching the straight line SL to the edge EL sets a predetermined pixel value for the pixel column constituting the edge EL in the overlapping region JA. If there is an image feature GE connected to the edge EL, this image feature GE is connected to the straight line SL. As a result, if the contour with the largest area is extracted within the overlapping region JA, only the image feature GE connected to the straight line SL (i.e., connected to the edge EL) is extracted. The straight line SL and the contour of the image feature GE connected by the straight line SL are referred to as the contour region 76.

[0125] The compositing processing unit 44B sets the region obtained after excluding the extracted contour region 76 from the first image 71 as the first mask region 73. The first mask region 73 is the region within the first image 71 used as pixels constituting the composite image 75. The contour region 76 of the first image 71 is excluded from the first mask region 73 and therefore is not used in the compositing process.

[0126] Next, the compositing processing unit 44B sets a second mask region 74 in the second image 72A. The second mask region 74 includes the entirety of the image element GE that is connected to the end edge EL of the overlapping region JA and is adjacent to the first mask region 73.

[0127] The compositing processing unit 44B sets a portion of the extracted contour region 76 from the overlapping region JA and a region inherent to the second image 72A that is not included in the overlapping region JA as a second mask region 74. Since the portion of the contour region 76 is excluded from the first mask region 73, the second mask region 74 is adjacent to the first mask region 73 without gap in the portion of the contour region 76. The second mask region 74 is the region in the second image 72A used as pixels constituting the composite image 75.

[0128] Therefore, the first mask region 73 includes the region inherent to the first image 71, and all remaining regions in the overlapping region JA excluding the contour region 76. The region inherent to the first image 71 is contained only in the first image 71 and is not included in the second image 72A. The second mask region 74 includes the region inherent to the second image 72A, and the contour region 76 in the overlapping region JA. The region inherent to the second image 72A is contained only in the second image 72A and is not included in the first image 71.

[0129] Here, the image element GE connected to the end edge EL will be explained. The end edge EL of the second image 72A side of the overlapping region JA is the end edge of the first image 71 in the +X direction. Therefore, the image element GE connected to the end edge EL of the overlapping region JA is connected to the end edge of the first image 71. In the image element GE connected to the end edge of the first image 71, there may be a part that is not captured in the first image 71 (the part P2 on the +X side compared to the end edge EL). That is, it is possible that only the part P1 on the -X side compared to the end edge EL in the aperture 7 is reflected in the first image 71.

[0130] Assuming the position of the edge EL is set at the boundary between the first mask region 73 and the second mask region 74, the portion P1 on the -X side is obtained from the first image 71, and the portion P2 on the +X side is obtained from the second image 72A. Therefore, when the position of the edge EL is set at the boundary between the first mask region 73 and the second mask region 74, information loss or errors may occur. On the other hand, the edge EL on the second image 72A side of the overlapping region JA is located at the center of the second image 72A. Therefore, the entirety of the image element GE (parts P1 and P2) connected to the edge EL of the overlapping region JA is reliably captured in the second image 72A. Therefore, by including a portion of the contour region 76 in the second mask region 74 of the second image 72A, it is possible to generate a composite image 75 that completely preserves the information of the image element GE of the hole 7 present at the position spanning the edge EL.

[0131] Furthermore, it is possible that there is no image element GE connected to the edge EL of the overlapping region JA. In this case, the synthesis processing unit 44B can set the first mask region 73 and the second mask region 74 with the edge EL as the boundary. That is, the first mask region 73 can be set to the entire first image 71 including the overlapping region JA. The second mask region 74 can be set to the area inherent in the second image 72A that excludes the overlapping region JA, with the edge EL as the boundary. When the object to be identified UP is an arrangement of multiple image elements GE (i.e., holes 7), the area in the image other than the image elements GE is equivalent to the "background" outside the object to be processed in the image recognition process. Therefore, when the position of the edge EL is equivalent to the background, even if image information loss or error is assumed, it will not affect the detection accuracy of the object to be identified UP.

[0132] <Synthetic Processing>

[0133] Synthesis treatment S2C (refer to) Figure 10 The process involves using the boundary between the first mask region 73 of the first image 71 and the second mask region 74 of the second image 72A as the joining point, and then joining the first image 71 and the second image 72A. Figure 13 As shown, the compositing processing unit 44B uses each pixel belonging to the first mask region 73 in the first image 71 and each pixel belonging to the second mask region 74 in the second image 72A to generate a composite image 75 formed by joining the first image 71 and the second image 72A. The generated composite image 75 is an image of a size equivalent to the enlarged shooting area 61A. The region in the composite image 75 corresponding to the first mask region 73 is an image region obtained from the first image 71. The region in the composite image 75 corresponding to the second mask region 74 is an image region obtained from the second image 72. Regarding each image element GE included in the composite image 75, since the entire image element GE can be obtained from either the first image 71 or the second image 72A, no loss or error of image information occurs regarding the image element GE. The image joining process S2 is performed in the manner described above.

[0134] (Object detection and processing)

[0135] Next, Figure 9 The object detection process S3 is as follows: the object UP is detected based on the composite image 75 generated by the image joining process S2, and the position coordinates of the detected object UP are obtained. Figure 14 This is a schematic diagram illustrating the object detection process S3.

[0136] like Figure 14As shown, the detection processing unit 44C extracts an image of the recognition region AR from the composite image 75, which is used for the detection processing of the recognition object UP. The recognition region AR is set to take into account the range of the expected positional deviation of the recognition object UP associated with the stitch formation relative to the design position coordinates of the recognition object UP set in the sewing data. Figure 14 The diagram schematically illustrates the setting of the recognition region AR relative to a target pattern RP of arbitrary shape. The detection processing unit 44C sets a recognition region AR for each of the multiple recognition objects UP that should be detected in a single line formation, and obtains an extracted image 77 by cropping an image of the set recognition region AR. For example, in... Figure 4 In the example, recognition regions AR are set for the seven recognition objects UP1 to UP7. Furthermore, Figure 10 , Figure 13 The synthesis process S2C (i.e., the generation of the synthesized image 75) can also be implemented only if the boundary (edge ​​EL) of the overlapping region JA is contained within the recognition region AR.

[0137] The detection processing unit 44C queues the acquired extracted images 77 (setting a processing order) and performs image recognition processing sequentially using a pattern matching method. The detection processing unit 44C reads reference images from the storage device 45 and obtains the position coordinates of the object UP to be recognized in the extracted images 77 by matching them with the reference images. Figure 14 As shown, the processor 44 that implements the functions of the image processing unit 42 may include multiple processor cores CR. A processor core CR is a unit of arithmetic processing configuration possessed by the processor 44. Each processor core CR can perform arithmetic processing according to a computer program. When the processor 44 includes multiple processor cores CR, the image processing unit 42 performs image recognition processing on each queued extracted image 77 in parallel using the multiple processor cores CR. If the number of available processor cores CR is K, the image recognition processing of K extracted images 77 can be processed simultaneously in parallel, thus shortening the processing time.

[0138] Through object detection processing S3, the detection processing unit 44C obtains each object UP (UP1 to UP7, refer to) that should be detected in one line formation. Figure 4 The position coordinates in the sewing machine coordinate system.

[0139] (Correction data calculation and processing)

[0140] The correction data calculation process S4 is based on the detection results obtained through the object detection process S3, and calculates correction data to correct the displacement of the surface of the sewing object S. The correction amount calculation unit 44D calculates the displacement of the object UP from its initial position to its current position based on the initial and current positions of the object UP involved in this sewing process. The initial position of the object UP is the position coordinate of the object UP as defined in the sewing data. The current position of the object UP is the position coordinate of the object UP detected by the object detection process S3. Based on the calculated displacement, the correction amount calculation unit 44D applies correction data to each correction point CP (reference point) on the target pattern RP. Figure 14 The correction position coordinates are calculated, and the correction data of the target pattern RP passing through the correction point CP is calculated so that the stitch CH is formed at the target position in the sewing machine coordinate system. Furthermore, in the initial sewing process (first sewing process), no displacement of the surface of the sewing object S associated with stitch formation occurs, and therefore no displacement of the identification object UP occurs. Therefore, in the initial sewing process (first sewing process), the target pattern RP in the initial state specified by the sewing data is used directly, and thus correction data is not calculated.

[0141] (Sewing process)

[0142] Sewing process S5 is a process of forming stitch CH based on the target pattern RP. Sewing process S5 is implemented by controlling various parts of the sewing machine 1 via the control device 30. The sewing process includes the first sewing process to the Nth sewing process (in... Figure 4 (N=10). The first sewing process is performed based on the target pattern RP in the initial state specified by the sewing data. The second to Nth sewing processes are performed based on the correction data (corrected target pattern RP) calculated by the correction data calculation process S4. In the first sewing process, the control device 30 outputs a control command to the actuator 17 to form the stitch CH according to the target pattern RP in the initial state specified by the sewing data. In the second sewing process and thereafter, the control device 30 obtains the correction data from the image processing unit 42 and outputs a control command to the actuator 17 to form the stitch CH according to the target pattern RP of the obtained correction data.

[0143] The end determination process S6 is a process that determines whether the sewing process of the sewing object S has ended. The control device 30 determines whether the sewing process of the sewing object S has ended based on the sewing data. If the sewing process has ended from the first to the (N-1)th sewing process, the control device 30 determines in the end determination process S6 that the sewing process has not ended. If the sewing process has ended at the Nth time, the control device 30 determines in the end determination process S6 that the sewing process has ended.

[0144] As described above, the following series of processes are repeated in the first to Nth sewing processes: at the end of each sewing process, an image capture process S1, an image stitching process S2, an object detection process S3, and a correction data calculation process S4 are performed. Correction data for the next sewing process is calculated, and the next sewing process is performed based on the calculated correction data. In this embodiment, multiple objects UP detected in a single stitch CH sewing process can be detected based on the composite image 75 obtained in a single capture by multiple capturing devices 41. As a result, the processing time required for the image capture process S1, image stitching process S2, object detection process S3, and correction data calculation process S4 performed at each sewing process is reduced.

[0145] [Effect]

[0146] As described above, according to this embodiment, images of the sewing object S captured by multiple imaging devices 41 are joined together, and multiple identification objects UP of the sewing object S are detected based on the combined image 75. Therefore, compared to the case where multiple identification objects UP are captured and detected one by one, the number of times the sewing object S is moved and the number of times it is captured can be reduced, thus shortening the time required for detecting the identification objects UP accompanying the sewing process. As a result, by shortening the detection time of the identification objects UP performed in each sewing process, the overall processing time of the sewing operation can be shortened. Furthermore, by setting an overlapping region JA in the field of view of the multiple imaging devices 41, the same image element GE in the overlapping region JA is consistent with each other, thus achieving high positional accuracy even when images are joined. In addition, by setting the joining position such that the entire image element GE detected in the overlapping region JA is included in either the first image 71 or the second image 72, the loss or error of information of the image element GE constituting the identification object UP can be prevented. As a result, the detection accuracy of the identification object UP can be improved. Furthermore, image elements GE that connect to the edge EL on the side of the second image 72 in the overlapping region JA are extracted. A second mask region 74 is set in the second image 72, which includes the entire extracted image element GE. Therefore, image elements GE that may be lost during joining can be easily extracted, and joining with the entire image element GE preserved can be achieved. Thus, the computational load associated with the joining process can be reduced, and the processing time can be shortened.

[0147] Furthermore, in this embodiment, the image element GE is formed by holes 7 scattered throughout the sewn object S. Even when the identification object UP is arranged in a pattern of multiple holes 7 in the sewn object S, it is possible to detect the identification object UP with high accuracy and in a short time. In addition, the multiple imaging devices 41 are arranged along at least one of the length direction (X-axis direction) and width direction (Y-axis direction) of the sewn area SA, so it is possible to generate a composite image 75 that can detect multiple identification objects UP simultaneously in accordance with the direction of the stitch CH. In addition, the multiple illumination devices 43 (43A, 43B) are respectively arranged on both sides of the arrangement of the multiple imaging devices 41 and between the arrangement of the multiple imaging devices 41, so even when generating a composite image 75 with a wide range, it is possible to achieve uniformity of brightness (pixel value) in the image, thereby improving detection accuracy.

[0148] [Other Implementation Methods]

[0149] In the above embodiment, an example with four imaging devices 41 is shown, but the number of imaging devices 41 can also be two, three, or five or more. The imaging devices 41 can be installed at six or more locations. Multiple imaging devices 41 can also be arranged in directions other than the length direction (X-axis direction) and width direction (Y-axis direction) of the sewing area SA. The number of lighting devices 43A and 43B is not limited; there can be one or two or more. An example is shown where the image element GE is formed in the hole 7 of the sewing object S, but the image element GE is not particularly limited and can also be an image element other than the hole 7.

[0150] In the above embodiment, the second image 72 is deformed to match the first image 71, but the first image 71 can also be deformed to match the second image 72. Alternatively, either the first image 71 or the second image 72 can be used as the two images to be joined. Alternatively, the second image 72 can be aligned without deforming it during image positioning processing S2A.

[0151] In the above embodiment, each pixel constituting the composite image 75 is taken from either the first image 71 or the second image 72. However, for each pixel in the composite image 75 corresponding to the overlapping region JA, both the first image 71 and the second image 72 may be used. For example, for the overlapping region JA, both the first image 71 and the second image 72 may be used for averaging. Alternatively, for the overlapping region JA, the image obtained by performing a logical AND, OR, or logical OR operation on the two images after binarizing the first image 71 and the second image 72 may be used.

[0152] The present invention includes the following methods. (1)

[0154] An image processing apparatus comprising: Multiple shooting devices are arranged at intervals in a horizontal plane relative to the sewing machine body, which has a sewing head and a holding member for holding and moving the sewing object; and The image processing unit detects identifying objects on the sewn object based on images captured by the plurality of imaging devices. The image processing unit has: A compositing processing unit that combines the images from the plurality of imaging devices; and The detection processing unit detects multiple of the identified objects based on the composite image obtained by combining the images from the synthesis processing unit. (2)

[0156] In the image processing apparatus described in (1), Each of the plurality of shooting devices is configured to have an overlapping region in which its field of view partially overlaps with the field of view of the other adjacent shooting devices. The synthesis processing unit, In the overlapping region of the first and second images being joined, image features common to at least a portion of the identified object are detected. The second image is deformed so that the same image elements in the overlapping area are consistent with each other, and the deformed second image is joined with the first image. (3)

[0158] In the image processing apparatus described in (2), The compositing processing unit sets the joining position in the overlapping region in such a way that the entire image element detected in the overlapping region is included in either the first image or the second image. (4)

[0160] In the image processing apparatus described in (2) or (3), The synthesis processing unit, The image features in the overlapping region that are connected to the edge of the second image are extracted. A first mask region is set in the first image to exclude the extracted image features. A second mask region is defined in the second image. This second mask region includes the entirety of the image element connected to the edge and is adjacent to the first mask region. The first image and the second image are joined by using the boundary between the first mask region of the first image and the second mask region of the second image as the joining position. (5)

[0162] In any of the image processing apparatuses described in (2) to (4), The image elements are holes formed by distributing them throughout the sewn object. The identification object is the arrangement of a plurality of holes in the sewn object in a prescribed pattern. (6)

[0164] In any of the image processing apparatuses described in (1) to (5), The plurality of shooting devices are arranged along at least one of the length and width directions of the sewing area, which is an area in which the head can be used to sew the object being sewn by moving the object being sewn through the holding member. (7)

[0166] In any of the image processing apparatuses described in (1) to (4), It also has multiple lighting devices, which are respectively arranged on both sides of the arrangement of the multiple shooting devices and between the arrangement of the multiple shooting devices. (8)

[0168] A sewing machine having: The sewing machine body has a head for sewing and a holding part for holding the object being sewn and moving the object being sewn. The image processing apparatus described in any one of (1) to (7); and A control device that controls the sewing machine body based on the processing results of the image processing device. (9)

[0170] An image processing method comprising the following steps: Images of the sewn object captured by multiple imaging devices, spaced apart in a predetermined direction in a horizontal plane relative to the sewing machine body, are combined. The sewing machine body has a sewing head and a holding member for holding and moving the sewn object. Based on the composite image formed by combining the images, multiple identifying objects of the sewn object are detected.

[0171] This application is based on Japanese Patent Application No. 2023-108671, filed on June 30, 2023, the contents of which are incorporated herein by reference.

Claims

1. An image processing apparatus comprising: Multiple shooting devices are arranged at intervals in a horizontal plane relative to the sewing machine body, which has a sewing head and a holding member for holding and moving the sewing object. as well as The image processing unit detects identifying objects on the sewn object based on images captured by the plurality of imaging devices. The image processing unit has: The compositing processing unit combines the images from the plurality of imaging devices; as well as The detection processing unit detects multiple of the identified objects based on the composite image obtained by combining the images from the synthesis processing unit.

2. The image processing apparatus according to claim 1, wherein, Each of the plurality of shooting devices is configured to have an overlapping region in which its field of view partially overlaps with the field of view of the other adjacent shooting devices. The synthesis processing unit, In the overlapping region of the first and second images being joined, image features common to at least a portion of the identified object are detected. The second image is deformed so that the same image elements in the overlapping area are consistent with each other, and the deformed second image is joined with the first image.

3. The image processing apparatus according to claim 2, wherein, The compositing processing unit sets the joining position in the overlapping region in such a way that the entire image element detected in the overlapping region is included in either the first image or the second image.

4. The image processing apparatus according to claim 3, wherein, The synthesis processing unit, The image features in the overlapping region that are connected to the edge of the second image are extracted. A first mask region is set in the first image to exclude the extracted image features. A second mask region is defined in the second image. This second mask region includes the entirety of the image element connected to the edge and is adjacent to the first mask region. The first image and the second image are joined by using the boundary between the first mask region of the first image and the second mask region of the second image as the joining position.

5. The image processing apparatus according to any one of claims 2 to 4, wherein, The image elements are holes formed by distributing them throughout the sewn object. The identification object is the arrangement of a plurality of holes in the sewn object in a prescribed pattern.

6. The image processing apparatus according to any one of claims 1 to 4, wherein, The plurality of shooting devices are arranged along at least one of the length and width directions of the sewing area, which is an area in which the head can be used to sew the object being sewn by moving the object being sewn through the holding member.

7. The image processing apparatus according to any one of claims 1 to 4, wherein, It also has multiple lighting devices, which are respectively arranged on both sides of the arrangement of the multiple shooting devices and between the arrangement of the multiple shooting devices.

8. A sewing machine, comprising: The sewing machine body has a head for sewing and a holding part for holding the object being sewn and moving the object being sewn. The image processing apparatus as described in any one of claims 1 to 4; and A control device that controls the sewing machine body based on the processing results of the image processing device.

9. An image processing method comprising the following steps: Images of the sewn object captured by multiple imaging devices, spaced apart in a predetermined direction in a horizontal plane relative to the sewing machine body, are combined. The sewing machine body has a sewing head and a holding member for holding and moving the sewn object. Based on the composite image formed by combining the images, multiple identifying objects of the sewn object are detected.

Images