Processing method of sheet material, processing system, and program

Abstract

To provide a processing method capable of determining an appropriate grip position for exposing a feature amount of a sheet material in a pre-step of work, so that work can be finally performed with an angle exposed even in a state where a large object is casually placed.SOLUTION: A processing method X for performing, by a computer, processing of determining an appropriate grip position in a sheet material Z in any state, includes: an image acquisition step S1 of imaging the sheet material Z to acquire image data; a detection step S2 of detecting a feature amount F included in the image data; and a position determination step S3 of determining the grip position by using the feature amount F.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a processing method for recognizing the form of a sheet material placed haphazardly and detecting an appropriate part to be gripped at a stage before automatically folding a sheet material such as a sheet, and a processing system for executing the processing.

Background Art

[0002] In the linen supply industry and accommodation facilities such as hotels, a large amount of sheet materials are handled in sheet replacement, washing, and the like. In particular, a considerable amount of labor is required to neatly fold a sheet material having a size equal to or larger than that of a human body, such as a sheet or a bath towel. Therefore, it is preferable to automate the sheet material folding operation, and the applicant of the present application has developed a folding device for that purpose.

[0003] On the other hand, in the process of feeding a sheet material such as a sheet into a folding device, it is necessary to align the form and orientation of the sheet material to some extent. For this operation, since the sheet material was opened by hand and the type of the sheet material was discriminated and dealt with, automation of this process has been demanded. In this regard, devices that utilize characteristic elements such as the corners or patterns of the sheet material have been developed as in Patent Document 1 or 2.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] Patent Document 1 discloses a system that targets small sheet materials such as towels and recognizes and grips their four corners. However, for large objects such as sheets and bath towels, it is difficult to recognize the four corners from a state where they are placed haphazardly rolled up or the like. Therefore, regardless of the size of the sheet material, it is necessary to automate the process of reliably exposing the corners and mechanically gripping them.

[0006] Also, Patent Document 2 discloses a system that determines the pattern of laundry by image recognition. However, in order to recognize the pattern, it is necessary to spread out and place the object, and having only this part involve human hands was rather inefficient.

[0007] The present invention has been made in view of the above actual situation, and provides a processing method capable of determining an appropriate gripping position for exposing the feature amount of the sheet material in the previous stage so that the work can be finally carried out with the corners exposed even when a large object is placed haphazardly.

Means for Solving the Problems

[0008] The invention of the present application for solving the above problems is a processing method for a computer to execute a process of determining an appropriate gripping position in a sheet material in any state, including an image acquisition step of imaging the sheet material to acquire image data, a detection step of detecting a feature amount included in the image data, and a positioning step of determining the gripping position using the feature amount.

[0009] According to the present invention, it is possible to grip a large sheet material at an appropriate gripping position without involving human hands and deliver it to the next step for exposing the corners of the sheet material.

[0010] In a preferred form of the present invention, the detection step determines segments of edge elements included in the sheet material using the feature amount, and the positioning step determines the gripping position based on the segments.

[0011] With such a configuration, it is possible to grip near the edge of the sheet material using image recognition.

[0012] In a preferred form of the present invention, the edge element is provided along the edge of the sheet material, the detection step determines the continuity of a plurality of the segments, groups the plurality of segments determined to have the continuity, and the positioning step determines the gripping position using the elements of the group.

[0013] With such a configuration, even when the edge element is visually recognized in a curved shape, it can be recognized with high accuracy.

[0014] In a preferred form of the present invention, the sheet material has the edge element distinguishable for each edge, the detection step discriminates and groups each edge element from the feature amount, and the positioning step determines a point that meets a predetermined condition from the feature amount corresponding to each of the groups as the gripping position.

[0015] With such a configuration, it is possible to grip as close as possible to the long side of the sheet material and easily lift the sheet material.

[0016] In a preferred form of the present invention, when the positioning step determines a plurality of gripping positions, a set of points that are separated from each other by a predetermined linear distance among the segments determined to have the continuity is determined as the gripping position.

[0017] With such a configuration, even when the sheet material is gripped at a plurality of positions, it can be gripped and lifted at an appropriate gripping position.

[0018] In a preferred form of the present invention, when the positioning step determines a plurality of gripping positions, a set of points that are separated from each other by a predetermined surface distance among the segments determined to have the continuity is determined as the gripping position.

[0019] By adopting such a configuration, even when gripping the sheet material at multiple positions, it is possible to grip and lift the sheet material at a more appropriate gripping position according to the shape of the sheet material.

[0020] The invention of the present application for solving the above problems is a program that causes a computer to function as an image acquisition unit that captures an image of a sheet material to acquire image data, a detection unit that detects the feature amount included in the image data, and a positioning unit that determines a gripping position of the sheet material using the feature amount, and determines an appropriate gripping position in the sheet material in an arbitrary state.

[0021] By adopting such a configuration, in order to expose the corner of a large sheet material without using a human hand, the sheet material can be gripped at an appropriate gripping position and delivered to the next process.

[0022] The invention of the present application for solving the above problems is a processing system in which a computer executes a process of determining an appropriate gripping position in a sheet material in an arbitrary state, and includes an image acquisition unit that captures an image of the sheet material to acquire image data, a detection unit that detects the feature amount included in the image data, and a positioning unit that determines the gripping position using the feature amount.

[0023] By adopting such a configuration, in order to expose the corner of a large sheet material without using a human hand, the sheet material can be gripped at an appropriate gripping position and delivered to the next process.

Effects of the Invention

[0024] According to the present invention, it is possible to provide a processing method capable of recognizing a feature amount and determining a more appropriate gripping position even in a state where a large object is placed randomly.

Brief Description of the Drawings

[0025]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0026] Hereinafter, with reference to the drawings, the processing method X according to each embodiment of the present invention will be described. The description will detail the configuration of the embodiment, the method of implementation, and other examples in this order. Note that each of the following embodiments is an example of the present invention, and the present invention is not limited to the following embodiments.

[0027] <Embodiment 1> As shown in FIG. 1, the processing method X according to the present embodiment includes an image acquisition step S1, a detection step S2, and a positioning step S3. Further, it includes a gripping operation step S4 of gripping the sheet material Z at the gripping position of the sheet material Z determined in the positioning step S3, lifting it, and delivering it to the next system. Also, the processing method X is executed by the processing system Y shown in FIG. 3.

[0028] <<Processing System>> As shown in FIG. 4, the processing system Y is a system of an apparatus that executes a processing method X for gripping a sheet material Z at an appropriate position and delivering it to a downstream process, and includes an imaging unit 1, a processing device 2, a manipulator 3, a display unit 4 (not shown), and a conveying unit 5.

[0029] The imaging unit 1 is a camera (line camera or area camera) that images the sheet material Z as the object with information on brightness, chroma, and hue, and preferably has a light source unit for more reliable image recognition.

[0030] As shown in FIG. 3, the processing device 2 is a computer device including a control unit 21 and a storage unit 22, and also includes an input device, a storage device, an interface, etc. The processing system Y controls the imaging unit 1, the manipulator 3, the display unit 4, and the conveying unit 5 by this processing device 2, and also executes each process related to image recognition.

[0031] The control unit 21 is an arithmetic device such as a CPU (processor) or a microcomputer, and executes instructions executable by a computer.

[0032] The storage unit 22 includes a non-volatile memory that stores a program P for executing the processing method X according to the present embodiment, and a volatile memory 221 that temporarily stores the image data captured by the imaging unit 1. Data and the program P are read from each of them and executed by the control unit 21. However, as shown in FIG. 3, the storage unit 22 does not necessarily need to clearly separate the non-volatile memory and the volatile memory 221, and may be provided integrally as long as it can perform the above-described roles.

[0033] The storage unit 22 stores a program P for executing each process according to the present embodiment. The program P causes the processing device 2, particularly the control unit 21, to function as an image acquisition unit P1, a detection unit P2, a positioning unit P3, and a gripping operation unit P4.

[0034] The image acquisition unit P1 controls the imaging unit 1 to image the sheet material Z, and captures the captured image so that it can be handled as image data by the control unit 21.

[0035] The detection unit P2 includes a brightness recognition unit P21 that detects the unevenness on the surface of the sheet material Z appearing in the image data using brightness, a determination unit P22 that determines the presence or absence of the feature quantity F using the brightness distribution in a specific region within the image data, an allocation unit P23 that allocates a bounding box B to the feature quantity F, a continuity specifying unit P24 that connects the bounding boxes B to specify the continuity of the feature quantity F, a discrimination unit P25 that groups the feature quantities F with different positions and modes, and a shape estimation unit P26 that estimates the position of the edge ZH based on the mode of the detected feature quantity F and further estimates the type of the edge ZH of the sheet material Z depending on the detection situation of the feature quantity F.

[0036] The positioning unit P3 includes a priority assignment unit P31 that assigns a priority for selecting a gripping position for each group of the feature quantity F, and a temporary selection unit P32 that temporarily selects candidates for the gripping position from the feature quantity F for which the continuity has been specified.

[0037] The gripping operation unit P4 controls the manipulator 3, grips and lifts the sheet material Z (object) at the determined gripping position, and delivers it to the next process.

[0038] Hereinafter, the behavior when the processing is executed by each part of the program P will be described in detail in the description of the corresponding process.

[0039] The manipulator 3 has an end effector 31 capable of gripping the sheet material Z, and is a so-called robot arm that lifts and conveys the sheet material Z, and a configuration that operates with six degrees of freedom (for example, a configuration having four shaft joints) is preferable. Further, the end effector 31 preferably has a suction part that sucks the sheet material Z and a pair of grippers that sandwich the vicinity thereof in order to surely maintain the state of gripping the sheet material Z.

[0040] The display unit 4 is a display that shows the captured image data and the progress of image recognition, and a liquid crystal screen or the like is used. Thereby, the user can check the status of the image recognition of the sheet material Z, and can also visually recognize it and correct it, such as repositioning the sheet material Z, even when the placement of the sheet material Z is inappropriate. Further, the display unit 4 may be configured to also serve as an input device as a touch panel.

[0041] The conveying unit 5 is a movable mounting table on which the user places the sheet material Z and conveys the sheet material Z to the imaging range of the imaging unit 1, and a form using a belt conveyor or the like is assumed. Further, in order to facilitate the recognition of the sheet material Z, the conveying unit 5 is preferably made of a color that is significantly different from the sheet material Z, particularly a dark color such as black.

[0042] The sheet material Z is a cloth material provided with an edge element ZF used for image recognition. In the present embodiment, it is assumed that the sheet material Z is mainly a face towel, a bath towel, a sheet used for bedding, etc., and is of a size that cannot be held in one hand of a person. However, a relatively small one such as a hand towel or a handkerchief may be applied to the present embodiment. In the following description, for convenience of explanation, the cloth material shown in the image may also be referred to as the sheet material Z, but in that case, it refers to the data on the image corresponding to the sheet material Z.

[0043] The edge element ZF is provided along the edge ZH of the sheet material Z, and is a portion that serves as a mark (or clue) for specifying the edge ZH by the processing method X according to the present embodiment. The actual edge element ZF is, for example, a portion where the color yarn, pattern, weave pattern, presence or absence of pile, etc. change, that is, a region where the state of the yarn (sewing) changes in the vicinity of the edge ZH of the sheet material Z.

[0044] More specifically, it refers to a portion where the edge ZH of the sheet material Z can be visually recognized due to a change in the sewing pattern, such as a hem or an ear of a general towel or sheet. Further, the weave pattern that appears due to a change in the fabric texture, the flat portion of the towel (region without pile), the portion where a pattern or characters are printed later, etc. are preferably also treated as the edge element ZF.

[0045] The feature quantity F is an index based on numerical data representing a state in which the sheet material Z is imaged by the imaging unit 1 and the edge element ZF appears in the captured image. Specifically, it is the position, color, and orientation of the edge element ZF on the image. More specifically, it indicates how much the values of each data assigned to each pixel, such as brightness and RGB, change between adjacent pixels or within a predetermined region of the image.

[0046] The index used for the feature quantity F varies depending on the state even for the same sheet material Z. For example, it is assumed that the way light hits changes depending on the rounding method, and even for the same color yarn, the RGB values are different on the image, and similarly, the brightness values are different. In addition, even for a plain sheet material Z, it is assumed that the yarn (especially the pile) wears down due to repeated use and washing, and the brightness distribution (pattern) appearing in the image changes.

[0047] In order to avoid fluctuations in the recognition of the feature quantity F as described above, it is preferable that the detection criteria for the feature quantity F be configured to be able to handle certain fluctuations. However, the detection criteria for the feature quantity F may be configured based on rules, such as setting predetermined parameters in advance, or may be formed by learning.

[0048] Hereinafter, the implementation method according to the present embodiment will be described in detail with reference to the drawings. In addition, the implementation method shown below is an example, and the implementation method is not limited thereto, and the order may be reversed.

[0049] <<Processing Method>> Each program P according to the present embodiment stored in the storage unit 22 and the processing method X executed thereby proceed according to the flowchart shown in FIG. 1.

[0050] The image acquisition step S1 (corresponding to the image acquisition unit P1) includes an imaging step S11 of imaging the sheet material Z with the imaging unit 1.

[0051] In the imaging step S11, the control unit 21 controls the imaging unit 1 to perform color imaging on the sheet material Z, reads the captured image data for use in the next detection step S2, simultaneously stores it in the storage unit 22 (particularly the volatile memory 221), and further displays it on the display unit 4. At this time, it is preferable that the control unit 21 also performs a process of automatically imaging by recognizing the timing when the sheet material Z placed randomly (or conveyed by the conveying unit 5) stops. By such a process, the user only needs to place the sheet material Z within the imaging range, and it is possible to prevent a delay in the system operation due to an oversight in operation.

[0052] The control unit 21 may simultaneously perform a process of trimming unnecessary portions according to the position of the sheet material Z in the captured image for the captured image.

[0053] The detection step S2 (corresponding to the detection unit P2) includes a brightness recognition step S21, a determination step S22 for determining the presence or absence of the feature amount F, an assignment step S23 for assigning a bounding box B to the recognized feature amount F, and a continuity identification step S24 for identifying the continuity of the feature amount F. Further, it is preferable that the detection step S2 includes a discrimination step S25 for discriminating a plurality of types of feature amounts F and a shape estimation step S26 for estimating the shape of the sheet material Z, particularly the type of side (long side, short side, etc.) according to the type of the feature amount F.

[0054] In the brightness recognition step S21, the control unit 21 extracts brightness data from the read image data. In this process, the control unit 21 performs two types: macro extraction and micro extraction.

[0055] In macro extraction, the control unit 21 first divides the area where the sheet material Z moves in the read image into areas of a predetermined size (for example, 10 pixels square), then calculates the average value of the brightness of the pixels included in each area, and further compares the average values of the brightness between two adjacent areas. If the difference between the average values at this time is less than or equal to a predetermined value (for example, 20 or less), it is determined that the two predetermined areas belong to the same area of the sheet material Z.

[0056] After the above-described macro extraction process, the control unit 21 executes a micro extraction process. In this micro extraction, the control unit 21 recognizes a pattern in which the value of brightness varies every time there is a one-pixel shift in the area determined to be within the same area in the macro extraction. That is, the control unit 21 recognizes the change in brightness and darkness between pixels.

[0057] This brightness recognition step S21 can be replaced with a color recognition process (color recognition step). In this case, the brightness data for each pixel used in the detection is replaced with color, that is, each RGB value, and the control unit 21 performs macro extraction to micro extraction. Also, the brightness recognition step S21 may be a process step that uses both brightness and color (RGB values).

[0058] In the determination step S22, the control unit 21 recognizes the change in the brightness pattern within one area of the sheet material Z recognized in the brightness recognition step S21, or compares the recognized brightness pattern with the data previously stored in the storage unit 22. Then, when the control unit 21 finds a place where the pattern of the change in brightness changes significantly, it determines that it is the place where the sewing pattern changes and is the part of the feature amount F where the texture or the like changes. Thereby, the processing system Y can handle elements included in general plain towels and sheets such as the presence or absence of hems, ears, and piles as the feature amount F.

[0059] The brightness values used in the brightness recognition step S21 and the determination step S22 use the principle that the unevenness due to sewing of the sheet material Z is recognized as brightness by the reflection of light. With such a configuration, even if the sheet material Z is plain, the control unit 21 can recognize the feature amount F.

[0060] In the brightness recognition step S21 and the determination step S22, the control unit 21 mainly performs detection using numerical values related to brightness. However, when color (RGB numerical values) can be used, such as when the edge element ZF is a colored thread, it is preferable to detect the feature amount F using numerical values related to color.

[0061] In the allocation step S23, the control unit 21 first allocates segments for each pixel so as to match the recognized feature amount F, and then allocates a rectangular bounding box B to the aggregate of the segments. Preferably, the control unit 21 displays the result of allocating the bounding box B on the display unit 4.

[0062] The bounding box B is an icon allocated to one interval divided into a predetermined length or less among the aggregate of the allocated segments. In the bounding box B, the length of the side facing the length direction (continuous direction) of the feature amount F (aggregate of segments) can be arbitrary as long as it is equal to or less than the predetermined length, and all sides in the direction of dividing the feature amount F have a constant length.

[0063] In the allocation step S23, the control unit 21 allocates a plurality of bounding boxes B along the feature amount F and proceeds to the next step.

[0064] When there are a plurality of types of feature amounts F, the control unit 21 combines with different data such as changing the color (balance of each RGB value) of the bounding box B for each group to reflect the result of the discrimination step S25 (described later).

[0065] In the continuity identification step S24, for a plurality of bounding boxes B, when they intersect, or contact, or are close (determined by, for example, the number of separated pixels being less than a predetermined value) at the end in the length direction (or an arbitrary position of the short side), the control unit 21 identifies the continuity of the feature amount F by linking the bounding boxes B to each other. By performing this process on the image, substantially, an edge element ZF having continuity is recognized.

[0066] Here, "continuity" refers to a region where the pixels recognized as the feature amount F are adjacent at a plurality of positions, or a group of a plurality of linked bounding boxes B. In addition, when only dealing with the feature quantity F and not using the bounding box B, the allocation step S23 and the continuity determination step S24 are omitted. Also, when there is one bounding box B, the continuity determination step S24 is omitted.

[0067] The continuity determination step S24 may be a process in which the control unit 21 determines the proximity of segments, links segments to each other or aggregates of segments, and as a result, connects the bounding boxes B to each other.

[0068] In the control unit 21 in the processing method X shown in FIG. 1, in the discrimination step S25, when it is determined that the feature quantity F for which continuity has been determined can be divided into a plurality of groups or there are a plurality of types as a result of receiving the processing of the continuity determination step S24, the feature quantity F is grouped.

[0069] Separately from the above, the control unit 21 in the discrimination step S25 may perform the processing in the order that, as shown in FIG. 2, when it is determined that there are a plurality of types of patterns or distribution locations of the feature quantity F at the stage when the determination step S22 is completed, the feature quantity F is grouped according to the type. Also, when the recognized feature quantity F is one type or less, the control unit 21 does not perform the processing of the discrimination step S25 and proceeds to the next step.

[0070] In the shape estimation step S26, the control unit 21 estimates the position of the edge ZH of the sheet material Z based on the position, continuity, and length direction of the detected feature quantity F, and then estimates the type of the edge ZH that may exist in the vicinity of the edge element ZF corresponding to the feature quantity F based on the aspect of the feature quantity F. At this time, the control unit 21 particularly determines the long side and the short side of the sheet material Z.

[0071] The shape estimation step S26 is preferably incorporated into the processing method X when the edge element ZF is provided along the edge ZH of the sheet material Z. For example, in a general towel, edge elements ZF with different sewing patterns are provided on sides of different lengths, such as ears along the long side and hems along the short side, and the control unit 21 identifies these. As a result, in the positioning step S3 and the gripping operation step S4, the manipulator 3 can more easily grip the sheet material Z at a position where it is easy to handle, and errors in the operation of transporting the sheet material Z can be reduced.

[0072] The positioning step S3 (corresponding to the positioning unit P3) includes a priority assignment step S31 and a gripping position determination step S32. In this case, the control unit 21 determines the gripping position of the sheet material Z using the feature amount F specified in the detection step S2.

[0073] In the priority assignment step S31, the control unit 21 assigns priorities to be determined for the gripping position to each group of the bounding boxes B (i.e., the original feature amount F) grouped in the grouping step S25. This priority is, for example, a numerical value, and is assigned as "1", "2", "3",... in order from the most promising group to be the gripping position. Based on the assigned priorities, in the gripping position determination step S32, the gripping position is determined starting from the group with the smallest priority numerical value.

[0074] In the priority assignment step S31, the control unit 21 first compares the lengths of the continuous grouped bounding boxes B (segments), and assigns priority numerical values in order from the longest one.

[0075] Subsequently, the control unit 21 identifies the coordinates of the midpoint of each group (XY coordinates in the image) and determines whether each coordinate is within the prohibited area (including the boundary). This prohibited area refers to an area where the end effector 31 interferes with objects other than the sheet material Z, such as the edge of the conveyance unit 5 or the vicinity of the base of the manipulator 3, and the sheet material Z cannot be gripped. For the group determined to have its midpoint within the prohibited area in this determination, the control unit 21 lowers the assigned priority to the lowest level (a value 1 or 2 greater than the maximum value at that time).

[0076] Regarding the relationship between the midpoint of the group and the prohibited area, it is also conceivable that only the vicinity of the midpoint is located within the prohibited area and most of the group is outside the prohibited area. Therefore, even if the vicinity of the midpoint of the group is within the prohibited area, if a certain proportion or more (for example, 70% or more) of the length of the group is outside the prohibited area, the control unit 21 preferably does not lower the priority and executes a process of re - determining the gripping position candidate of the group to the point closest to the midpoint and outside the prohibited area.

[0077] The control unit 21 can also use information on whether the periphery of the group is bulging on the sheet material Z as a criterion for judgment in adjusting the priority. At the stage of the brightness recognition step S21, since the control unit 21 can estimate the vertical relationship between each area of the sheet material Z, if there is an area of the sheet material Z higher than the vicinity of the grouped bounding box B using this, the priority of that group is lowered. Thereby, when the manipulator 3 grips and lifts the sheet material Z, it is possible to prevent a situation where the sheet material Z does not spread well and the work is delayed in the subsequent steps of the present invention (including the placement step described later).

[0078] In order to recognize the shape of the sheet material Z at this time with higher accuracy, the processing system Y may be configured to image the sheet material Z with a plurality of imaging units 1 and handle the shape of the sheet material Z in topography.

[0079] In the priority assignment step S31, after the control unit 21 assigns a priority, it is preferable to perform a process of lowering the priority for a group of bounding boxes B (or rather, feature amounts F) that satisfy a predetermined condition. The "predetermined condition" here means, for example, when the feature amount F within the group is located along the short side of the sheet material Z, or when the length of the continuous bounding box B is equal to or less than a certain value. By lowering the priority assigned to these, it is possible to prevent a situation where the sheet material Z sags along the long side and the height to be lifted by the manipulator 3 becomes unnecessarily high, or a situation where the sheet material Z does not spread when lifted by the manipulator 3 because an edge element ZF with a short exposed length is gripped.

[0080] In the gripping position determination step S32, the control unit 21 determines either the detected feature amount F or the estimated edge ZH as the position where the sheet material Z is to be gripped by the end effector 31. As an example, the control unit 21 selects one point among the feature amounts F as the gripping position and determines its XY coordinates. At this time, it is preferable to use a method that can determine the gripping position based on a predetermined criterion, such as near the midpoint, among the feature amounts F or edges ZH for which continuity has been specified.

[0081] In the gripping position determination step S32, the control unit 21 determines the midpoint (or the re-set gripping position candidate) of the group with the smallest priority value as the gripping position based on the priorities for each group that were assigned and further adjusted in the priority assignment step S31.

[0082] The gripping operation step S4 (corresponding to the gripping operation unit P4) includes a gripping step S41, a lifting step S42, and a handover step S43. In the gripping operation step S4, the control unit 21 operates (controls) the manipulator 3 to grip the sheet material Z and transfers it to the next step.

[0083] In the gripping step S41, the control unit 21 controls the manipulator 3 to grip the gripping position of the sheet material Z determined in the positioning step S3 with the end effector 31. However, since the gripping position determined in the positioning step S3 is merely XY coordinates on the image, the control unit 21 converts the coordinates into coordinates corresponding to the manipulator 3. For the Z coordinate of the manipulator 3, a pre-specified value, or a value determined using the vertically measured height estimated from the captured image, etc. is assumed. At the location specified as above, the manipulator 3 grips the sheet material Z with the end effector 31. Further, the control unit 21 may adjust the orientation of the end effector 31 so that the direction in which the gripper opens is perpendicular or parallel to the edge element ZF.

[0084] In the lifting step S42, the control unit 21 lifts the sheet material Z gripped by the end effector 31 to a predetermined height. This lifting height may be a pre-specified height or a value appropriately changed according to the state of the sheet material Z. This makes it easier for the edges and corners of the sheet material Z to be exposed in a hanging state, creating a state in which the next step (the operation of placing the sheet material Z in a state where the corners are exposed) can be easily executed.

[0085] In the handover step S43, the control unit 21 hands over the lifted sheet material Z from the manipulator 3 that is gripping it in the present invention to the downstream manipulator that executes the next step. At this time, it is preferable for the manipulator 3 to have the downstream manipulator grip a location close to the gripping position of the sheet material Z and perform the handover as if passing it. This enables the shape of the lifted sheet material Z to be maintained as much as possible.

[0086] The processing method X according to this embodiment may have, as the next processing step after the gripping operation step S4, a placing step of placing the delivered sheet material Z with its edge ZH or corner exposed. Thereby, the corner of the sheet material Z can be mechanically gripped by the folding manipulator provided immediately after the present invention, and the folding operation can be easily performed. Further, the placing step requires a program P for the processing device 2 to execute the processing, and a downstream manipulator and a movable placing table, and these can be executed after being added to the present invention.

[0087] The processing method X according to this embodiment may have the following modification examples. However, the modification examples shown below are merely examples, and the presence or absence of each example is determined independently of each other, unless there is a specific dependency relationship.

[0088] <<Modification Example>> In the determination of colors in a color image, the RGB color system was used in the above description, but other systems may be used as long as they are quantifiable color systems. For example, the CMYK color system, the XYZ color system, the Lab color system, etc. can be applied. Even in these cases, it is preferable that the control unit 21 is configured to be able to cope with variations caused by individual differences and wear of the sheet material Z as described above, and further that the configuration and method can be used in the detection step S2.

[0089] As a configuration capable of coping with variations, for example, it is preferable to use the three - point color value feature amount TF as the feature amount F. The three - point color value feature amount TF is merely an example of the feature amount F, and an index calculated by other methods may be used.

[0090] The three - point color value feature amount TF is calculated by the following formula based on the maximum values (MR, MG, MB, ML), minimum values (mR, mG, mB, mL), and average values (AR, AG, AB, AL) of the distribution of each RGB value and / or brightness (hereinafter referred to as character L) according to the RGB color system (or other color systems) of each pixel in a predetermined region within the image. Red - component feature amount = (MR - mR) / AR Green - component feature amount = (MG - mG) / AG Blue - component feature amount = (MB - mB) / AB Characteristic quantity of L component = (ML - mL) / AL As described above, regardless of the shape of the sheet material Z, the positional relationship with the light source unit, or the deterioration of the sheet material Z, the accuracy of detecting the characteristic quantity F can be maintained.

[0091] The processing device 2 and / or the detection unit P2 may have an abnormal placement notification unit. When the characteristic quantity F cannot be detected by the detection unit P2 (and the detection step S2) due to the shape of the sheet material Z, this abnormal placement notification unit notifies the user to that effect and requests the user to reposition the sheet material Z so that the edge element ZF can be imaged. The form of notification is assumed to be, for example, displayed as a pop-up on the display unit 4 or emitting sound from a separately provided speaker.

[0092] In the image acquisition step S1 of the processing method X according to the present embodiment, the sheet material Z may be photographed by a plurality of cameras, the topography of the sheet material Z may be acquired, and the detection step S2 and the positioning step S3 may be executed using the topography. At this time, up to the detection step S2 is as described above, but in the positioning step S3, particularly in the priority assignment step S31, the control unit 21 can use the "height" of the bounding box B based on the topography.

[0093] In the priority assignment step S31 in this case, the control unit 21 determines the order of priority for each group of the bounding boxes B grouped in the discrimination step S25 using the length, midpoint, and highest point of the group.

[0094] For example, the control unit 21 raises the priority of a group having a length equal to or greater than a preset value for the length of the group, identifies the highest point within each group for the highest point, and further determines the positional relationship with the midpoint. In response to this, the control unit 21 generally raises the priority of the group with the highest height of the highest point and assigns priority so that the edge element ZF at the highest position can be gripped. However, when the distance between the highest point and the middle point within the group is extremely far apart (the highest point is near the edge of the group, and when gripping the highest point, it may not be possible to grip near the center of the edge element ZF), etc., when certain conditions are met, the control unit 21 executes a process of assigning priority using the middle point instead of the highest point and determining the gripping position.

[0095] The processing method X (processing system Y) according to this embodiment may have a learning process for improving the accuracy of the detection step S2 and the positioning step S3 instead of a processing step using a preset criterion such as the three-color value feature amount TF. This learning process, for example, causes the processing system Y to read in teacher data, and the control unit 21 determines and optimizes parameters for detecting the feature amount F and specifying continuity. Specifically, in this learning process, for an image obtained by imaging the sheet material Z, pixels that are the feature amount F and continuity are specified by human processing, and a dataset of the labeled image and the original image is used as teacher data.

[0096] Each of the learning processes shown above is merely a specific example, and the processing method X (processing system Y) according to the present invention may have existing learning algorithms such as YOLO (registered trademark), YOLACT, and MASK-RCNN.

[0097] In addition to the above, the processing system Y may have a learning process for improving the accuracy of the detection step S2 and the positioning step S3 while feeding back data that is the result of processing the sheet material Z. In this case, the learning process is assumed to include a feature amount detection pattern learning process L1, a continuity specification pattern learning process L2, and a gripping position determination pattern learning process L3. Each learning method will be described in detail below.

[0098] The feature amount detection pattern learning process L1 is a learning process for more accurately detecting a portion corresponding to the edge element ZF in the detection of the feature amount F appearing in the image, particularly in the assignment of segments (segmentation). Assuming that the processing is actually performed, due to the way light hits the sheet material Z and the wear of sewing of the sheet material Z, individual differences occur in the appearance of the feature amount F even in the image, which can reduce the accuracy of segmentation. Therefore, in the process of the feature amount detection pattern learning step L1, the control unit 21 sets a combination of the brightness value and the RGB value (preferably the three-color value feature amount TF) for each pixel as an input data set together with the segmentation region, stores this in the storage unit 22, and associates it with the state when the sheet material Z is lifted in the subsequent lifting step S42. At this time, the state of the sheet material Z is imaged by the imaging unit for the lifted state. If the vertical length of the sheet material Z is greater than or equal to a predetermined value, a penalty (negative value) is associated with the data set, and if it is less than the predetermined value, a reward (positive value) is associated with the data set. The control unit 21 repeats this for each series of processes and reproduces, by segmentation, a state close to the data set when a reward is obtained. Thereby, the feature amount F can be detected more accurately regardless of the individual differences of the sheet material Z.

[0099] The continuity identification pattern learning step L2 is a learning step for more accurately identifying the continuity of the feature amount F in the continuity identification step S24. For example, in FIG. 7, the third group bounding box B3 and the fourth group bounding box B4 are highly likely to be judged by the user as being continuous. In addition, there are cases where the edge element ZF that is originally continuous is blocked by the folded-up part above depending on the way the sheet material Z is rounded, and is judged as discontinuous in the image. To eliminate this, the continuity identification pattern learning step L2 learns the above patterns and detects a feature amount F that is continuous for a longer time.

[0100] In the process of the continuity identification pattern learning step L2, the control unit 21 stores the position and orientation of the grouped bounding box B and the information of the segment to which it belongs in the storage unit 22, and further stores it in the storage unit 22 as a data set combined with the distribution of colors (brightness value, RGB value, etc.) in the image. The control unit 21 associates this dataset with the state of the lifted sheet material Z and the reward / punishment data, which is the same as described above. The control unit 21 repeats this for each series of processes, and reproduces in the continuity identification step S24 a state close to the dataset when a reward is obtained. Thereby, regardless of the individual differences of the sheet material Z, the continuity of the feature amount F can be detected more accurately, and a more appropriate gripping position can be determined.

[0101] The gripping position determination pattern learning step L3 is a learning step for improving the probability of being able to determine a more appropriate gripping position in the positioning step S3. In the positioning step S3, the control unit 21 determines, from the feature amount F having continuity, those that satisfy a predetermined criterion as the gripping position, and associates the data used therewith, for example, the data set obtained by summarizing the end portions, intermediate points of the group of bounding boxes B, and the coordinate data of the positional relationship of each group, with the state of the lifted sheet material Z and the reward / punishment data, which is the same as described above. The control unit 21 repeats this for each series of processes, and optimizes the process of gripping position determination, such as adding fluctuations to the coordinate data, so that the process of the positioning step S3 is performed in a state close to the dataset when a reward is obtained. Thereby, regardless of the individual differences of the sheet material Z, a more appropriate gripping position can be determined.

[0102] When the sheet material Z handled in the present invention is a plain towel or the like, as described above, the edge element ZF and the feature amount F thereof correspond to the hem, ears, and pile, and the control unit 21 in the detection step S2 detects these. At this time, the processes performed by the control unit 21 from the detection step S2 to the positioning step S3 can be classified as shown in the table of FIG. 8.

[0103] When applying the classification shown in FIG. 8 to the present embodiment, it is assumed that in the positioning step S3, the portions to be preferentially determined as the gripping position among the hem, ears, and pile are preset. Note that the ○ marks in FIG. 8 represent the state in which each feature amount F can be recognized, and the × marks represent the state in which each feature amount F cannot be recognized.

[0104] The first row of the table shown in FIG. 8 indicates the case where the control unit 21 cannot detect any of the hem, ears, and pile. At this time, the control unit 21 determines that subsequent processing cannot be performed, in other words, it cannot proceed to the positioning step S3. Thereafter, the control unit 21 executes processes such as removing the sheet material Z from the line of the conveying unit 5, or notifying the user of an error, and requesting the user to re-place the sheet material Z in a state where some feature amount F can be detected. However, it is preferable that the control unit 21 can operate the manipulator 3 to change the curling manner (placement state) of the sheet material Z.

[0105] The second, third, and fifth rows of the table shown in FIG. 8 indicate the case where the control unit 21 detects any one of the hem, ears, and pile. At this time, in the positioning step S3, the control unit 21 preferentially compares it with the portion to be the gripping position. When the setting of this gripping position matches the type of the detected feature amount F, the control unit 21 determines the gripping position from the feature amount F. For example, if the portion to be preferentially the gripping position is set as the ears and the feature amount F detected by the control unit 21 also relates to the ears, then in the positioning step S3, the gripping position is determined from the feature amount F related to the ears. When the setting of the gripping position is different from the type of the detected feature amount F, the control unit 21 determines the gripping position according to another criterion from the region where the feature amount F is not detected. This criterion is assumed to be, for example, setting the widest identical region as the gripping position in the brightness recognition step S21.

[0106] The fourth, sixth, and seventh rows of the table shown in FIG. 8 indicate the case where the control unit 21 detects any two of the hem, ears, and pile. At this time, similar to the previous case, in the positioning step S3, the control unit 21 preferentially compares it with the portion to be the gripping position. When the setting of this gripping position matches the type of the detected feature amount F, the control unit 21 determines the gripping position from the feature amount F. When setting the gripping position and when the types of the detected feature amounts F are different, the control unit 21 estimates the position of the remaining one feature amount F or the gripping position in place thereof from the two detected feature amounts F, and determines the gripping position therefrom.

[0107] The eighth row of the table shown in FIG. 8 shows the case where the control unit 21 has detected three types: hem, ear, and pile. In this case, the control unit 21 determines the gripping position in accordance with the portion set as the priority gripping position in the positioning step S3.

[0108] The processing method X (including modification examples) shown above is represented and executed by a computer-processable program P.

[0109] Hereinafter, with reference to the drawings, examples in which the processing method X according to the present embodiment is executed will be described in detail. Further, each of the following examples is an example of the processing for executing the present embodiment, and the mode and the order of each step may be changed.

[0110] <<Example 1>> FIG. 6 is an image obtained by imaging the sheet material Z with the imaging unit 1 and further trimming by the control unit 21. At this time, the edge element ZF of the sheet material Z appears as the feature amount F in the image. In response to this, the control unit 21 detects the feature amount F in the detection step S2. Since the edge element ZF in this example is a colored yarn, as described above, the three-color value feature amount TF using the RGB color system is used.

[0111] To the feature amount F detected as described above, the control unit 21 assigns the bounding box B in the assignment step S23. Here, when the control unit 21 performs the process of the discrimination step S25, in the assignment step S23, different groups of bounding boxes B such as the first group bounding box B1 and the second group bounding box B2 are assigned.

[0112] Furthermore, in the continuity determination step S24, the control unit 21 determines the continuity of each feature amount F by combining the bounding box B with each of the first group bounding box B1 and the second group bounding box B2.

[0113] When a plurality of types of feature amounts F are shown in the image, the control unit 21 executes the process of the discrimination step S25. In the example in FIG. 6, the modes (visual modes of the edge elements ZF) of each feature amount F to which the first group bounding box B1 and the second group bounding box B2 are assigned are different. Such differences in modes are detected by elements such as, for example, different colors (distribution of each RGB value) of the feature amount F, different positions (continuously parallel and no intersections or merging points are recognized), different sewing patterns (one is sewn in a straight line, while the other is sewn in a zigzag, etc.).

[0114] After the first group bounding box B1 and the second group bounding box B2 are recognized, in the shape estimation step S26, the control unit 21 estimates the long side and the short side of the sheet material Z. Also, even when the process of the shape estimation step S26 is not performed, in the positioning step S3, the position of the edge ZH of the sheet material Z is estimated in the shape estimation step S26. Through the above processes, the control unit 21 determines the position of any feature amount F or the position of the estimated edge ZH as the gripping position to be gripped by the manipulator 3.

[0115] In the priority assignment step S31, the control unit 21 assigns a priority to the group of the bounding box B to be the gripping position from the information on the edge ZH of the sheet material Z and its length and width estimated by the process of the shape estimation step S26 and the information on the position and mode of the corresponding feature amount F. This priority is a numerical value as described above. For example, the control unit 21 assigns a priority of "1" to the second group bounding box B2 in FIG. 6 and a priority of "2" to the first group bounding box B1. In this case, the control unit 21 estimates that the second group bounding box B2 is closer to the edge ZH of the sheet material Z than the first group bounding box B1, and accordingly assigns a smaller value of priority to the former.

[0116] In the gripping position determination step S32, the control unit 21 determines the gripping position from the group of bounding boxes B with a small priority value, which is the second group bounding box B2 in the example of FIG. 6. The gripping position determined at this time may be selected from any position on the second group bounding box B2, or may be selected from a position that is separated from the second group bounding box B2 by a predetermined distance (for example, 10 mm) toward the edge ZH of the sheet material Z.

[0117] Also, in the gripping position determination step S32, the control unit 21 preferably determines the vicinity of the center of the second group bounding box B2 as the gripping position. More preferably, when the corner of the sheet material Z is recognized at the end of the second group bounding box B2, the control unit 21 determines the gripping position at a position shifted from the center of the second group bounding box B2 to the side opposite to the corner. By such a method, the end effector 31 can easily grip the vicinity of the midpoint of the side of the sheet material Z, suppress the height at which the sheet material Z is lifted by the manipulator 3, and suppress the space required for the operation to be low. In addition, it is possible to suppress the possibility that the lifted and sagging sheet material Z contacts the surrounding devices and gets caught, hindering the operation.

[0118] <<Example 2>> FIG. 7 shows a state in which the first group bounding box B1, the second group bounding box B2, the third group bounding box B3, and the fourth group bounding box B4 are assigned to each feature amount F and the continuity of each is specified. Particularly in the state shown in FIG. 7, the ends of each group of the second group bounding box B2 to the fourth group bounding box B4 are close to each other, and appear to be continuous to the user. In such a case, it is preferable that the control unit 21 performs a process of specifying the continuity between the groups on the condition that the ends of the groups are separated by a predetermined length (number of pixels) or less. Alternatively, in the priority assignment step S31, it is preferable that the control unit 21 assigns consecutive priorities to each of the second group bounding box B2 to the fourth group bounding box B4.

[0119] However, in the above case, the control unit 21 assigns the minimum priority to the group in the center. That is, in the state of FIG. 7, the control unit 21 assigns priority "1" to the third group bounding box B3, priority "2" to the fourth group bounding box B4, and priority "3" to the second group bounding box B2. By such a method, in the positioning step S3, it becomes easier for the control unit 21 to select the vicinity of the center of the edge element ZF having continuity as the gripping position.

[0120] Hereinafter, the processing method X according to the second embodiment of the present invention will be described in detail with reference to the drawings. However, for the content common to the first embodiment, the description will be omitted using the same reference numerals.

[0121] <<Embodiment 2>> As shown in FIG. 9, the processing system Y according to the present embodiment includes two manipulators 3. Correspondingly, also in the positioning step S3, the control unit 21 determines two gripping positions, that is, a pair of coordinates.

[0122] In the gripping position determination step S32 in this embodiment, the control unit 21 determines two gripping positions from the group with the highest priority. At this time, the gripping positions (a set of coordinates indicating points) are determined so that the two points to be gripped are separated by a predetermined linear distance. The "predetermined linear distance" here is a length determined by the size of the end effector 31, the size of the gripper of the downstream manipulator that grips the sheet material Z in the next step of the present invention, the stretching length required in subsequent steps, and the like.

[0123] In the gripping position determination step S32, there may be a case where the length of the group selected by priority is shorter than the "predetermined linear distance" for determining the gripping position. In this case, the control unit 21 first changes the group to be used to the next priority. If the gripping position still cannot be determined, the control unit 21 uses the information related to the shape of the sheet material Z obtained in the detection step S2 (particularly the brightness recognition step S21 and the shape estimation step S26) to determine a set of points (for example, two points) separated by a predetermined surface distance (although an estimated value is assumed to be used, an actual measured value is preferred if it can be actually measured) in an arbitrary group (outside the prohibited area) as the gripping position. This surface distance is the distance between two points separated by a predetermined distance along the surface of the sheet.

[0124] When the number of manipulators 3 of the processing system Y according to this embodiment increases to 3 or more, the determined gripping positions also increase corresponding to the number of manipulators 3. The control unit 21 also executes the process of determining each point (set of coordinates) of the gripping position related to the gripping position determination step S32 in the same manner as the above-described process.

Explanation of Reference Numerals

[0125] X Processing method Y Processing system 1 Imaging unit 2 Processing device 3 Manipulator 4 Display unit 5 Conveying unit P Program P1 Image acquisition unit P2 Feature quantity detection unit P3 Position Determination Unit P4 Gripping Operation Unit S1 Image Acquisition Process S2 Detection Process S3 Position Determination Process S4 Gripping Operation Process F Feature Quantity B Bounding Box Z Sheet Material ZF Edge Element ZH Edge

Claims

1. A processing method for a computer to execute a process of determining an appropriate gripping position on a sheet material in any state, an image acquisition step of imaging the sheet material to acquire image data, a detection step of detecting a feature amount included in the image data, a positioning step of determining the gripping position using the feature amount, and a processing method including the above.

2. In the detection step, a segment of an edge element provided on the sheet material is determined using the feature amount, and in the positioning step, the gripping position is determined based on the segment. The processing method according to claim 1.

3. The edge element is provided along the edge of the sheet material, in the detection step, the continuity of a plurality of the segments is determined, and the plurality of segments determined to have the continuity are grouped, and in the positioning step, the gripping position is determined using the elements of the group. The processing method according to claim 2.

4. The sheet material has the edge elements distinguishable for each edge, in the detection step, each edge element is discriminated from the feature amount and grouped, and in the positioning step, a point that meets a predetermined condition is determined as the gripping position from the feature amounts corresponding to each of the groups. The processing method according to claim 2.

5. When the positioning step determines a plurality of gripping positions, in the segment determined to have the continuity, a set of points separated from each other by a predetermined linear distance is determined as the gripping position. The processing method according to claim 3.

6. When determining a plurality of gripping positions in the positioning step, in the segment determined to have the continuity, a set of points spaced apart from each other by a predetermined distance along the surface is determined as the gripping position. The processing method according to claim 3.

7. A program that causes a computer to function as an image acquisition unit that captures an image of a sheet material to acquire image data, a detection unit that detects the feature amount included in the image data, and a positioning unit that determines a gripping position of the sheet material using the feature amount, and determines an appropriate gripping position in the sheet material in any state.

8. A processing system in which a computer executes a process of determining an appropriate gripping position in a sheet material in any state, the processing system including an image acquisition unit that captures an image of the sheet material to acquire image data, a detection unit that detects the feature amount included in the image data, and a positioning unit that determines the gripping position using the feature amount. ​

Images