Sheet material processing method, processing system, and program

Abstract

The present invention addresses the problem of providing a processing method that makes it possible to determine in a pre-stage an appropriate gripping position for exposing a feature of a sheet material so that a corner of the target material can eventually be exposed to enable work, even in cases where the target material is large and placed haphazardly. In order to solve the problem, the present invention is a processing method X for a computer to execute processing for determining an appropriate gripping position on a sheet material Z in any state, the processing method X comprising: an image acquisition step S1 for capturing an image of the sheet material Z to acquire image data; a detection step S2 for detecting a feature F included in the image data; and a position determination step S3 for determining the gripping position using the feature F.

Description

Sheet material processing method, processing system, and program

[0001] The present invention relates to a processing method for recognizing the shape of a sheet material, such as a sheet, placed haphazardly and detecting an appropriate portion to grasp before the sheet material is automatically folded, and a processing system for executing this processing.

[0002] In the linen supply industry and accommodation facilities such as hotels, large quantities of sheet materials are handled for changing and washing sheets. In particular, neatly folding sheet materials such as sheets and bath towels, which are the same size as or larger than the human body, requires considerable effort. Therefore, it is desirable to automate the folding of sheet materials, and the applicant of the present application has developed a folding device for this purpose.

[0003] On the other hand, when feeding sheet materials such as sheets into a folding device, it is necessary to align the shape and orientation of the sheet materials to a certain extent. This task has traditionally been performed by manually opening the sheet materials or identifying the type of sheet material, so there has been a demand for automation of this process. To address this issue, devices have been developed that utilize distinctive features of the sheet materials, such as corners or patterns, as disclosed in Patent Documents 1 and 2.

[0004] U.S. Patent No. 9909252 JP 2023-011021 A

[0005] Patent Literature 1 discloses a system that recognizes and grasps a small sheet material such as a towel by recognizing its four corners. However, it is difficult to recognize the four corners of a large object such as a sheet or bath towel when the object is casually rolled up. Therefore, it is necessary to automate the process of reliably exposing the corners and mechanically grasping the object, regardless of the size of the sheet material.

[0006] Furthermore, Patent Document 2 discloses a system that determines the pattern of laundry by image recognition. However, in order to recognize the pattern, the object needs to be spread out and placed on the laundry, and relying on human intervention for this process alone is rather inefficient.

[0007] The present invention has been made in consideration of the above-described situation, and provides a processing method that can determine an appropriate gripping position to expose the characteristics of a sheet material in a preliminary stage, so that even if a large object is placed carelessly, the corners can ultimately be exposed and work can be performed on it.

[0008] [1] The present invention, which solves the above-mentioned problems, is a processing method for a computer to execute a process for determining an appropriate gripping position for sheet material in any state, and includes an image acquisition process for capturing an image of the sheet material to acquire image data, a detection process for detecting feature amounts contained in the image data, and a position determination process for determining the gripping position using the feature amounts.

[0009] According to the present invention, a large sheet material can be gripped at an appropriate gripping position without manual intervention and handed over to the next process for exposing the corners of the sheet material.

[0010] [2] In a preferred embodiment of the present invention, the detection step uses the feature to determine segments of edge elements of the sheet material, and the position determination step determines the gripping position based on the segments, in the processing method described in [1].

[0011] By adopting such a configuration, the vicinity of the edge of the sheet material can be grasped using image recognition.

[0012] [3] In a preferred embodiment of the present invention, the edge element is provided along the edge of the sheet material, and the detection step determines the continuity of a plurality of the segments and divides the plurality of segments determined to have the continuity into groups, and the position determination step determines the gripping position using elements of the group, which is a processing method described in [1] or [2].

[0013] By adopting such a configuration, edge elements can be recognized with high accuracy even when they are visually recognized as curved lines.

[0014] [4] In a preferred embodiment of the present invention, the sheet material has edge elements that can be distinguished for each edge, and the detection process distinguishes and groups each of the edge elements based on the feature amounts, and the position determination process determines a point that meets predetermined conditions as the gripping position from the feature amounts corresponding to each of the groups, in a processing method described in any one of [1] to [3].

[0015] By adopting such a configuration, it is possible to grip the long side of the sheet material as close as possible, and the sheet material can be easily lifted.

[0016] [5] In a preferred embodiment of the present invention, the position determination step is a processing method according to any one of [1] to [4], in which, when determining multiple gripping positions, a set of points that are separated from each other by a predetermined linear distance within the segment that has been determined to have the continuity is determined as the gripping position.

[0017] By adopting such a configuration, even when the sheet material is gripped at a plurality of positions, it can be gripped and lifted at an appropriate gripping position.

[0018] [6] In a preferred embodiment of the present invention, the position determination step is a processing method according to any one of [1] to [5], in which, when determining multiple gripping positions, a set of points that are separated from each other by a predetermined creepage distance within the segment that has been determined to have the continuity is determined as the gripping position.

[0019] By using such a configuration, even when the sheet material is gripped at a plurality of positions, it can be gripped and lifted at a gripping position that is more appropriate for the shape of the sheet material.

[0020] [7] The present invention, which solves the above problem, is a program that causes a computer to function as an image acquisition unit that captures an image of a sheet material and acquires image data, a detection unit that detects the feature amounts contained in the image data, and a position determination unit that determines the gripping position of the sheet material using the feature amounts, thereby determining the appropriate gripping position for the sheet material in any state.

[0021] With this configuration, the corners of a large sheet material can be exposed without human intervention by gripping the sheet material at an appropriate gripping position and transferring it to the next process.

[0022] [8] The present invention, which solves the above problem, is a processing system in which a computer executes a process to determine an appropriate gripping position for sheet material in any state, and includes an image acquisition unit that captures an image of the sheet material to acquire image data, a detection unit that detects the feature amounts contained in the image data, and a position determination unit that determines the gripping position using the feature amounts.

[0023] With this configuration, the corners of a large sheet material can be exposed without human intervention by gripping the sheet material at an appropriate gripping position and transferring it to the next process.

[0024] According to the present invention, it is possible to provide a processing method that can recognize features and determine a more appropriate gripping position even when a large object is placed haphazardly.

[0025] FIG. 1 is a flowchart of a processing method according to a first embodiment of the present invention. FIG. 2 is a flowchart of a processing method according to a first embodiment of the present invention. FIG. 3 is a block diagram of hardware that executes a processing method according to a first embodiment of the present invention. FIG. 4 is a schematic diagram of hardware that executes a processing method according to a first embodiment of the present invention. FIG. 5 is an image showing an image recognition state of feature amounts according to a first embodiment of the present invention. FIG. 6 is an image showing an image recognition state of feature amounts according to a first embodiment of the present invention. FIG. 7 is a table showing detection patterns of feature amounts according to a first embodiment of the present invention and corresponding processing cases. FIG. 8 is a schematic diagram of hardware that executes a processing method according to a second embodiment of the present invention.

[0026] Hereinafter, a processing method X according to each embodiment of the present invention will be described with reference to the drawings. The description will be made in detail in the order of the configuration of the embodiment, the method of implementation, and other examples. Note that each of the embodiments shown below is an example of the present invention, and the present invention is not limited to each of the following embodiments.

[0027] 1, a processing method X according to this embodiment includes an image acquisition step S1, a detection step S2, and a position determination step S3, and further includes a gripping operation step S4 in which the sheet material Z is gripped at the gripping position of the sheet material Z determined in the position determination step S3, lifted up, and handed over to a next system. The processing method X is also executed by a processing system Y shown in FIG.

[0028] <<Processing System>> As shown in Figure 4, the processing system Y is a system of devices that executes a processing method X to grasp a sheet material Z at an appropriate position and hand it over to a downstream process, and is equipped with an imaging unit 1, a processing device 2, a manipulator 3, a display unit 4 (not shown), and a conveying unit 5.

[0029] The imaging unit 1 is a camera (line camera or area camera) that captures the target sheet material Z with information on brightness, saturation, and hue, and is preferably configured to include a light source unit for more reliable image recognition.

[0030] 3, the processing device 2 is a computer device including a control unit 21 and a storage unit 22, and also includes an input device, a storage device, an interface, etc. The processing system Y uses the processing device 2 to control the imaging unit 1, the manipulator 3, the display unit 4, and the transport unit 5, as well as to execute various processes related to image recognition.

[0031] The control unit 21 is an arithmetic unit such as a CPU (processor) or a microcomputer, and executes computer-executable instructions.

[0032] The storage unit 22 includes a nonvolatile memory that stores a program P that executes the processing method X according to this embodiment, and a volatile memory 221 that temporarily stores image data captured by the imaging unit 1, and the data and program P are read from each memory and executed by the control unit 21. However, as shown in Fig. 3, the storage unit 22 does not need to have a clearly separated nonvolatile memory and volatile memory 221, and may be provided as an integrated memory as long as they can fulfill the above-mentioned roles.

[0033] A program P for executing each process according to this embodiment is stored in the storage unit 22. The program P causes the processing device 2, particularly the control unit 21, to function as an image acquisition unit P1, a detection unit P2, a position determination unit P3, and a grip operation unit P4.

[0034] The image acquisition unit P1 controls the imaging unit 1 to capture an image of the sheet material Z, and imports the captured image so that it can be handled as image data by the control unit 21.

[0035] The detection unit P2 includes a brightness recognition unit P21 that detects the unevenness of the surface of the sheet material Z that appears in the image data using brightness; a judgment unit P22 that determines the presence or absence of a feature F using the distribution of brightness in a specific area within the image data; an assignment unit P23 that assigns a bounding box B to the feature F; a continuity identification unit P24 that connects the bounding boxes B to identify the continuity of the feature F; a distinction unit P25 that groups feature F of different positions and forms; and a shape estimation unit P26 that estimates the position of the edge ZH based on the form of the detected feature F, and further estimates the type of the edge ZH of the sheet material Z depending on the detection status of the feature F.

[0036] The position determination unit P3 has a priority assignment unit P31 that assigns a priority to each group of feature values ​​F for selection as a gripping position, and a temporary selection unit P32 that temporarily selects a candidate gripping position from the feature values ​​F whose continuity has been identified.

[0037] The gripping operation unit P4 controls the manipulator 3 to grip and lift the sheet material Z (object) at the determined gripping position, and deliver it to the next process.

[0038] Below, the behavior of each part of the program P when it executes processing will be described in detail in the explanation of the corresponding step.

[0039] The manipulator 3 is a so-called robot arm that has an end effector 31 capable of grasping the sheet material Z, lifts up and transports the sheet material Z, and is preferably configured to operate with six degrees of freedom (for example, a configuration having four axis joints). Furthermore, the end effector 31 is preferably configured to have a suction portion that sucks the sheet material Z and a pair of grippers that clamp the vicinity of the suction portion in order to reliably maintain the state in which the sheet material Z is grasped.

[0040] The display unit 4 is a display that displays the captured image data and the progress of image recognition, and is made of a liquid crystal screen or the like. This allows the user to check the status of image recognition of the sheet material Z, and if the placement of the sheet material Z is not appropriate, the user can visually confirm this and make corrections such as repositioning the sheet material Z. The display unit 4 may also be configured as a touch panel that doubles as an input device.

[0041] The transport unit 5 is a movable platform on which a user places the sheet material Z and transports the sheet material Z into the imaging range of the imaging unit 1, and is assumed to be in the form of a belt conveyor or the like. In addition, in order to make the sheet material Z easy to recognize, the transport unit 5 is preferably configured in a color that is significantly different from the sheet material Z, particularly a dark color such as black.

[0042] The sheet material Z is a cloth material having edge elements ZF used for image recognition, and in this embodiment, it is assumed to be a cloth material that is too large to fit in one hand, such as a face towel, a bath towel, or a sheet used for bedding. However, relatively small items such as hand towels and handkerchiefs may also be applied to this embodiment. In the following description, for convenience of explanation, the cloth material shown in the image may also be referred to as the sheet material Z, but in this case, it refers to the data on the image corresponding to the sheet material Z.

[0043] The edge element ZF is provided along the edge ZH of the sheet material Z and serves as a mark (or clue) for identifying the edge ZH in the processing method X according to this embodiment. The actual edge element ZF is a region where, for example, the color thread, pattern, weave pattern, presence or absence of pile, etc. changes, i.e., a region where the thread (sewing) pattern changes near the edge ZH of the sheet material Z.

[0044] More specifically, it refers to a portion where the edge ZH of the sheet material Z can be seen due to a change in the stitching pattern, such as the hem or selvedge of a typical towel or sheet. In addition, it is preferable that a woven pattern that appears due to a change in the weave, a flat portion of a towel (an area without pile), or a portion where a pattern or lettering has been printed later is also treated as an edge element ZF.

[0045] The feature amount F is an index based on numerical data that represents the state in which the edge element ZF is captured in the captured image of the sheet material Z captured by the imaging unit 1. Specifically, it represents the position, color, and orientation of the edge element ZF on the image, and more specifically, it represents how much the value of each data assigned to each pixel, such as brightness or RGB, changes between adjacent pixels or within a specified region of the image.

[0046] The index used for the feature quantity F changes depending on the state of the sheet material Z, even for the same sheet material Z. For example, the way light hits the sheet material Z changes depending on how it is rolled, and it is expected that even colored threads of the same color will have different RGB values ​​in the image, and similarly, different brightness values. In addition, even for plain sheet material Z, it is expected that the threads (especially the pile) will wear down with repeated use and washing, changing the brightness distribution (pattern) that appears in the image.

[0047] In order to avoid the above-mentioned fluctuations in the recognition of the feature F, it is preferable to configure the detection criteria for the feature F so as to be able to accommodate certain fluctuations. However, the detection criteria for the feature F may be configured on a rule-based basis, such as by setting predetermined parameters in advance, or may be configured by learning.

[0048] Hereinafter, a method for carrying out the present embodiment will be described in detail with reference to the drawings. The ...

[0049] <<Processing Method>> Each program P according to this embodiment stored in the storage unit 22 and a processing method X executed by the program P proceed according to the flowchart shown in FIG.

[0050] The image acquisition step S1 (corresponding to the image acquisition unit P1) includes an image acquisition step S11 in which the sheet material Z is imaged by the image acquisition unit 1.

[0051] In the imaging step S11, the control unit 21 controls the imaging unit 1 to capture a color image of the sheet material Z, reads the captured image data for use in the next detection step S2, and simultaneously stores the image data in the storage unit 22 (particularly the volatile memory 221) and displays it on the display unit 4. At this time, it is preferable that the control unit 21 also performs processing to recognize when a sheet material Z that has been placed casually (or transported by the conveying unit 5) comes to a standstill and automatically captures an image. With this processing, the user only needs to place the sheet material Z within the imaging range, preventing delays in system operation due to forgetting to operate the device.

[0052] The control unit 21 may simultaneously perform a process of trimming unnecessary portions of the captured image in accordance with the position of the sheet material Z in the image.

[0053] The detection process S2 (corresponding to the detection unit P2) includes a brightness recognition process S21, a determination process S22 for determining whether or not a feature F is present, an assignment process S23 for assigning a bounding box B to the recognized feature F, and a continuity identification process S24 for identifying the continuity of the feature F. The detection process S2 preferably also includes a distinction process S25 for distinguishing between a plurality of types of feature F, and a shape estimation process S26 for estimating the shape of the sheet material Z, particularly the type of side (long side, short side, etc.), based on the type of feature F.

[0054] In the brightness recognition step S21, the control unit 21 extracts brightness data from the image data that has been read in. In this process, the control unit 21 performs two types of extraction: macro extraction and micro extraction.

[0055] In macro extraction, the control unit 21 first divides the area of ​​the read image where the sheet material Z moves into areas of a predetermined size (for example, 10 pixels square), then calculates the average brightness of the pixels contained in each area, and then compares the average brightness values ​​between two adjacent areas.If the difference between the average values ​​is less than a predetermined value (for example, less than 20), it determines that the two predetermined areas belong to the same area of ​​the sheet material Z.

[0056] After the macro-extraction process described above, the control unit 21 executes the micro-extraction process. In this micro-extraction, the control unit 21 recognizes a pattern in which the brightness value fluctuates for each pixel shift in the areas determined to be within the same area in the macro-extraction. In other words, the control unit 21 recognizes the change in brightness between pixels.

[0057] This brightness recognition step S21 can be replaced with a color recognition process (color recognition step). In this case, the brightness data for each pixel used for detection is replaced with color, i.e., each RGB value, and the control unit 21 performs macro extraction or micro extraction. Furthermore, the brightness recognition step S21 may be a processing step that uses both brightness and color (RGB values).

[0058] In the determination step S22, the control unit 21 recognizes the change in the brightness pattern in one area of ​​the sheet material Z recognized in the brightness recognition step S21, or compares the recognized brightness pattern with data previously stored in the memory unit 22. If the control unit 21 then finds a location where the brightness change pattern changes significantly, it determines that this is a location where the stitching pattern changes and is a part of the feature value F where the weave pattern, etc. changes. This allows the processing system Y to treat elements contained in typical plain towels and sheets, such as the presence or absence of hems, selvedge edges, and pile, as feature values ​​F.

[0059] The brightness values ​​used in the brightness recognition step S21 and the determination step S22 utilize the principle that unevenness caused by stitching on the sheet material Z is recognized as brightness through light reflection. With this configuration, the control unit 21 can recognize the feature amount F even if the sheet material Z is plain.

[0060] In the brightness recognition process S21 and the judgment process S22, the control unit 21 mainly performs detection using numerical values ​​related to brightness, but if color (RGB numerical values) can be used, such as when the edge element ZF is a colored thread, it is preferable to detect the feature F using numerical values ​​related to color.

[0061] In the assignment step S23, the control unit 21 first assigns a segment to each pixel so as to match the recognized feature value F, and then assigns a rectangular bounding box B to the collection of the segments. Preferably, the control unit 21 displays the result of assigning the bounding box B on the display unit 4.

[0062] A bounding box B is an icon assigned to each section of the collection of assigned segments that is divided into sections of a predetermined length or less. In the bounding box B, the length of the sides facing the length direction (continuity direction) of the feature F (collection of segments) is arbitrary as long as it is equal to or less than the predetermined length, and all sides in the direction dividing the feature F have a constant length.

[0063] In the allocation step S23, the control unit 21 allocates a plurality of bounding boxes B according to the feature amount F, and then proceeds to the next step.

[0064] If there are multiple types of feature F, the control unit 21 combines different data, such as changing the color of the bounding box B (balance of each RGB value) for each group, to reflect the results of the distinction step S25 (described below).

[0065] In the continuity identification step S24, the control unit 21 links multiple bounding boxes B together when they intersect, contact, or are close to each other at their longitudinal ends (or at any position on the short sides) to identify the continuity of the feature F. By performing this process on the image, essentially, continuous edge elements ZF are recognized.

[0066] Here, "continuity" refers to an area where multiple pixels recognized as feature F are located adjacent to each other, or a group of multiple linked bounding boxes B. Note that if only feature F is handled and bounding boxes B are not used, the assignment step S23 and the continuity identification step S24 are omitted. Also, if there is only one bounding box B, the continuity identification step S24 is omitted.

[0067] The continuity identification step S24 may be a process in which the control unit 21 determines the proximity of segments, links the segments together, or groups of segments together, and as a result, connects the bounding boxes B together.

[0068] In the processing method X shown in Figure 1, the control unit 21 performs a distinction step S25 in which, after the processing of the continuity identification step S24, if it is determined that the feature F whose continuity has been identified can be divided into multiple groups or that there are multiple types, the control unit 21 groups the feature F.

[0069] Alternatively, in the distinguishing step S25, when it is determined that there are multiple types of patterns or distribution locations of feature F at the stage when the determining step S22 is completed, the control unit 21 may perform processing in the following order: grouping the feature F according to the types, as shown in Fig. 2. Furthermore, when there is one or less type of recognized feature F, the control unit 21 does not perform the distinguishing step S25 and proceeds to the next step.

[0070] In the shape estimation step S26, the control unit 21 estimates the position of the edge ZH of the sheet material Z based on the position, continuity, and length direction of the detected feature F, and then estimates the type of edge ZH that may exist near the edge element ZF corresponding to the feature F based on the state of the feature F. At this time, the control unit 21 particularly determines the long and short sides of the sheet material Z.

[0071] The shape estimation step S26 is preferably incorporated into the processing method X when edge elements ZF are provided along the edge ZH of the sheet material Z. For example, in a typical towel, edge elements ZF with different stitching patterns are provided on sides of different lengths, such as selvedge along the long side and a hem along the short side, and the control unit 21 distinguishes between these. This makes it easier for the manipulator 3 to grip the sheet material Z at an easy-to-handle position in the position determination step S3 and the gripping operation step S4, thereby reducing errors in the operation of conveying the sheet material Z.

[0072] The position determination process S3 (corresponding to the position determination unit P3) includes a priority assignment process S31 and a gripping position determination process S32, in which the control unit 21 determines the gripping position of the sheet material Z using the feature F identified in the detection process S2.

[0073] In a priority assignment step S31, the control unit 21 assigns a priority to each group of the bounding boxes B (or feature amounts F) grouped in the distinction step S25 to determine the gripping position. This priority is, for example, a numerical value, and is assigned in order from the most likely group to the most likely group for the gripping position, such as "1," "2," "3," .... Based on the assigned priorities, in a gripping position determination step S32, the gripping position is determined from the group with the smallest numerical priority value.

[0074] In the priority assignment step S31, the control unit 21 first compares the lengths of contiguous bounding boxes B (segments) that have been grouped, and assigns priority values ​​to the longest bounding boxes B in descending order.

[0075] Next, the control unit 21 identifies the coordinates (X and Y coordinates in the image) of the midpoint of each group and determines whether or not each coordinate is within a prohibited area (including the boundary). This prohibited area refers to an area where the end effector 31 interferes with something other than the sheet material Z and cannot grasp the sheet material Z, such as the edge of the conveying unit 5 or near the base of the manipulator 3. For groups whose midpoints are determined to be within the prohibited area, the control unit 21 lowers the priority assigned to them to the lowest (a value 1 or 2 greater than the maximum value at that time).

[0076] Regarding the relationship between the midpoint of a group and the prohibited area, it is possible that only the vicinity of the midpoint is located within the prohibited area, and the majority of the group is outside the prohibited area. For this reason, even if the vicinity of the midpoint of a group is within the prohibited area, if a certain amount or more of the length of the group (for example, 70% or more) is outside the prohibited area, the control unit 21 preferably does not lower the priority, but instead executes a process of re-determining the candidate grip position for that group to a point that is outside the prohibited area and closest to the midpoint.

[0077] The control unit 21 can also use information on whether the area around a group on the sheet material Z is raised as a criterion for adjusting the priority. Since the control unit 21 can estimate the vertical relationship between each area of ​​the sheet material Z in the brightness recognition step S21, if there is an area of ​​the sheet material Z that is higher than the grouped bounding box B near the bounding box B, the control unit 21 uses this information to lower the priority of that group. This prevents the sheet material Z from unfolding properly when the manipulator 3 grasps and lifts it, which can cause delays in the processes after this invention (including the placement process described below).

[0078] In order to recognize the shape of the sheet material Z at this time with higher accuracy, the processing system Y may be configured to capture images of the sheet material Z using multiple imaging units 1 and handle the shape of the sheet material Z using topography.

[0079] In the priority assignment step S31, after assigning priorities, the control unit 21 preferably performs a process of lowering the priority of a group of bounding boxes B (or, in other words, feature amounts F) that satisfy a predetermined condition. The "predetermined condition" here refers to, for example, when the feature amounts F in the group are located along the short sides of the sheet material Z, or when the length of the continuous bounding boxes B is equal to or less than a certain value. By lowering the priority assigned to these, it is possible to prevent a situation in which the sheet material Z sags along the long sides, causing the height at which the manipulator 3 lifts the sheet material Z to be higher than necessary, or a situation in which the sheet material Z does not unfold when lifted by the manipulator 3 due to grasping a portion of the edge element ZF with a short exposed length.

[0080] In the gripping position determination step S32, the control unit 21 determines either the detected feature amount F or the estimated edge portion ZH as the position where the sheet material Z is gripped by the end effector 31. For example, the control unit 21 selects one point of the feature amount F as the gripping position and determines its XY coordinates. At this time, it is preferable to use a method that can determine the gripping position based on a predetermined criterion, such as near the midpoint of the feature amount F or edge portion ZH for which continuity has been identified.

[0081] In the grasping position determination step S32, the control unit 21 determines the midpoint (or the re-set grasping position candidate) of the group with the smallest priority value as the grasping position based on the priority assigned to each group in the priority assignment step S31 and further adjusted.

[0082] The gripping operation step S4 (corresponding to the gripping operation unit P4) includes a gripping step S41, a lifting step S42, and a delivery step S43. In the gripping operation step S4, the control unit 21 operates (controls) the manipulator 3 to grip the sheet material Z and deliver it to the next step.

[0083] In the gripping step S41, the control unit 21 controls the manipulator 3 to grip the sheet material Z at the gripping position determined in the position determination step S3 with the end effector 31. However, because the gripping position determined in the position determination step S3 is merely an XY coordinate on the image, the control unit 21 converts the coordinate into a coordinate corresponding to the manipulator 3. The Z coordinate of the manipulator 3 is assumed to be a pre-specified value, or a value determined using a vertical height estimated from a captured image. The manipulator 3 grips the sheet material Z with the end effector 31 at the position specified above. Furthermore, the control unit 21 may adjust the orientation of the end effector 31 so that the gripper opening direction is perpendicular or parallel to the edge element ZF.

[0084] In the lifting step S42, the control unit 21 lifts the sheet material Z gripped by the end effector 31 to a predetermined height. This lifting height may be a predetermined height or may be changed as appropriate depending on the state of the sheet material Z. This makes it easier for the sides and corners of the sheet material Z to be exposed in a drooping state, creating a state in which it is easier to perform the next step (the operation of placing the sheet material Z in a state in which the corners are exposed).

[0085] In the transfer step S43, the control unit 21 transfers the lifted sheet material Z from the manipulator 3 that holds it in the present invention to the downstream manipulator that will execute the next step. At this time, it is preferable that the manipulator 3 have the downstream manipulator hold a portion of the sheet material Z close to the gripping position, and transfer the sheet material Z by hand. This allows the shape of the lifted sheet material Z to be maintained as much as possible.

[0086] The processing method X according to this embodiment may include a placing step, as the processing step following the gripping operation step S4, in which the delivered sheet material Z is placed with its edge ZH or corners exposed. This allows the folding manipulator provided immediately after the present invention to mechanically grip the corners of the sheet material Z, making it easy to fold. The placing step also requires a program P for the processing device 2 to execute the process, as well as a downstream manipulator and a movable table, and these are added to the present invention to make it executable.

[0087] The processing method X according to this embodiment may have the following modifications. However, the modifications shown below are merely examples, and the presence or absence of each modification is determined independently unless there is a particular dependency between them.

[0088] <<Modifications>> In the above explanation, the RGB color system is used to determine the color in a color image, but other color systems that can be quantified may be used, such as the CMYK color system, the XYZ color system, the Lab color system, etc. Even in these cases, it is preferable that the control unit 21 is configured to be able to deal with variations due to individual differences and wear of the sheet material Z as described above, and that the configuration and method be such that they can be used in the detection step S2.

[0089] In order to accommodate variations, it is preferable to use, for example, a three-point color value feature TF as the feature F. The three-point color value feature TF is merely one example of the feature F, and an index calculated by another method may also be used.

[0090] The three-point color value feature TF is calculated using the following formulas based on the maximum values ​​(MR, MG, MB, ML), minimum values ​​(mR, mG, mB, mL), and average values ​​(AR, AG, AB, AL) of the distribution of RGB values ​​and / or lightness (hereinafter referred to as the letter L) in the RGB color system (other color systems may be used) of each pixel in a predetermined region within the image: R component feature = (MR - mR) / AR G component feature = (MG - mG) / AG B component feature = (MB - mB) / AB L component feature = (ML - mL) / AL As described above, the accuracy of detecting the feature F can be maintained regardless of the shape of the sheet material Z, its positional relationship with the light source, or deterioration of the sheet material Z.

[0091] The processing device 2 and / or the detection unit P2 may have an abnormal placement notification unit. If the detection unit P2 (and the detection step S2) cannot detect the feature F due to the shape of the sheet material Z, this abnormal placement notification unit notifies the user of this fact and requests that the sheet material Z be repositioned so that the edge element ZF can be imaged. The notification may be displayed as a pop-up on the display unit 4, or sound from a separately provided speaker, for example.

[0092] The processing method X according to this embodiment may be configured such that in the image acquisition step S1, the sheet material Z is photographed with a plurality of cameras to acquire a topography of the sheet material Z, and the detection step S2 and position determination step S3 are performed using the topography. In this case, the steps up to the detection step S2 are as described above, but in the position determination step S3, particularly in the priority assignment step S31, the control unit 21 can use the "height" of the bounding box B based on the topography.

[0093] In this case, in the priority assignment step S31, the control unit 21 determines the order of priority for each group of bounding boxes B grouped in the distinction step S25, using the length, midpoint, and highest point of the group.

[0094] For example, the control unit 21 prioritizes groups whose lengths are equal to or greater than a predetermined value, and identifies the highest point within each group and determines its position relative to the midpoint. In response to this, the control unit 21 generally prioritizes the group with the highest highest point and assigns a priority so that the edge element ZF located at the highest position can be grasped. However, if a predetermined condition is met, such as when the distance between the highest point and the midpoint within a group is extremely large (i.e., the highest point is near the end of the group, and grasping the highest point may result in an inability to grasp the center of the edge element ZF), the control unit 21 assigns a priority using the midpoint instead of the highest point and executes a process for determining the grasping position.

[0095] The processing method X (processing system Y) according to this embodiment may include a learning step for improving the accuracy of the detection step S2 and the position determination step S3, instead of the processing step using a preset standard such as a three-point color value feature TF. In this learning step, for example, training data is loaded into the processing system Y, and the control unit 21 determines and optimizes parameters for detecting the feature F and identifying continuity. Specifically, in this learning step, pixels that are the feature F and continuity are identified by manual processing for an image of the sheet material Z, and the labeled image and a dataset of the original image are used as training data.

[0096] The learning steps described above are merely specific examples, and the processing method X (processing system Y) according to the present invention may have existing learning algorithms such as YOLO (registered trademark), YOLACT, and MASK-RCNN.

[0097] In addition to the above, the processing system Y may have a learning process in which, while processing the sheet material Z, the resulting data is fed back to improve the accuracy of the detection process S2 and the position determination process S3. In this case, the learning process is assumed to include a feature detection pattern learning process L1, a continuity identification pattern learning process L2, and a gripping position determination pattern learning process L3. Each learning method will be described in detail below.

[0098] The feature detection pattern learning step L1 is a learning step for detecting feature values ​​F appearing in an image, particularly for more accurately detecting portions corresponding to edge elements ZF in segment assignment (segmentation). In actual processing, individual differences in the appearance of feature values ​​F in an image may occur due to factors such as the way light hits the sheet material Z and wear and tear on the stitching of the sheet material Z, which may reduce the accuracy of segmentation. Therefore, in the feature detection pattern learning step L1, the control unit 21 combines a combination of brightness values ​​and RGB values ​​(preferably, three-point color value feature values ​​TF) for each pixel with the segmentation region to create an input data set, stores this in the memory unit 22, and associates it with the state of the sheet material Z when it is lifted in the subsequent lifting step S42. At this time, the state of the sheet material Z is imaged using a lifting state imaging unit, and a penalty (negative value) is associated with the data set if the vertical length of the sheet material Z is equal to or greater than a predetermined value, and a reward (positive value) is associated with the data set if the vertical length is less than the predetermined value. The control unit 21 repeats this process for each series of processes, and reproduces, through segmentation, a state of the data set that is close to the state when the reward is being obtained. This makes it possible to detect the feature amount F more accurately, regardless of individual differences in the sheet material Z.

[0099] The continuity identifying pattern learning step L2 is a learning step for more accurately identifying the continuity of the feature F in the continuity identifying step S24. For example, in FIG. 7, the third group bounding box B3 and the fourth group bounding box B4 are likely to be perceived by the user as being continuous. In addition, depending on how the sheet material Z is curled, an edge element ZF that is actually continuous may be blocked by an overlapping portion, and may be perceived as discontinuous in the image. To resolve this, the continuity identifying pattern learning step L2 learns such patterns and detects longer continuous feature values ​​F.

[0100] In the continuity identification pattern learning step L2, the control unit 21 stores the positions and orientations of the grouped bounding boxes B and information on the segments belonging to them in the memory unit 22, and further stores the data set together with the distribution of colors (brightness values, RGB values, etc.) in the image in the memory unit 22. The control unit 21 links this data set with the state of the lifted sheet material Z and the reward / punishment data similar to those described above. The control unit 21 repeats this process for each series of processes, and reproduces a state close to the data set when a reward is being obtained in the continuity identification step S24. This makes it possible to more accurately detect the continuity of the feature amount F regardless of individual differences in the sheet material Z, and to determine a more appropriate gripping position.

[0101] The gripping position determination pattern learning process L3 is a learning process for improving the probability of determining a more appropriate gripping position in the position determination process S3. In the position determination process S3, the control unit 21 determines a gripping position from the continuous feature values ​​F that meet a predetermined criterion. The control unit 21 associates the data used in the process, such as a dataset summarizing the coordinate data of the ends, midpoints, and positional relationships of each group of the bounding box B, with the state of the lifted sheet material Z and the reward / punishment data described above. The control unit 21 repeats this process for each series of processes, optimizing the gripping position determination process by, for example, varying the coordinate data so that the position determination process S3 is performed in a state close to the dataset when a reward is being obtained. This allows for a more appropriate gripping position to be determined regardless of individual differences in the sheet material Z.

[0102] When the sheet material Z used in the present invention is a plain towel or the like, as described above, the edge elements ZF and feature quantities F correspond to the hem, selvedge, and pile, which are detected by the control unit 21 in the detection step S2. In this case, the processes performed by the control unit 21 from the detection step S2 to the position determination step S3 can be categorized into cases as shown in the table in FIG.

[0103] 8 is applied to this embodiment, it is assumed that the part to be preferentially determined as the gripping position from among the hem, selvedge, and pile is set in advance in the position determination step S3. Note that in Fig. 8, a circle indicates a state in which each feature F was recognized, and an x ​​indicates a state in which each feature F was not recognized.

[0104] 8 indicates a case where the control unit 21 was unable to detect any of the hem, selvedge, or pile. In this case, the control unit 21 determines that subsequent processing is not possible, in other words, that the control unit 21 is unable to proceed to the position determination step S3. The control unit 21 then performs processing such as removing the sheet material Z from the line of the conveying unit 5, or notifying the user of an error and requesting that the sheet material Z be repositioned so that some feature value F can be detected. However, it is preferable that the control unit 21 be able to operate the manipulator 3 to change the curling state (positioning state) of the sheet material Z.

[0105] The second, third, and fifth rows of the table shown in FIG. 8 indicate cases where the control unit 21 detects any one of the hem, selvedge, and pile. In this case, the control unit 21 checks the detected selvedge against the portion to be prioritized as the gripping position in the position determination step S3. If the setting of the gripping position matches the type of the detected feature F, the control unit 21 determines the gripping position from the feature F. For example, if the portion to be prioritized as the gripping position is set to be selvedge and the feature F detected by the control unit 21 also relates to the selvedge, the control unit 21 determines the gripping position from the feature F related to the selvedge in the position determination step S3. If the setting of the gripping position differs from the type of the detected feature F, the control unit 21 determines the gripping position based on a different criterion from the area where the feature F was not detected. This criterion may be, for example, determining the widest identical area as the gripping position in the brightness recognition step S21.

[0106] The fourth, sixth, and seventh rows of the table shown in Fig. 8 indicate cases where the control unit 21 detects any two of the hem, selvedge, and pile. In this case, the control unit 21 checks the detected gripping position against the portion to be prioritized in the position determination step S3, as in the case described above. If the gripping position setting matches the type of detected feature F, the control unit 21 determines the gripping position from the feature F. If the gripping position setting and the type of detected feature F differ, the control unit 21 estimates the position of the remaining feature F, or an alternative gripping position, from the two detected feature F types, and determines the gripping position from there.

[0107] 8 shows a case where the control unit 21 detects three types of fabric: hem, selvedge, and pile. In this case, the control unit 21 determines the gripping position in accordance with the part that is set as the priority gripping position in the position determination step S3.

[0108] The processing method X (including the modified examples) described above is expressed and executed by a program P that can be processed by a computer.

[0109] Examples of the processing method X according to this embodiment will be described in detail below with reference to the drawings. Each of the following examples is an example of the processing of this embodiment, and the form and order of each step may be changed.

[0110] 6 shows an image of the sheet material Z captured by the imaging unit 1 and then trimmed by the control unit 21. At this time, the edge elements ZF of the sheet material Z are captured in the image as feature amounts F. In response to this, the control unit 21 detects the feature amounts F in the detection step S2. Since the edge elements ZF in this embodiment are colored yarns, a three-point color value feature amount TF using the RGB color system, etc., is used as described above.

[0111] In the assignment step S23, the control unit 21 assigns a bounding box B to the feature amount F detected as described above. If the control unit 21 has performed the processing of the distinction step S25, then in the assignment step S23, bounding boxes B of different groups, such as a first group bounding box B1 and a second group bounding box B2, are assigned.

[0112] Furthermore, in the continuity identifying step S24, the control unit 21 combines the bounding boxes B in each of the first group bounding box B1 and the second group bounding box B2, and identifies the continuity of each feature amount F.

[0113] 6, the first bounding box B1 and the second bounding box B2 are assigned to the feature amounts F (visual aspects of the edge elements ZF). These differences in aspect are detected by factors such as different colors (distribution of RGB values) of the feature amounts F, different positions (parallel and continuous, with no recognizable intersections or confluences), and different sewing patterns (one is sewn in a straight line while the other is sewn in a zigzag pattern, for example).

[0114] After the first group bounding box B1 and the second group bounding box B2 are recognized, in a shape estimation step S26, the control unit 21 estimates the long and short sides of the sheet material Z. Even if the processing of the shape estimation step S26 is not performed, the position of the edge ZH of the sheet material Z is estimated in the shape estimation step S26 in the position determination step S3. After the above processing, the control unit 21 determines the position of any of the feature amounts F or the position of the estimated edge ZH as the gripping position to be gripped by the manipulator 3.

[0115] In the priority assignment step S31, the control unit 21 assigns priorities to groups of bounding boxes B that should be used as gripping positions based on information on the edge ZH of the sheet material Z and its length estimated by the processing in the shape estimation step S26, and information on the position and state of the corresponding feature F. As described above, these priorities are numerical values. For example, the control unit 21 assigns a priority of "1" to the second group bounding box B2 and a priority of "2" to the first group bounding box B1 in Fig. 6. In this case, the control unit 21 estimates that the second group bounding box B2 is closer to the edge ZH of the sheet material Z than the first group bounding box B1, and therefore assigns a lower priority value to the former.

[0116] In the gripping position determination step S32, the control unit 21 determines a gripping position from the group of bounding boxes B with the smallest priority value, i.e., the second group bounding box B2 in the example of Fig. 6. The gripping position determined at this time may be selected from any position on the second group bounding box B2, or may be selected from a position a predetermined distance (e.g., 10 mm) away from the second group bounding box B2 toward the edge ZH of the sheet material Z.

[0117] Furthermore, in the gripping position determination step S32, the control unit 21 preferably determines the gripping position to be near the center of the second group bounding box B2, and more preferably, if a corner of the sheet material Z is recognized at the edge of the second group bounding box B2, the control unit 21 determines the gripping position to be a position shifted from the center of the second group bounding box B2 to the side opposite the corner. This method makes it easier for the end effector 31 to grip the edge of the sheet material Z near its midpoint, thereby reducing the height to which the manipulator 3 lifts the sheet material Z and minimizing the space required for operation. In addition, this reduces the possibility that the lifted and hanging sheet material Z will come into contact with and get caught on surrounding equipment, causing interference with operation.

[0118] <<Example 2>> Figure 7 shows a state in which a first group bounding box B1, a second group bounding box B2, a third group bounding box B3, and a fourth group bounding box B4 have been assigned to each feature amount F and the continuity of each has been determined. In particular, in the state shown in Figure 7, the ends of each group of the second group bounding box B2 to the fourth group bounding box B4 are close to each other, and the end appears to the user to be continuous. In such a case, it is preferable that the control unit 21 perform processing to determine the continuity between the groups on the condition that the ends of the groups are separated by a predetermined distance (number of pixels) or less. Alternatively, it is preferable that in the priority assignment step S31, the control unit 21 assigns consecutive priorities to the second group bounding box B2 to the fourth group bounding box B4.

[0119] However, in the above case, the control unit 21 assigns the lowest priority to the group in the center. That is, in the state of Fig. 7, the control unit 21 assigns a priority of "1" to the third group bounding box B3, a priority of "2" to the fourth group bounding box B4, and a priority of "3" to the second group bounding box B2. This method makes it easier for the control unit 21 to select a gripping position near the center of the continuous edge elements ZF in the position determination step S3.

[0120] Hereinafter, a processing method X according to a second embodiment of the present invention will be described in detail with reference to the drawings. However, the same reference numerals will be used to designate the same parts as those in the first embodiment, and the description thereof will be omitted.

[0121] 9, a processing system Y according to this embodiment includes two manipulators 3. Accordingly, in the position determination step S3, the control unit 21 determines two gripping positions, i.e., two sets of coordinates.

[0122] In the gripping position determination step S32 in this embodiment, the control unit 21 determines two gripping positions from a group with a high priority. At this time, the control unit 21 determines the gripping positions (sets of coordinates indicating points) so that the two gripping points are separated by a predetermined linear distance. Here, the "predetermined linear distance" is a length determined by the size of the end effector 31, the size of the gripper of the downstream manipulator that grips the sheet material Z in the next step of the present invention, the stretching length required in other subsequent steps, and the like.

[0123] In the gripping position determination step S32, the length of the group selected by priority may be shorter than the "predetermined linear distance" required to determine the gripping position. In this case, the control unit 21 first changes the group to be used to the next priority. If the gripping position cannot still be determined, the control unit 21 uses information regarding the shape of the sheet material Z obtained in the detection step S2 (particularly the brightness recognition step S21 and the shape estimation step S26) to determine a pair of points (e.g., two points) in an arbitrary group (outside the prohibited area) separated by a predetermined creepage distance (estimated values ​​are assumed, but actual measurements are preferred if possible) as the gripping position. This creepage distance is the distance between two points separated by a predetermined distance along the surface of the sheet.

[0124] When the number of manipulators 3 in the processing system Y according to this embodiment increases to three or more, the number of determined gripping positions also increases corresponding to the number of manipulators 3. The control unit 21 also executes the process of determining each point (set of coordinates) of the gripping position in the gripping position determination step S32 in the same manner as the process described above.

[0125] X Processing method Y Processing system 1 Imaging unit 2 Processing device 3 Manipulator 4 Display unit 5 Conveying unit P Program P1 Image acquisition unit P2 Feature amount detection unit P3 Position determination unit P4 Gripping operation unit S1 Image acquisition process S2 Detection process S3 Position determination process S4 Gripping operation process F Feature amount B Bounding box Z Sheet material ZF Edge element ZH Edge

Claims

1. A processing method for a computer to execute a process for determining an appropriate gripping position for sheet material in any state, the processing method including an image acquisition process for capturing an image of the sheet material to obtain image data, a detection process for detecting features contained in the image data, and a position determination process for determining the gripping position using the features.

2. The processing method according to claim 1, wherein the detection step uses the feature amount to determine segments of edge elements of the sheet material, and the position determination step determines the gripping position based on the segments.

3. The processing method of claim 2, wherein the edge elements are provided along the edge of the sheet material, the detection step determines the continuity of a plurality of the segments and divides the plurality of segments determined to have the continuity into groups, and the position determination step determines the gripping position using the elements of the group.

4. The processing method according to claim 2, wherein the sheet material has edge elements that can be distinguished for each edge, the detection process distinguishes and groups each of the edge elements based on the features, and the position determination process determines a point that meets predetermined conditions as the gripping position based on the features corresponding to each of the groups.

5. The processing method according to claim 3, wherein, when determining multiple gripping positions, the position determination step determines, as the gripping positions, a set of points that are spaced apart from each other by a predetermined straight-line distance in the segment that has been determined to have continuity.

6. The processing method of claim 3, wherein, when determining multiple gripping positions, the position determination step determines, as the gripping positions, a pair of points that are spaced apart from each other by a predetermined creepage distance in the segment that has been determined to have continuity.

7. A program that causes a computer to function as an image acquisition unit that captures an image of a sheet material to acquire image data, a detection unit that detects the feature amounts contained in the image data, and a position determination unit that uses the feature amounts to determine a gripping position for the sheet material, and that determines an appropriate gripping position for the sheet material in any state.

8. A processing system in which a computer executes a process to determine an appropriate gripping position for sheet material in any state, the processing system including: an image acquisition unit that images the sheet material to acquire image data; a detection unit that detects the feature amounts contained in the image data; and a position determination unit that determines the gripping position using the feature amounts.

Images