Sheet material processing method, processing system, and program

Abstract

To provide a processing method capable of determining an appropriate grip position for exposing a feature amount of a sheet material in a pre-step of work, so that work can be finally performed with an angle exposed even in a state where a large object is casually placed.SOLUTION: A processing method X for performing, by a computer, processing of determining an appropriate grip position in a sheet material Z in any state, includes: an image acquisition step S1 of imaging the sheet material Z to acquire image data; a detection step S2 of detecting a feature amount F included in the image data; and a position determination step S3 of determining the grip position by using the feature amount F.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a processing method for recognizing the shape of a sheet material, such as a sheet, placed haphazardly and detecting an appropriate portion to grasp before the sheet material is automatically folded, and a processing system for executing this processing. [Background technology]

[0002] In the linen supply industry and accommodation facilities such as hotels, large quantities of sheet materials are handled for changing and washing sheets. In particular, neatly folding sheet materials such as sheets and bath towels, which are the same size as or larger than the human body, requires considerable effort. Therefore, it is desirable to automate the folding of sheet materials, and the applicant of the present application has developed a folding device for this purpose.

[0003] On the other hand, in the process of feeding sheet materials such as sheets into a folding device, it is necessary to align the shape and orientation of the sheet materials to a certain extent. This task has been performed by manually opening the sheet materials or identifying the type of sheet material, so there has been a demand for automation of this process. To address this issue, devices have been developed that utilize distinctive features of the sheet materials, such as the corners or patterns, as in Patent Documents 1 and 2. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 9,909,252 [Patent Document 2] Japanese Patent Publication No. 2023-011021 Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 discloses a system that recognizes and grasps the four corners of a small sheet material such as a towel. However, it is difficult to recognize the four corners of a large object such as a sheet or bath towel when it is casually rolled up. For this reason, it is necessary to automate the process of reliably exposing the corners and mechanically grasping the object, regardless of the size of the sheet material.

[0006] Furthermore, Patent Document 2 discloses a system that determines the pattern of laundry by image recognition. However, in order to recognize the pattern, the object needs to be spread out and placed on the laundry, and relying on human intervention for this process alone is rather inefficient.

[0007] The present invention has been made in consideration of the above-described situation, and provides a processing method that can determine an appropriate gripping position to expose the characteristics of a sheet material in a preliminary stage, so that even if a large object is placed carelessly, the corners can ultimately be exposed and work can be performed on it. [Means for solving the problem]

[0008] The present invention, which solves the above-mentioned problems, is a processing method for a computer to execute a process for determining an appropriate gripping position for sheet material in any state, and includes an image acquisition process for capturing an image of the sheet material to obtain image data, a detection process for detecting features contained in the image data, and a position determination process for determining the gripping position using the features.

[0009] According to the present invention, a large sheet material can be gripped at an appropriate gripping position without manual intervention and handed over to the next process for exposing the corners of the sheet material.

[0010] In a preferred embodiment of the present invention, the detecting step uses the feature amount to determine a segment of an edge element provided on the sheet material, and the position determining step determines the gripping position based on the segment.

[0011] With this configuration, the vicinity of the edge of the sheet material can be grasped using image recognition.

[0012] In a preferred form of the invention, the edge elements are provided along the edge of the sheet material, the detection step determines the continuity of a plurality of the segments and divides the plurality of segments determined to have the continuity into groups, and the position determination step determines the gripping position using elements of the groups.

[0013] With this configuration, even when the edge elements are visually recognized as curved, they can be recognized with high accuracy.

[0014] In a preferred embodiment of the present invention, the sheet material has edge elements that can be distinguished for each edge, and the detection process distinguishes and groups each of the edge elements based on the feature values, and the position determination process determines a point that meets predetermined conditions as the gripping position based on the feature values ​​corresponding to each of the groups.

[0015] With this configuration, the sheet material can be gripped as close to the long side as possible, and the sheet material can be easily lifted.

[0016] In a preferred embodiment of the present invention, when determining multiple gripping positions, the position determination step determines a set of points that are separated from each other by a predetermined linear distance within the segment that is determined to have continuity as the gripping positions.

[0017] With this configuration, even when the sheet material is gripped at a plurality of positions, it can be gripped and lifted at an appropriate gripping position.

[0018] In a preferred embodiment of the present invention, when the position determination step determines multiple gripping positions, a set of points that are separated from each other by a predetermined creepage distance within the segment that is determined to have continuity is determined as the gripping positions.

[0019] With this configuration, even when the sheet material is gripped at a plurality of positions, it can be gripped and lifted at a gripping position that is more appropriate for the shape of the sheet material.

[0020] The present invention, which solves the above-mentioned problems, is a program that causes a computer to function as an image acquisition unit that captures an image of a sheet material to acquire image data, a detection unit that detects the features contained in the image data, and a position determination unit that uses the features to determine the gripping position of the sheet material, thereby determining the appropriate gripping position for the sheet material in any state.

[0021] With this configuration, the corners of a large sheet material can be exposed without human intervention by gripping the sheet material at an appropriate gripping position and transferring it to the next process.

[0022] The present invention, which solves the above-mentioned problems, is a processing system in which a computer executes a process to determine an appropriate gripping position for sheet material in any state, and includes an image acquisition unit that captures an image of the sheet material to acquire image data, a detection unit that detects the feature amounts contained in the image data, and a position determination unit that determines the gripping position using the feature amounts.

[0023] With this configuration, the corners of a large sheet material can be exposed without human intervention by gripping the sheet material at an appropriate gripping position and transferring it to the next process. [Effects of the Invention]

[0024] According to the present invention, it is possible to provide a processing method that can recognize features and determine a more appropriate gripping position even when a large object is placed haphazardly. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a flowchart of a processing method according to a first embodiment of the present invention. [Figure 2] 1 is a flowchart of a processing method according to a first embodiment of the present invention. [Figure 3] FIG. 1 is a block diagram of hardware for executing a processing method according to a first embodiment of the present invention. [Figure 4] FIG. 1 is a schematic diagram of hardware for executing a processing method according to a first embodiment of the present invention. [Figure 5] FIG. 1 is a schematic diagram of hardware for executing a processing method according to a first embodiment of the present invention. [Figure 6] 10 is an image showing an image recognition state of feature amounts in the first embodiment of the present invention. [Figure 7] 10 is an image showing an image recognition state of feature amounts in the first embodiment of the present invention. [Figure 8] 1 is a table showing detection patterns of feature amounts and corresponding processing cases in the first embodiment of the present invention. [Figure 9] FIG. 10 is a schematic diagram of hardware for executing a processing method according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0026] The processing method X according to each embodiment of the present invention will be described below with reference to the drawings. The description will be made in detail in the order of the configuration of the embodiment, the method of implementation, and other examples. It should be noted that the following embodiments are merely examples of the present invention, and the present invention is not limited to the following embodiments.

[0027] <Embodiment 1> 1, the processing method X according to this embodiment includes an image acquisition step S1, a detection step S2, and a position determination step S3, and further includes a gripping operation step S4 in which the sheet material Z is gripped at the gripping position of the sheet material Z determined in the position determination step S3, lifted up, and handed over to the next system. The processing method X is also executed by a processing system Y shown in FIG.

[0028] <<Processing System>> As shown in Figure 4, the processing system Y is a system of devices that executes a processing method X to grasp a sheet material Z at an appropriate position and hand it over to a downstream process, and is equipped with an imaging unit 1, a processing device 2, a manipulator 3, a display unit 4 (not shown), and a conveying unit 5.

[0029] The imaging unit 1 is a camera (line camera or area camera) that captures an image of the target sheet material Z with information on brightness, saturation, and hue, and is preferably configured to include a light source unit for more reliable image recognition.

[0030] 3, the processing device 2 is a computer device including a control unit 21 and a storage unit 22, and also includes an input device, a storage device, an interface, etc. The processing system Y uses this processing device 2 to control the imaging unit 1, the manipulator 3, the display unit 4, and the transport unit 5, as well as to execute various processes related to image recognition.

[0031] The control unit 21 is an arithmetic unit such as a CPU (processor) or a microcomputer, and executes computer-executable instructions.

[0032] The storage unit 22 includes a nonvolatile memory that stores a program P that executes the processing method X according to this embodiment, and a volatile memory 221 that temporarily stores image data captured by the imaging unit 1, and the data and program P are read from each memory and executed by the control unit 21. However, as shown in Fig. 3, the storage unit 22 does not need to have a clearly separated nonvolatile memory and volatile memory 221, and may be provided as an integrated memory as long as they can fulfill the above-mentioned roles.

[0033] A program P for executing each process according to this embodiment is stored in the storage unit 22. The program P causes the processing device 2, particularly the control unit 21, to function as an image acquisition unit P1, a detection unit P2, a position determination unit P3, and a grip operation unit P4.

[0034] The image acquisition unit P1 controls the imaging unit 1 to capture an image of the sheet material Z, and captures the captured image so that it can be handled as image data by the control unit 21.

[0035] The detection unit P2 includes a brightness recognition unit P21 that detects the unevenness of the surface of the sheet material Z that appears in the image data using brightness; a judgment unit P22 that determines the presence or absence of a feature F using the distribution of brightness in a specific area within the image data; an assignment unit P23 that assigns a bounding box B to the feature F; a continuity identification unit P24 that connects the bounding boxes B to identify the continuity of the feature F; a distinction unit P25 that groups feature F of different positions and forms; and a shape estimation unit P26 that estimates the position of the edge ZH based on the form of the detected feature F, and further estimates the type of the edge ZH of the sheet material Z depending on the detection status of the feature F.

[0036] The position determination unit P3 has a priority assignment unit P31 that assigns a priority to each group of feature values ​​F for selection as a gripping position, and a temporary selection unit P32 that temporarily selects a candidate gripping position from the feature values ​​F whose continuity has been identified.

[0037] The gripping operation unit P4 controls the manipulator 3 to grip and lift the sheet material Z (object) at the determined gripping position, and deliver it to the next process.

[0038] Below, the behavior of each part of the program P when it executes processing will be described in detail in the explanation of the corresponding step.

[0039] The manipulator 3 is a so-called robot arm that has an end effector 31 capable of grasping the sheet material Z, lifts up and transports the sheet material Z, and is preferably configured to operate with six degrees of freedom (for example, a configuration having axis joints at four locations).Furthermore, the end effector 31 is preferably configured to have a suction portion that sucks the sheet material Z and a pair of grippers that clamp the vicinity of the suction portion in order to reliably maintain the state in which the sheet material Z is grasped.

[0040] The display unit 4 is a display that displays the captured image data and the progress of image recognition, and is made of a liquid crystal screen or the like. This allows the user to check the status of image recognition of the sheet material Z, and if the placement of the sheet material Z is not appropriate, the user can visually confirm this and make corrections such as repositioning the sheet material Z. The display unit 4 may also be configured as a touch panel that doubles as an input device.

[0041] The transport unit 5 is a movable platform on which a user places the sheet material Z and transports the sheet material Z into the imaging range of the imaging unit 1, and is assumed to be in the form of a belt conveyor or the like. In addition, in order to make the sheet material Z easier to recognize, the transport unit 5 is preferably made of a color that is significantly different from the sheet material Z, particularly a dark color such as black.

[0042] The sheet material Z is a cloth material having edge elements ZF used for image recognition, and in this embodiment, it is assumed to be a cloth material that is too large to fit in one hand, such as a face towel, a bath towel, or a sheet used for bedding. However, relatively small items such as hand towels and handkerchiefs may also be applied to this embodiment. In the following description, for convenience of explanation, the cloth material shown in the image may also be referred to as the sheet material Z, but in this case, it refers to the data on the image corresponding to the sheet material Z.

[0043] The edge element ZF is provided along the edge ZH of the sheet material Z and serves as a mark (or clue) for identifying the edge ZH in the processing method X according to this embodiment. The actual edge element ZF is a region where, for example, the color thread, pattern, weave pattern, presence or absence of pile, etc. changes, i.e., a region where the thread (sewing) pattern changes near the edge ZH of the sheet material Z.

[0044] More specifically, it refers to the parts of the sheet material Z where the edge ZH can be seen due to changes in the stitching pattern, such as the hems and selvedge edges of a typical towel or sheet. It is also preferable that woven patterns that appear due to changes in the weave, flat parts of the towel (areas without pile), and parts with patterns or letters printed on them afterwards are also treated as edge elements ZF.

[0045] The feature amount F is an index based on numerical data that represents the state of the edge element ZF in the captured image when the sheet material Z is imaged by the imaging unit 1. Specifically, it represents the position, color, and orientation of the edge element ZF on the image, and more specifically, it represents how much the value of each data assigned to each pixel, such as brightness or RGB, changes between adjacent pixels or within a specified area of ​​the image.

[0046] The index used for the feature quantity F changes depending on the condition of the sheet material Z, even for the same sheet material Z. For example, the way light hits the sheet material Z changes depending on how it is rolled, and it is expected that even threads of the same color will have different RGB values ​​in the image, and similarly, different brightness values. Additionally, even for plain sheet material Z, repeated use and washing can wear down the threads (especially the pile), changing the brightness distribution (pattern) that appears in the image.

[0047] In order to avoid the above-mentioned fluctuations in the recognition of the feature F, it is preferable to configure the detection criteria for the feature F so as to be able to cope with certain fluctuations. However, the detection criteria for the feature F may be configured on a rule basis, for example, by setting predetermined parameters in advance, or may be formed by learning.

[0048] Hereinafter, a method for carrying out the present embodiment will be described in detail with reference to the drawings. The ... as an example, and the method for carrying out the present embodiment is not limited to this example, and the order of the steps may be changed.

[0049] <<Processing method>> Each program P according to this embodiment stored in the storage unit 22 and a processing method X executed by the program P proceed according to the flowchart shown in FIG.

[0050] The image acquiring step S1 (corresponding to the image acquiring section P1) includes an imaging step S11 in which the sheet material Z is imaged by the imaging section 1.

[0051] In the imaging step S11, the control unit 21 controls the imaging unit 1 to capture a color image of the sheet material Z, reads the captured image data for use in the next detection step S2, and simultaneously stores it in the memory unit 22 (particularly the volatile memory 221) and displays it on the display unit 4. At this time, it is preferable that the control unit 21 also performs processing to recognize the timing when the sheet material Z, which has been placed casually (or carried by the conveying unit 5), comes to a standstill and automatically captures an image. With this processing, the user only needs to place the sheet material Z within the imaging range, preventing delays in system operation due to forgetting to operate the device.

[0052] The control unit 21 may simultaneously perform a process of trimming unnecessary portions of the captured image in accordance with the position of the sheet material Z in the image.

[0053] The detection process S2 (corresponding to the detection unit P2) includes a brightness recognition process S21, a determination process S22 for determining whether or not a feature F is present, an assignment process S23 for assigning a bounding box B to the recognized feature F, and a continuity identification process S24 for identifying the continuity of the feature F. The detection process S2 preferably also includes a distinction process S25 for distinguishing between multiple types of feature F, and a shape estimation process S26 for estimating the shape of the sheet material Z, particularly the type of side (long side, short side, etc.), based on the type of feature F.

[0054] In the brightness recognition step S21, the control unit 21 extracts brightness data from the image data that has been read in. In this process, the control unit 21 performs two types of extraction: macro extraction and micro extraction.

[0055] In macro extraction, the control unit 21 first divides the area of ​​the read image where the sheet material Z moves into areas of a predetermined size (for example, 10 pixels square), then calculates the average brightness of the pixels contained in each area, and then compares the average brightness values ​​between two adjacent areas.If the difference between the average values ​​is less than a predetermined value (for example, less than 20), it determines that the two predetermined areas belong to the same area of ​​the sheet material Z.

[0056] After the macro-extraction process described above, the control unit 21 executes the micro-extraction process. In this micro-extraction, the control unit 21 recognizes a pattern in which the brightness value fluctuates for each pixel shift in the areas determined to be within the same area in the macro-extraction. In other words, the control unit 21 recognizes the change in brightness between pixels.

[0057] This brightness recognition step S21 can be replaced with a color recognition process (color recognition step). In this case, the brightness data for each pixel used for detection is replaced with color, i.e., each RGB value, and the control unit 21 performs macro extraction or micro extraction. Furthermore, the brightness recognition step S21 may be a processing step that uses both brightness and color (RGB values).

[0058] In the determination step S22, the control unit 21 recognizes the change in the brightness pattern in one area of ​​the sheet material Z recognized in the brightness recognition step S21, or compares the recognized brightness pattern with data previously stored in the memory unit 22. If the control unit 21 then finds a location where the pattern of brightness changes significantly, it determines that this is a location where the sewing pattern changes, and is a portion of the feature amount F where the weave pattern or the like changes. This allows the processing system Y to treat elements contained in typical plain towels and sheets, such as the presence or absence of hems, selvedge edges, and pile, as feature quantities F.

[0059] The brightness values ​​used in the brightness recognition step S21 and the determination step S22 utilize the principle that unevenness caused by stitching on the sheet material Z is recognized as brightness through light reflection. With this configuration, the control unit 21 can recognize the feature amount F even if the sheet material Z is plain.

[0060] In the brightness recognition process S21 and the judgment process S22, the control unit 21 mainly performs detection using numerical values ​​related to brightness, but if color (RGB numerical values) can be used, such as when the edge element ZF is a colored thread, it is preferable to detect the feature F using numerical values ​​related to color.

[0061] In the assignment step S23, the control unit 21 first assigns a segment to each pixel so as to match the recognized feature value F, and then assigns a rectangular bounding box B to the collection of the segments. Preferably, the control unit 21 displays the result of assigning the bounding box B on the display unit 4.

[0062] Bounding box B is an icon assigned to each section of the collection of assigned segments that is divided into sections of a predetermined length or less. In bounding box B, the length of the sides facing the length direction (continuous direction) of feature F (collection of segments) can be any length as long as it is equal to or less than the predetermined length, and all sides in the direction dividing feature F have a constant length.

[0063] In the allocation step S23, the control unit 21 allocates a plurality of bounding boxes B according to the feature amount F, and then proceeds to the next step.

[0064] When there are multiple types of feature amounts F, the control unit 21 combines different data, such as changing the color (balance of each RGB value) of the bounding box B for each group, to reflect the results of the distinction step S25 (described later).

[0065] In the continuity identification step S24, the control unit 21 links multiple bounding boxes B together when they intersect, contact, or are close to each other at their longitudinal ends (or at any position on the short sides) to identify the continuity of the feature value F. By performing this process on the image, essentially, continuous edge elements ZF are recognized.

[0066] Here, "continuity" refers to an area where multiple pixels recognized as feature F are located adjacent to each other, or a group of multiple bounding boxes B that are linked together. Note that the allocation step S23 and the continuity identifying step S24 are omitted when only the feature amount F is handled and the bounding box B is not used. Also, when there is one bounding box B, the continuity identifying step S24 is omitted.

[0067] The continuity identification step S24 may be a process in which the control unit 21 determines the proximity of segments, links the segments together, or groups of segments together, and as a result, connects the bounding boxes B together.

[0068] In the processing method X shown in FIG. 1, the control unit 21 performs a distinction step S25 in which, after the processing of the continuity identification step S24, if it is determined that the feature F whose continuity has been identified can be divided into multiple groups or that there are multiple types, the control unit 21 groups the feature F.

[0069] Alternatively, as shown in FIG. 2, in the distinction step S25, when it is determined that there are multiple types of patterns or distribution locations of the feature F at the stage when the determination step S22 is completed, the control unit 21 may perform processing in the following order: group the feature F according to the types. If the number of recognized feature amounts F is one or less, the control unit 21 does not perform the processing of the distinction step S25 and proceeds to the next step.

[0070] In the shape estimation step S26, the control unit 21 estimates the position of the edge ZH of the sheet material Z based on the position, continuity, and length direction of the detected feature F, and then estimates the type of edge ZH that may exist near the edge element ZF corresponding to the feature F based on the state of the feature F. At this time, the control unit 21 particularly determines the long and short sides of the sheet material Z.

[0071] The shape estimation step S26 is preferably incorporated into the processing method X when edge elements ZF are provided along the edge ZH of the sheet material Z. For example, in a typical towel, edge elements ZF with different stitching patterns are provided on sides of different lengths, such as selvedge along the long side and a hem along the short side, and the control unit 21 distinguishes between these. This makes it easier for the manipulator 3 to grip the sheet material Z at an easy-to-handle position in the position determination step S3 and the gripping operation step S4, thereby reducing errors in the work of conveying the sheet material Z.

[0072] The position determination process S3 (corresponding to the position determination unit P3) includes a priority assignment process S31 and a gripping position determination process S32, in which the control unit 21 determines the gripping position of the sheet material Z using the feature F identified in the detection process S2.

[0073] In a priority assignment step S31, the control unit 21 assigns a priority to each group of the bounding boxes B (or feature amounts F) grouped in the distinction step S25 to determine the gripping position. This priority is, for example, a numerical value, and is assigned in order from the most likely group to be used as the gripping position, such as "1," "2," "3," .... Based on the assigned priorities, in a gripping position determination step S32, the gripping position is determined starting from the group with the smallest numerical priority value.

[0074] In the priority assignment step S31, the control unit 21 first compares the lengths of contiguous grouped bounding boxes B (segments), and assigns priority values ​​to the longest bounding boxes B in descending order.

[0075] Next, the control unit 21 identifies the coordinates (X and Y coordinates in the image) of the midpoint of each group and determines whether or not each coordinate is within a prohibited area (including the boundary). This prohibited area refers to an area where the end effector 31 interferes with something other than the sheet material Z and cannot grasp the sheet material Z, such as the edge of the conveying unit 5 or near the base of the manipulator 3. For groups whose midpoints are determined to be within the prohibited area, the control unit 21 lowers the priority assigned to them to the lowest (a value 1 or 2 greater than the maximum value at that time).

[0076] Regarding the relationship between the midpoint of a group and the prohibited area, it is possible that only the vicinity of the midpoint is located within the prohibited area, and the majority of the group is outside the prohibited area. Therefore, even if the vicinity of the midpoint of a group is within the prohibited area, if a certain amount or more (for example, 70% or more) of the length of the group is outside the prohibited area, it is preferable that the control unit 21 does not lower the priority, but instead executes a process of re-determining the candidate grip position of that group to a point that is outside the prohibited area and closest to the midpoint.

[0077] The control unit 21 can also use information on whether the periphery of the group on the sheet material Z is raised or not as a criterion for determining the priority adjustment. At the brightness recognition step S21, the control unit 21 can estimate the vertical relationship between each area of ​​the sheet material Z, and using this, if there is an area of ​​the sheet material Z that is higher than the grouped bounding box B in the vicinity of the grouped bounding box B, the priority of that group is lowered. This prevents the sheet material Z from unfolding properly when it is grasped and lifted by the manipulator 3, which can cause delays in the processes after the present invention (including the placement process described later).

[0078] In order to recognize the shape of the sheet material Z at this time with higher accuracy, the processing system Y may be configured to capture images of the sheet material Z using a plurality of imaging units 1 and handle the shape of the sheet material Z using topography.

[0079] In the priority assignment step S31, after assigning priorities, the control unit 21 preferably performs processing to lower the priorities of groups of bounding boxes B (that is, feature amounts F) that satisfy a predetermined condition. The "predetermined condition" here refers to, for example, when the feature F in the group is located along the short side of the sheet material Z, or when the continuous length of the bounding box B is equal to or less than a certain value. By lowering the priority assigned to these, it is possible to prevent situations in which the sheet material Z sags along the long side and the height lifted by the manipulator 3 becomes higher than necessary, or when the sheet material Z does not spread when lifted by the manipulator 3 because the part of the edge element ZF with a short exposed length is grasped.

[0080] In the gripping position determination step S32, the control unit 21 determines either the detected feature F or the estimated edge ZH as the position where the sheet material Z is gripped by the end effector 31. For example, the control unit 21 selects one point of the feature F as the gripping position and determines its XY coordinates. At this time, it is preferable to use a method that can determine the gripping position based on a predetermined criterion, such as near the midpoint of the feature F or edge ZH whose continuity has been identified.

[0081] In the gripping position determination step S32, the control unit 21 determines the midpoint (or the re-set gripping position candidate) of the group with the smallest priority value as the gripping position based on the priority assigned to each group in the priority assignment step S31 and further adjusted.

[0082] The gripping operation step S4 (corresponding to the gripping operation unit P4) includes a gripping step S41, a lifting step S42, and a delivery step S43. In the gripping operation step S4, the control unit 21 operates (controls) the manipulator 3 to grip the sheet material Z and deliver it to the next step.

[0083] In the gripping step S41, the control unit 21 controls the manipulator 3 to grip the sheet material Z at the gripping position determined in the position determination step S3 with the end effector 31. However, since the gripping position determined in the position determination step S3 is merely an XY coordinate on the image, the control unit 21 converts the coordinate into a coordinate corresponding to the manipulator 3. The Z coordinate of the manipulator 3 may be a pre-specified value or a value determined using a vertical height estimated from a captured image. The manipulator 3 grips the sheet material Z with the end effector 31 at the position specified above. Furthermore, the control unit 21 may adjust the orientation of the end effector 31 so that the gripper opening direction is perpendicular or parallel to the edge element ZF.

[0084] In the lifting step S42, the control unit 21 lifts the sheet material Z gripped by the end effector 31 to a predetermined height. This lifting height may be a height specified in advance, or may be a value that is changed as appropriate depending on the state of the sheet material Z. This makes it easier for the sides and corners of the sheet material Z to be exposed in a drooping state, creating a state in which it is easier to perform the next step (the work of placing the sheet material Z in a state in which the corners are exposed).

[0085] In the delivery step S43, the control unit 21 delivers the lifted sheet material Z from the manipulator 3 that holds it in the present invention to the downstream manipulator that will execute the next step. At this time, it is preferable that the manipulator 3 have the downstream manipulator hold a portion of the sheet material Z close to the gripping position, and deliver it by hand. This allows the shape of the lifted sheet material Z to be maintained as much as possible.

[0086] The processing method X according to this embodiment may include a placing step, as the processing step following the gripping operation step S4, in which the delivered sheet material Z is placed with its edge ZH or corners exposed. This allows the folding manipulator provided immediately after the present invention to mechanically grip the corners of the sheet material Z, making it easy to perform the folding operation. The placing step also requires a program P for the processing device 2 to execute the process, as well as a downstream manipulator and a movable placing table, and these are added to the present invention to make it executable.

[0087] The processing method X according to this embodiment may have the following modifications. However, the modifications shown below are merely examples, and the presence or absence of each modification is determined independently unless there is a particular dependency between them.

[0088] <<Example of changes>> In the above explanation, the RGB color system is used to determine the color in a color image, but other color systems that can be quantified may be used, such as the CMYK color system, the XYZ color system, the Lab color system, etc. Even in these cases, it is preferable that the control unit 21 is configured to be able to deal with variations due to individual differences and wear of the sheet material Z as described above, and that the configuration and method be such that they can be used in the detection step S2.

[0089] To accommodate variations, it is preferable to use, for example, a three-point color value feature TF as the feature F. The three-point color value feature TF is merely one example of the feature F, and an index calculated by another method may also be used.

[0090] The three-point color value feature TF is calculated using the following formula based on the maximum value (MR, MG, MB, ML), minimum value (mR, mG, mB, mL), and average value (AR, AG, AB, AL) of the distribution of each RGB value and / or lightness (hereinafter referred to as the letter L) in the RGB color system (other color systems may be used) of each pixel in a specified area within an image. R component feature amount = (MR−mR) / AR G component feature value = (MG - mG) / AG B component feature value = (MB-mB) / AB L component feature amount = (ML-mL) / AL As a result, the accuracy of detecting the feature amount F can be maintained regardless of the shape of the sheet material Z, the positional relationship with the light source unit, or deterioration of the sheet material Z.

[0091] The processing device 2 and / or the detection unit P2 may have an abnormal placement notification unit. If the detection unit P2 (and the detection step S2) is unable to detect the feature F due to the shape of the sheet material Z, this abnormal placement notification unit notifies the user of this fact and requests that the sheet material Z be repositioned so that the edge element ZF can be imaged. The notification may take the form of a pop-up display on the display unit 4, a sound from a separately provided speaker, or the like.

[0092] The processing method X according to this embodiment may be configured such that in the image acquisition step S1, the sheet material Z is photographed with multiple cameras, a topography of the sheet material Z is acquired, and the topography is used to perform the detection step S2 and the position determination step S3. In this case, the detection step S2 is as described above, but in the position determination step S3, particularly in the priority assignment step S31, the control unit 21 can use the "height" of the bounding box B based on the topography.

[0093] In this case, in the priority assignment step S31, the control unit 21 determines the order of priority for each group of bounding boxes B grouped in the distinction step S25, using the length, midpoint, and highest point of the group.

[0094] For example, the control unit 21 increases the priority of groups whose lengths are equal to or greater than a preset value, and as for the highest point, identifies the highest point within each group and further determines its positional relationship with the midpoint. In response to this, the control unit 21 in principle increases the priority of the group with the highest highest point, and assigns a priority so that the edge element ZF at the highest position can be gripped. However, when certain conditions are met, such as when the distance between the highest point and the midpoint within a group is extremely large (the highest point is near the end of the group, and grasping the highest point may result in not being able to grasp the center of the edge element ZF), the control unit 21 assigns priority using the midpoint instead of the highest point and performs a process to determine the grasping position.

[0095] The processing method X (processing system Y) according to this embodiment may have a learning process for improving the accuracy of the detection process S2 and the position determination process S3, instead of a processing process using a preset standard such as a three-point color value feature TF. In this learning process, for example, training data is read into the processing system Y, and the control unit 21 determines and optimizes parameters for detecting the feature amount F and identifying continuity. Specifically, in this learning process, pixels that are the feature amount F and continuity are identified by human processing for an image of the sheet material Z, and the labeled image and a data set of the original image are used as training data.

[0096] The learning steps described above are merely specific examples, and the processing method X (processing system Y) according to the present invention may include existing learning algorithms such as YOLO (registered trademark), YOLACT, and MASK-RCNN.

[0097] In addition to the above, the processing system Y may have a learning process in which, while processing the sheet material Z, the resulting data is fed back to improve the accuracy of the detection process S2 and the position determination process S3. In this case, the learning process is assumed to include a feature detection pattern learning process L1, a continuity identification pattern learning process L2, and a gripping position determination pattern learning process L3. Each learning method will be described in detail below.

[0098] The feature detection pattern learning step L1 is a learning step for detecting the feature F appearing in the image, particularly for more accurately detecting the portion corresponding to the edge element ZF in segmentation. In actual processing, the way the light hits the sheet material Z, wear of the stitching on the sheet material Z, and other factors will cause individual differences in how the feature F appears on the image, which can reduce the accuracy of the segmentation. For this reason, in the processing of the feature detection pattern learning step L1, the control unit 21 combines the combination of the brightness value and RGB value (preferably the three-point color value feature TF) for each pixel with the segmentation area to create an input data set, stores this in the memory unit 22, and corresponds it to the state when the sheet material Z is lifted in the subsequent lifting step S42. At this time, the state of the sheet material Z is imaged by the lifted state imaging unit, and if the vertical length of the sheet material Z is equal to or greater than a predetermined value, a penalty (negative value) is associated with the dataset, and if it is less than the predetermined value, a reward (positive value) is associated with the dataset. The control unit 21 repeats this process for each series of processes, and reproduces, through segmentation, a state close to the dataset when a reward is obtained. This makes it possible to detect the feature amount F more accurately, regardless of individual differences in the sheet material Z.

[0099] The continuity identifying pattern learning step L2 is a learning step for identifying the continuity of the feature amount F more accurately in the continuity identifying step S24. 7, for example, the third group bounding box B3 and the fourth group bounding box B4 are likely to be judged by the user to be continuous. In addition, depending on how the sheet material Z is curled, an edge element ZF that is originally continuous may be interrupted by an overlapping portion, and may be judged to be discontinuous on the image. To solve this problem, the continuity specific pattern learning step L2 learns the above-mentioned patterns and detects longer continuous feature amounts F.

[0100] In the processing of the continuity specific pattern learning step L2, the control unit 21 stores the position and orientation of the grouped bounding boxes B and information on the segments belonging to them in the memory unit 22, and further stores them in the memory unit 22 as a data set combined with the distribution of colors in the image (brightness values, RGB values, etc.). The control unit 21 links this data set with the state of the lifted sheet material Z and the reward / punishment data as described above. The control unit 21 repeats this process for each series of processes, and reproduces a state close to the data set when a reward is being obtained in the continuity determination step S24. This makes it possible to more accurately detect the continuity of the feature amount F regardless of individual differences in the sheet material Z, and to determine a more appropriate gripping position.

[0101] The gripping position determination pattern learning step L3 is a learning step for improving the probability that a more appropriate gripping position can be determined in the position determination step S3. In the position determination process S3, the control unit 21 determines the grasping position from the continuous feature F that meets predetermined criteria, and links the data used there, such as a data set that summarizes the coordinate data of the ends, midpoints, and positional relationships of each group of the bounding box B, to the state of the lifted sheet material Z and reward / punishment data as described above. The control unit 21 repeats this process for each series of processes, and optimizes the process of determining the gripping position by adding variations to the coordinate data so that the processing of the position determination step S3 is performed in a state close to the data set when the reward is being obtained. This makes it possible to determine a more appropriate gripping position regardless of individual differences in the sheet material Z.

[0102] When the sheet material Z handled in the present invention is a plain towel or the like, as described above, its edge element ZF and feature value F correspond to the hem, selvedge, and pile, which are detected by the control unit 21 in the detection step S2. At this time, the processes performed by the control unit 21 from the detection step S2 to the position determination step S3 can be categorized into cases as shown in the table in FIG.

[0103] When the case classification shown in FIG. 8 is applied to this embodiment, it is assumed that the part to be preferentially determined as the gripping position among the hem, selvedge, and pile is set in advance in the position determining step S3. In FIG. 8, a circle indicates a state in which each feature amount F was recognized, and an x ​​indicates a state in which each feature amount F was not recognized.

[0104] 8 indicates a case where the control unit 21 is unable to detect any of the hem, selvedge, or pile. In this case, the control unit 21 determines that further processing is not possible, in other words, that the process cannot proceed to the position determination step S3. Thereafter, the control unit 21 performs processing such as removing the sheet material Z from the line of the conveying unit 5, or notifying the user of the error and requesting that the sheet material Z be repositioned in a state where some feature amount F can be detected. However, it is preferable that the control unit 21 can operate the manipulator 3 and perform processing to change the curling state of the sheet material Z (positioning state).

[0105] The second, third, and fifth rows of the table shown in Fig. 8 indicate cases where the control unit 21 detects any one of the hem, selvedge, and pile. In this case, the control unit 21 checks the detected gripping position against the part that is to be prioritized in the position determination step S3. If the gripping position setting matches the type of the detected feature amount F, the control unit 21 determines the gripping position from the feature amount F. For example, if the part to be prioritized as the gripping position is set to be the ear and the feature F detected by the control unit 21 also relates to the ear, the gripping position is determined from the feature F relating to the ear in the position determination step S3. When the type of the set gripping position differs from the type of the detected feature F, the control unit 21 determines the gripping position based on a different criterion from the area where the feature F was not detected. This criterion may be, for example, determining the widest identical area as the gripping position in the brightness recognition step S21.

[0106] The fourth, sixth, and seventh rows of the table shown in Fig. 8 show cases where the control unit 21 detects any two of the hem, selvedge, and pile. In this case, the control unit 21 checks the detected two with respect to the part to be prioritized as the gripping position in the position determination step S3, as in the case described above. If the setting of this gripping position matches the type of the detected feature amount F, the control unit 21 determines the gripping position from the feature amount F. When the gripping position setting and the type of detected feature F are different, the control unit 21 estimates the position of the remaining feature F, or an alternative gripping position, from the two types of detected feature F, and determines the gripping position from there.

[0107] 8 shows a case where the control unit 21 detects three types of fabric: hem, selvedge, and pile. In this case, the control unit 21 determines the gripping position in accordance with the part that is set as the priority gripping position in the position determination step S3.

[0108] The processing method X (including the modified examples) described above is expressed and executed by a program P that can be processed by a computer.

[0109] Examples of the processing method X according to this embodiment will be described in detail below with reference to the drawings. Each of the following examples is an example of the processing for carrying out this embodiment, and the form and order of each step may be changed.

[0110] <<Example 1>> 6 shows an image of the sheet material Z captured by the imaging unit 1 and then trimmed by the control unit 21. At this time, an edge element ZF of the sheet material Z is captured in the image as a feature amount F. In response to this, the control unit 21 detects the feature amount F in the detection step S2. In this embodiment, the edge element ZF is a colored thread, and therefore, as described above, a three-point color value feature TF using the RGB color system is used.

[0111] In the assignment step S23, the control unit 21 assigns a bounding box B to the feature amount F detected as described above. Here, if the control unit 21 has performed the processing of the distinction step S25, in the assignment step S23, bounding boxes B of different groups, such as a first group bounding box B1 and a second group bounding box B2, are assigned.

[0112] Furthermore, in the continuity identifying step S24, the control unit 21 combines the bounding boxes B with the first group bounding box B1 and the second group bounding box B2, and identifies the continuity of each feature amount F.

[0113] 6, the first group bounding box B1 and the second group bounding box B2 are assigned to feature amounts F (visual aspects of edge elements ZF) that are different from each other. Such differences in aspects are detected by factors such as different colors (distribution of RGB values) of feature amounts F, different positions (parallel and continuous, with no intersections or confluences recognized), and different sewing patterns (one is sewn in a straight line while the other is sewn in a zigzag pattern, for example).

[0114] After the first group bounding box B1 and the second group bounding box B2 are recognized, in a shape estimation step S26, the control unit 21 estimates the long and short sides of the sheet material Z. Even if the processing of the shape estimation step S26 is not performed, the position of the edge ZH of the sheet material Z is estimated in the shape estimation step S26 in the position determination step S3. After the above processing, the control unit 21 determines the position of any of the feature amounts F or the position of the estimated edge ZH as the gripping position for gripping by the manipulator 3.

[0115] In the priority assignment process S31, the control unit 21 assigns a priority to a group of bounding boxes B to be used as gripping positions based on information on the edge ZH of the sheet material Z and its length estimated through the processing of the shape estimation process S26, and information such as the position and shape of the corresponding feature F. As mentioned above, this priority is a numerical value, and for example, the control unit 21 assigns a priority of "1" to the second group bounding box B2 and a priority of "2" to the first group bounding box B1 in Fig. 6. In this case, the control unit 21 estimates that the second group bounding box B2 is closer to the edge ZH of the sheet material Z than the first group bounding box B1, and therefore assigns a lower priority value to the former.

[0116] In the gripping position determination step S32, the control unit 21 determines a gripping position from the group of bounding boxes B with the smallest priority value, i.e., the second group bounding box B2 in the example of Fig. 6. The gripping position determined at this time may be selected from any position on the second group bounding box B2, or may be selected from a position a predetermined distance (e.g., 10 mm) away from the second group bounding box B2 toward the edge ZH of the sheet material Z.

[0117] Furthermore, in the gripping position determination process S32, it is preferable for the control unit 21 to determine the gripping position to be near the center of the second group bounding box B2, and more preferably, if a corner of the sheet material Z is recognized at the end of the second group bounding box B2, it determines the gripping position to be a position shifted from the center of the second group bounding box B2 to the opposite side of the corner. This method makes it easier for the end effector 31 to grasp the midpoint of the edge of the sheet material Z, reducing the height to which the manipulator 3 lifts the sheet material Z and reducing the space required for the work. In addition, it is possible to reduce the possibility that the lifted and hanging sheet material Z will come into contact with surrounding equipment, get caught, and cause an interruption to the work.

[0118] <<Example 2>> FIG. 7 shows a state in which a first group bounding box B1, a second group bounding box B2, a third group bounding box B3, and a fourth group bounding box B4 are assigned to each feature amount F, and the continuity of each is specified. In particular, in the state shown in FIG. 7, the edges of each group of the second group bounding box B2 to the fourth group bounding box B4 are close to each other, and appear to the user to be continuous. In such a case, it is preferable that the control unit 21 performs processing to identify the continuity between the groups on the condition that the ends of the groups are separated by a predetermined length (number of pixels) or less. Alternatively, it is preferable that the control unit 21 assigns consecutive priorities to the second group bounding box B2 to the fourth group bounding box B4 in the priority assignment step S31.

[0119] However, in the above case, the control unit 21 assigns the lowest priority to the group in the center. That is, in the state of Fig. 7, the control unit 21 assigns a priority of "1" to the third group bounding box B3, a priority of "2" to the fourth group bounding box B4, and a priority of "3" to the second group bounding box B2. This method makes it easier for the control unit 21 to select a gripping position near the center of the continuous edge element ZF in the position determination step S3.

[0120] Hereinafter, a processing method X according to a second embodiment of the present invention will be described in detail with reference to the drawings. However, the same reference numerals will be used to designate the same parts as those in the first embodiment, and the description thereof will be omitted.

[0121] <<Embodiment 2>> 9, the processing system Y according to this embodiment includes two manipulators 3. Accordingly, in the position determination step S3, the control unit 21 determines two gripping positions, that is, two sets of coordinates.

[0122] In the gripping position determination step S32 in this embodiment, the control unit 21 determines two gripping positions from a group with a high priority. At this time, the control unit 21 determines the gripping positions (a set of coordinates indicating the points) so that the two gripping points are separated by a predetermined linear distance. Here, the "predetermined linear distance" is a length determined by the size of the end effector 31, the size of the gripper of the downstream manipulator that grips the sheet material Z in the next step of the present invention, the stretching length required in other subsequent steps, etc.

[0123] In the gripping position determination step S32, the length of the group selected based on the priority may be shorter than the "predetermined linear distance" for determining the gripping position. In this case, the control unit 21 first changes the group to be used to the next priority. If the gripping position still cannot be determined, the control unit 21 uses information related to the shape of the sheet material Z obtained in the detection step S2 (particularly the brightness recognition step S21 and the shape estimation step S26) to determine a pair of points (for example, two points) in an arbitrary group (outside the prohibited area) separated by a predetermined creepage distance (it is assumed that an estimated value is used, but if actual measurement is possible, an actual measured value is preferable) as the gripping position. This creepage distance is the distance between two points separated by a predetermined distance when tracing the surface of the sheet.

[0124] When the number of manipulators 3 in the processing system Y according to this embodiment increases to three or more, the number of determined gripping positions also increases corresponding to the number of manipulators 3. The control unit 21 also executes the process of determining each point (set of coordinates) of the gripping position in the gripping position determination step S32 in the same manner as the process described above. [Explanation of symbols]

[0125] X Processing Method Y Processing System 1. Imaging unit 2 Processing equipment 3 Manipulator 4 Display 5. Conveyor P Program P1 Image acquisition unit P2 Feature detection unit P3 positioning section P4 Grip operation unit S1 Image acquisition process S2 Detection step S3 Positioning process S4 Gripping operation process F feature B Bounding Box Z sheet material ZF edge element ZH Edge

Claims

1. A processing method for a computer to execute a process for determining an appropriate gripping position for a sheet material in any state, comprising: an image acquisition step of capturing an image of the sheet material to acquire image data; a detection step of detecting a feature amount included in the image data; a position determination step of determining the gripping position using the feature amount; Including, The detecting step includes determining segments of edge elements included in the sheet material using the feature amount, determining continuity of a plurality of the segments, and classifying the plurality of segments determined to have the continuity into groups; The edge element is a region where the yarn behavior changes along or within the sheet material, The position determination step determines the gripping position using elements of the group obtained from the feature amount.

2. the detecting step includes determining and grouping the edge elements based on the feature amounts; the position determination step determines, as the grip position, one point that meets a predetermined condition from the feature amounts corresponding to each of the groups; The processing method according to claim 1 .

3. In the position determination step, when a plurality of gripping positions are determined, a set of points that are spaced apart from each other by a predetermined linear distance in the segment determined to have the continuity is determined as the gripping positions. The processing method according to claim 1 .

4. In the position determination step, when a plurality of gripping positions are determined, a set of points that are spaced apart from each other by a predetermined creepage distance in the segment determined to have the continuity is determined as the gripping positions. The processing method according to claim 1 .

5. The detecting step an assignment step of assigning a bounding box to the segment when determining the continuity of the segment; a continuity identification step of linking the plurality of bounding boxes together under predetermined conditions to identify the continuity; Including, The processing method according to claim 1 .

6. Computer, an image acquisition unit that captures an image of the sheet material and acquires image data; a detection unit that detects feature amounts included in the image data, determines segments of edge elements of the sheet material using the feature amounts, treats the edge elements as areas where the state of threads changes along the sides or inside the sheet material, determines the continuity of a plurality of the segments, and classifies the plurality of segments that are determined to have the continuity into groups; a position determination unit that determines a gripping position using the elements of the group obtained from the feature amount; and determines an appropriate gripping position for the sheet material in any state.

7. A processing system in which a computer executes a process to determine an appropriate gripping position for a sheet material in any state, an image acquisition unit that captures an image of the sheet material and acquires image data; a detection unit that detects feature amounts included in the image data, determines segments of edge elements of the sheet material using the feature amounts, treats the edge elements as areas where the state of threads changes along the sides or inside the sheet material, determines the continuity of a plurality of the segments, and classifies the plurality of segments that are determined to have the continuity into groups; a position determination unit that determines the grip position using the elements of the group obtained from the feature amount; A processing system comprising:

Images