Sheet processing methods, systems and procedures

Abstract

This invention provides a processing method that can determine a suitable gripping position for exposing the feature quantity of a sheet in a preliminary stage, so that even when a large object is placed haphazardly, the corner can be exposed during operation. The present invention, which solves the above-mentioned problems, is a processing method X for enabling a computer to perform processing to determine a suitable gripping position in a sheet Z in any state, comprising: an image acquisition step S1 of photographing the sheet Z and acquiring image data; a detection step S2 of detecting the feature quantity F contained in the image data; and a positioning step S3 of determining the gripping position using the feature quantity F.

Description

Technical Field

[0001] This invention relates to a processing method and a processing system for identifying the shape of randomly placed sheets and detecting the appropriate part to be held in the stage before automatically folding sheets such as bed sheets. Background Technology

[0002] In the linen supply industry and accommodation facilities such as hotels, a large number of sheets are processed for tasks such as changing and washing bed sheets. In particular, considerable labor is required to neatly fold sheets, bath towels, and other sheets that are the same size as or larger than a human body. Therefore, it is preferable to automate the folding of these sheets, and the applicant of this application has developed a folding device for this purpose.

[0003] On the other hand, in the process of feeding sheets or other sheets into the folding device, it is necessary to ensure that the shape and orientation of the sheets are consistent to a certain extent beforehand. Since this operation is currently handled by manually opening the sheets or identifying the type of sheet, automation of this process is sought. At this time, as shown in Patent Document 1 or 2, devices utilizing the characteristic elements of the sheets, such as corners and patterns, have been developed.

[0004] Existing technical documents Patent documents Patent Document 1: US Patent No. 9909252 Patent Document 2: Japanese Patent Application Publication No. 2023-011021 Summary of the Invention The technical problem that the invention aims to solve Patent document 1 discloses a system for identifying and holding the four corners of small sheets such as towels. However, for larger objects such as bed sheets and bath towels, it is difficult to identify the four corners from their casually crumpled state. Therefore, regardless of the size of the sheet, the process of reliably exposing and mechanically holding the corners needs to be automated.

[0005] Furthermore, Patent Document 2 discloses a system for determining the pattern of laundry through image recognition. However, in order to identify the pattern, the object needs to be unfolded and placed, and if this part is done manually, it will be inefficient.

[0006] The present invention was made in view of the above-mentioned actual situation, and provides a processing method that can determine a suitable holding position for exposing the feature amount of the sheet in the previous stage, so that even when a large object is placed haphazardly, the corner can be exposed in the end.

[0007] Solutions for solving technical problems [1] The present invention, which solves the above-mentioned technical problems, is a processing method for enabling a computer to perform processing to determine a suitable gripping position in a sheet in any state. The processing method includes: an image acquisition step of photographing the sheet and acquiring image data; a detection step of detecting a feature quantity contained in the image data; and a positioning step of using the feature quantity to determine the gripping position.

[0008] According to the present invention, large sheets can be held in a suitable holding position without human intervention and transferred to the next process for exposing the corners of the sheet.

[0009] [2] According to the processing method described in [1], wherein, In a preferred embodiment of the invention, the detection process uses the feature quantity to determine the segment of the edge element of the sheet, and the positioning process determines the holding position based on the segment.

[0010] By adopting this configuration, image recognition can be used to grasp the vicinity of the edge of the sheet.

[0011] [3] According to the processing method described in [1] or [2], wherein, In a preferred embodiment of the invention, the edge element is provided along the edge of the sheet, the detection process determines the continuity of multiple segments, and groups the multiple segments that are determined to have the continuity, and the positioning process uses the elements of the group to determine the holding position.

[0012] By adopting this configuration, even if the edge elements are visually identified as curved, they can be identified with high accuracy.

[0013] [4] The processing method according to any one of [1] to [3], wherein, In a preferred embodiment of the invention, the sheet has an edge element that can be identified by each edge portion. The detection process identifies each edge element based on the feature quantity and groups them. The positioning process determines a point that meets a predetermined condition as the holding position based on the feature quantity corresponding to each group.

[0014] By adopting this configuration, the long side of the sheet can be held as tightly as possible, making it easier to lift the sheet.

[0015] [5] The processing method according to any one of [1] to [4], wherein, In a preferred embodiment of the invention, when multiple gripping positions are determined, the positioning process identifies a group of points within the segment that is judged to have continuity and are spaced apart by a predetermined straight-line distance as the gripping position.

[0016] By adopting this configuration, even when the sheet is held in multiple positions, it can be held and lifted in the appropriate holding position.

[0017] [6] The processing method according to any one of [1] to [5], wherein, In a preferred embodiment of the invention, when multiple gripping positions are determined, the positioning process identifies a group of points within the segment that are determined to have continuity and are spaced apart by a predetermined surface distance as the gripping position.

[0018] By adopting such a configuration, even when the sheet is held in multiple positions, it is possible to hold and lift the sheet in a more suitable position by matching the shape of the sheet. [7] The present invention, which solves the above problems, is a program that enables a computer to function as an image acquisition unit, a detection unit, and a positioning unit, and to determine a suitable gripping position in the sheet in any state. The image acquisition unit captures images of the sheet and acquires image data; the detection unit detects the feature quantity contained in the image data; and the positioning unit uses the feature quantity to determine the gripping position of the sheet.

[0020] By adopting this configuration, the corners of the large sheet are exposed without human intervention, allowing the sheet to be held in a suitable gripping position and transferred to the next process. [8] The present invention, which solves the above problems, is a processing system in which a computer performs a process to determine a suitable gripping position in a sheet in any state. The processing system includes: an image acquisition unit that captures an image of the sheet and acquires image data; a detection unit that detects the feature quantity contained in the image data; and a positioning unit that uses the feature quantity to determine the gripping position.

[0022] By adopting this configuration, the corners of large sheets are exposed without human intervention, allowing the sheets to be held in a suitable position and transferred to the next process.

[0023] Invention Effects According to the present invention, a processing method is provided that can identify characteristic quantities to determine a more suitable holding position even when a large object is placed randomly. Attached Figure Description

[0024] Figure 1 This is a flowchart of the processing method in Embodiment 1 of the present invention.

[0025] Figure 2 This is a flowchart of the processing method in Embodiment 1 of the present invention.

[0026] Figure 3This is a block diagram of the hardware that performs the processing method in Embodiment 1 of the present invention.

[0027] Figure 4 This is a schematic diagram of the hardware that performs the processing method in Embodiment 1 of the present invention.

[0028] Figure 5 This is a schematic diagram of the hardware that performs the processing method in Embodiment 1 of the present invention.

[0029] Figure 6 This is an image illustrating the image recognition state of the feature quantities in Embodiment 1 of the present invention.

[0030] Figure 7 This is an image illustrating the image recognition state of the feature quantities in Embodiment 1 of the present invention.

[0031] Figure 8 This is a table illustrating the detection modes of feature quantities and their corresponding processing scenarios in Embodiment 1 of the present invention.

[0032] Figure 9 This is a schematic diagram of the hardware that performs the processing method in Embodiment 2 of the present invention. Detailed Implementation

[0033] Hereinafter, the processing methods X of various embodiments of the present invention will be described using the accompanying drawings. The description will be detailed in the order of the configuration of the embodiments, the implementation methods, and other embodiments.

[0034] It should be noted that the embodiments shown below are all examples of the present invention, and the present invention is not limited to the following embodiments.

[0035] <Implementation Method 1> like Figure 1 As shown, the processing method X of this embodiment includes an image acquisition step S1, a detection step S2, and a positioning step S3, and also includes a holding operation step S4. This holding operation step S4 holds the sheet Z at a holding position determined by the positioning step S3, lifts the sheet Z, and transfers it to the next system. Furthermore, processing method X consists of… Figure 3 The processing system Y shown is executed.

[0036] <<Processing System>> like Figure 4 As shown, the processing system Y is an equipment system that performs the processing method X, which is used to hold the sheet Z at a suitable position and transfer it to the downstream process. The equipment system includes an imaging unit 1, a processing device 2, an operator 3, a display unit 4 (not shown), and a conveying unit 5.

[0037] The imaging unit 1 is a camera (line scan camera or area scan camera) that takes pictures of the sheet Z, which is the object, based on brightness, chroma, and hue information. In order to perform image recognition more reliably, it is preferable to have a structure with a light source unit.

[0038] like Figure 3 As shown, the processing device 2 is a computer device that includes a control unit 21 and a storage unit 22, and also includes an input device, a storage device, an interface, etc. The processing system Y controls the imaging unit 1, the operator 3, the display unit 4, and the transport unit 5 through the processing device 2, and also performs various processing related to image recognition.

[0039] The control unit 21 is a computing device such as a CPU (processor) or microcomputer that executes computer-executable instructions.

[0040] The storage unit 22 includes a non-volatile memory for storing the program P that executes the processing method X of this embodiment, and a volatile memory 221 for temporarily storing image data captured by the imaging unit 1. Data and program P are read from each memory and executed by the control unit 21. However, as... Figure 3 As shown, the storage unit 22 does not need to clearly distinguish between non-volatile memory and volatile memory 221; as long as it can perform the above-mentioned functions, it can be set up as a whole.

[0041] The storage unit 22 stores a program P for performing various processes in this embodiment. The program P enables the processing device 2, and in particular the control unit 21, to function as an image acquisition unit P1, a detection unit P2, a positioning unit P3, and a gripping operation unit P4.

[0042] The image acquisition unit P1 controls the shooting unit 1 to shoot the sheet Z, and the shooting unit 21 processes the captured image into image data.

[0043] The detection unit P2 includes: a brightness recognition unit P21, which uses brightness detection image data to detect the unevenness of the surface of sheet Z; a determination unit P22, which uses the brightness distribution in a specific area within the image data to determine whether there is a feature quantity F; an allocation unit P23, which allocates a bounding box B to the feature quantity F; a continuity determination unit P24, which connects the bounding box B to determine the continuity of the feature quantity F; a differentiation unit P25, which groups the feature quantities F according to different positions and shapes; and a shape estimation unit P26, which estimates the position of the edge portion ZH based on the shape of the detected feature quantity F, and further estimates the type of the edge portion ZH of sheet Z based on the detection of the feature quantity F.

[0044] The positioning unit P3 has: a priority assignment unit P31, which assigns a priority to each group of feature quantities F for selection as a holding position; and a temporary selection unit P32, which temporarily selects candidates for holding positions based on feature quantities F that have been determined to be continuous.

[0045] The P4 control unit controls the operator 3 to hold and lift the sheet Z (object) at the determined holding position and transfer it to the next process.

[0046] The following describes in detail the behavior of each part of program P when it is processed, using the description of the corresponding procedures.

[0047] The manipulator 3 is a so-called robotic arm with an end effector 31 capable of holding the sheet Z, lifting and transporting the sheet Z, preferably a structure that operates with 6 degrees of freedom (e.g., a structure with 4 axis joints). In addition, in order to reliably maintain the state of holding the sheet Z, the end effector 31 is preferably a structure with an adsorption part that attracts the sheet Z and a pair of clamps that hold the sheet Z in the vicinity.

[0048] Display unit 4 is a display that shows the captured image data and the image recognition process, using an LCD screen or similar device. This allows the user to not only check the image recognition status of sheet Z, but also to visually check and correct any improper placement of sheet Z. Furthermore, display unit 4 can also function as a touch panel and thus as an input device.

[0049] The transport unit 5 is a movable platform that allows the user to place the sheet Z and transport it to the shooting range of the shooting unit 1. It is envisioned to be in the form of a conveyor belt or the like. In addition, in order to facilitate the identification of the sheet Z, the transport unit 5 is preferably made of a color that is very different from the sheet Z, especially a dark color such as black.

[0050] Sheet Z refers to the fabric including the edge elements ZF used for image recognition. In this embodiment, it is primarily envisioned as an item that is too large to be folded up by one hand, such as a face towel, bath towel, or bed sheet. However, smaller items such as handkerchiefs or square towels can also be used in this embodiment. It should be noted that, for ease of explanation in the following description, the fabric projected into the image is sometimes referred to as sheet Z, in which case it refers to the data on the image corresponding to sheet Z.

[0051] Edge element ZF is provided along the edge ZH of sheet Z and serves as a marker (or clue) for determining the edge ZH according to the processing method X of this embodiment. The actual edge element ZF is, for example, a place where the color thread, pattern, texture, presence or absence of nap, etc., changes; that is, the area near the edge ZH of sheet Z where the shape of the seam (sewing) changes.

[0052] More specifically, this refers to areas where the edges of a sheet material, such as the folds and selvage of a regular towel or bed sheet, can be visually identified due to variations in the sewing pattern. Furthermore, it is preferable to also treat areas such as weave patterns resulting from variations in fabric texture, flat areas of towels (areas without attached fibers), and areas subsequently printed with patterns or text, as edge elements.

[0053] The feature quantity F is a numerical data indicator that represents the state of the edge element ZF reflected in the image captured by the imaging unit 1 of the sheet Z. Specifically, it refers to the position, color, and orientation of the edge element ZF in the image. More specifically, it represents how much the values ​​of each data point, such as brightness and RGB, assigned to each pixel vary between adjacent pixels or within a specified area of ​​the image.

[0054] The index used for the characteristic quantity F varies even for the same sheet Z, depending on its condition. For example, imagine that the way light is irradiated changes due to the way it is rolled; even for threads of the same color, the RGB values ​​in the image will be different, and similarly, the brightness values ​​will also be different. Furthermore, even for a plain-colored sheet Z, it can be imagined that repeated use, washing, and wear of the threads (especially the lint) will change the distribution of brightness (pattern) in the image.

[0055] To avoid variations in the identification of the aforementioned feature quantity F, the detection criterion for feature quantity F is preferably configured to accommodate certain variations. However, the detection criterion for feature quantity F can be based on rules with pre-set parameters, or it can be formed through learning.

[0056] The implementation method of this embodiment will be described in detail below with reference to the accompanying drawings. Furthermore, the implementation method shown below is merely an example, and the implementation method is not limited to this; the order may be adjusted.

[0057] <<Handling Methods>> Each program P of this embodiment stored in the storage unit 22 and the processing method X executed according to each program P are processed according to... Figure 1 The flowchart shown is followed.

[0058] The image acquisition process S1 (corresponding to the image acquisition unit P1) includes an image acquisition process S11 in which the image acquisition unit 1 captures the sheet Z.

[0059] In the shooting process S11, the control unit 21 controls the shooting unit 1 to take a color picture of the sheet Z. The captured image data is read for use in the next inspection process S2, and the captured image data is stored in the storage unit 22 (specifically, the volatile memory 221) and then displayed on the display unit 4. At this time, the control unit 21 preferably simultaneously identifies when the sheet Z, which is randomly placed (or transported by the conveyor unit 5), is stationary and performs automatic shooting. With this process, the user only needs to place the sheet Z within the shooting range, preventing delays in system operation due to forgetting to operate.

[0060] The control unit 21 can simultaneously process the captured image to crop out unwanted parts that correspond to the position of the sheet Z within the image.

[0061] The detection process S2 (corresponding to the detection unit P2) includes a brightness recognition process S21, a determination process S22 for determining the presence or absence of a feature quantity F, an allocation process S23 for assigning a boundary box B to the identified feature quantity F, and a continuity determination process S24 for determining the continuity of the feature quantity F. Furthermore, the detection process S2 preferably includes a differentiation process S25 for distinguishing between multiple feature quantities F, and a shape estimation process S26 for inferring the shape of the sheet Z based on the type of feature quantity F, particularly the shape estimation of the type of edge (long edge and short edge, etc.).

[0062] In the brightness recognition process S21, the control unit 21 extracts brightness data from the read image data. In this process, the control unit 21 performs both macroscopic and microscopic extraction.

[0063] In macroscopic extraction, the control unit 21 first divides the area mapped by the sheet Z in the read image into regions of a predetermined size (e.g., 10 pixels square). Next, it calculates the average brightness of the pixels contained in each region and then compares the average brightness between two adjacent regions. If the difference between the average values ​​at this time is below a predetermined value (e.g., below 20), it is determined that the two predetermined regions belong to the same region of the sheet Z.

[0064] Following the macroscopic extraction process described above, the control unit 21 performs a microscopic extraction process. In this microscopic extraction, the control unit 21 identifies patterns in the brightness values ​​of each misaligned pixel within regions determined to be in the same area by the macroscopic extraction. That is, the control unit 21 identifies the brightness variations between pixels.

[0065] The brightness recognition process S21 can be replaced with a color recognition process (color recognition process). In this case, the control unit 21 replaces the brightness data of each pixel used for detection with color, i.e., RGB values, and performs macroscopic to microscopic extraction. Alternatively, the brightness recognition process S21 can also be used as a processing process that uses both brightness and color (RGB values).

[0066] In the judgment process S22, the control unit 21 identifies changes in the brightness pattern within a region of the sheet Z identified in the brightness recognition process S21, or compares the identified brightness pattern with data pre-stored in the storage unit 22. Based on this, if a significant change in the brightness pattern is detected, the control unit 21 determines that this is a location of a change in the sewing pattern, specifically a portion of the characteristic quantity F where the weave or other features have changed.

[0067] Therefore, the processing system Y can process elements such as folds, edges, and the presence or absence of lint, which are common features of plain-colored towels and sheets, as feature quantities F.

[0068] The brightness value used in the brightness recognition process S21 and the judgment process S22 is based on the principle that the unevenness caused by the stitching of the sheet Z is recognized as brightness through light reflection. With this configuration, even if the sheet Z is plain, the control unit 21 can recognize the feature quantity F.

[0069] In the brightness recognition process S21 and the determination process S22, although the control unit 21 mainly uses the detection of values ​​related to brightness, when the edge element ZF is a colored line or the like, and color (RGB value) can be used, it is preferable to use the value related to color to detect the feature quantity F.

[0070] In the allocation process S23, in order to match the identified feature quantity F, the control unit 21 first allocates a segment to each pixel, and then allocates a rectangular bounding box B to the collection of those segments. Preferably, the control unit 21 displays the result of allocating the bounding box B on the display unit 4.

[0071] Bounding box B is an icon used to assign a range of segments within an assigned set of segments that is divided into intervals of a specified length. In bounding box B, the length of the edges in the length direction (continuous direction) towards the feature quantity F (the set of segments) can be arbitrary as long as it is below the specified length, while the edges in the direction that divides the feature quantity F are all of fixed length.

[0072] In the allocation process S23, the control unit 21 allocates multiple bounding boxes B along the feature quantity F and then transfers them to the next process.

[0073] When there are multiple feature values ​​F, the control unit 21 changes the color of the bounding box B (balance of RGB values) for each group and combines it with different data to reflect the result of the differentiation process S25 (described later).

[0074] In the continuity determination step S24, regarding multiple bounding boxes B, if they intersect, touch, or approach each other at the ends (or any position of the short side) in the length direction (determined based on the number of pixels between them being less than a predetermined value), the control unit 21 determines the continuity of the feature quantity F by establishing an association between the bounding boxes B. By performing this processing on the image, edge features ZF with continuity are actually identified.

[0075] Here, "continuity" refers to a region where pixels identified as feature quantity F are located adjacently at multiple locations, or to a group of multiple bounding boxes B that have been associated.

[0076] It should be noted that when only feature quantity F is processed without using bounding box B, the assignment step S23 and the continuity determination step S24 are omitted. Additionally, when there is only one bounding box B, the continuity determination step S24 is omitted.

[0077] The continuity determination process S24 can also be processed as follows: the control unit 21 determines the proximity of the segments and establishes a relationship between the segments or the group of segments, resulting in the boundary boxes B being connected to each other.

[0078] Figure 1 In the processing method X shown, the control unit 21 receives the processing of the continuity determination process S24 in the differentiation process S25. When the characteristic quantity F that is determined to be continuous can be divided into multiple groups or is judged to have multiple types, the characteristic quantity F is grouped.

[0079] In addition, such as Figure 2 As shown, the control unit 21 in the differentiation process S25 can also process in the following order: at the stage where the determination process S22 ends, if there are multiple patterns or distribution locations of the determined feature quantity F, the feature quantity F is grouped according to its type.

[0080] Furthermore, if one or fewer feature quantities are identified, the control unit 21 will not perform the differentiation process S25 and will proceed to the next process.

[0081] In the shape estimation process S26, after estimating the position of the edge ZH of the sheet Z based on the position, continuity, and length direction of the detected feature quantity F, the control unit 21 estimates the type of edge ZH that may exist near the edge element ZF corresponding to the feature quantity F, based on the shape of the feature quantity F. At this time, the control unit 21 specifically determines the long side and short side of the sheet Z.

[0082] When the edge element ZF is set along the edge ZH of the sheet Z, the shape estimation process S26 is preferably incorporated into the processing method X. For example, for a general towel, edge elements ZF with different sewing methods are set on sides of different lengths with the long side as the selvage and the short side as the fold, and the control unit 21 identifies these. As a result, in the positioning process S3 and the holding operation process S4, the operator 3 can easily hold the sheet Z in a position that is easy to handle, thereby reducing errors in the sheet Z handling operation.

[0083] The positioning process S3 (corresponding to the positioning unit P3) has a priority assignment process S31 and a holding positioning process S32. In this process, the control unit 21 uses the feature quantity F determined in the detection process S2 to determine the holding position of the sheet Z.

[0084] In the priority assignment process S31, the control unit 21 assigns a priority to each group of bounding boxes B (i.e., feature quantities F) that were grouped in the differentiation process S25, and assigns a priority that should be determined as the gripping position. This priority is, for example, a numerical value, and is assigned sequentially as "1", "2", "3", etc., starting from the group that is the most powerful as the gripping position. Based on the assigned priorities, in the gripping positioning process S32, the gripping position is determined starting from the group with the lowest priority value.

[0085] In the priority assignment process S31, the control unit 21 first compares the continuous lengths of each grouped boundary box B (segment) and assigns priority values ​​sequentially starting from the longest.

[0086] Next, the control unit 21 determines the midpoint coordinates (XY coordinates within the image) of each group and determines whether each coordinate is inside the prohibited area (including the boundary). The prohibited area refers to the area where the end effector 31 cannot hold the sheet Z due to interference between the edge of the conveying unit 5, the base of the manipulator 3, or other objects outside the sheet Z. Based on this determination, for groups determined to have their midpoints within the prohibited area, the control unit 21 lowers the assigned priority to the lowest level (a value 1 or 2 greater than the maximum value at that moment).

[0087] Regarding the relationship between the midpoint of a group and the prohibited area, the case where only the area near the midpoint is within the prohibited area, while the majority of the group is outside the prohibited area, is also considered. Therefore, even if the area near the midpoint of a group is within the prohibited area, but a fixed proportion or more (e.g., more than 70%) of the group's length is outside the prohibited area, the control unit 21 preferably does not lower the priority, but instead performs the process of re-determining the candidate holding position of the group as the point that is outside the prohibited area and closest to the midpoint.

[0088] The control unit 21 can also use information about whether the area around the sheet Z group is raised as a criterion for priority adjustment.

[0089] In the brightness recognition process S21, since the control unit 21 can infer the vertical relationship between the regions of sheet Z, it uses this information to lower the priority of a group when there is a higher sheet Z region near the grouped boundary box B.

[0090] Therefore, it is possible to prevent the sheet Z from failing to unfold properly when the manipulator 3 is used to hold and lift the sheet Z, which would cause the operation to stall in subsequent processes of the present invention (including the loading process described later).

[0091] To identify the shape of sheet Z at this time with higher accuracy, the processing system Y can also capture images of sheet Z using multiple imaging units 1 and process the shape of sheet Z using morphology analysis.

[0092] In the priority assignment process S31, the control unit 21 preferably performs a process of downgrading the priority of the group of bounding boxes B (i.e. feature quantity F) that meet the specified conditions after assigning the priority.

[0093] The "specified conditions" here refer, for example, to the case where the feature quantity F within the group is located along the short side of the sheet Z, or to the case where the continuous length of the bounding box B is below a fixed value. By lowering the priority assigned to these groups, it is possible to prevent the sheet Z from drooping along the long side, causing the manipulator 3 to be lifted to a height higher than required, or to prevent the sheet Z from unfolding when the manipulator 3 is lifted because the short edge element ZF is being held.

[0094] In the gripping and positioning process S32, the control unit 21 determines either the detected feature quantity F or the estimated edge portion ZH as the position where the end effector 31 grips the sheet Z. For example, the control unit 21 selects a point from the feature quantity F as the gripping position and determines its XY coordinates. In this case, it is preferable to determine a method that uses a predetermined reference, such as a continuous feature quantity F or a point near the midpoint of the edge portion ZH, to determine the gripping position.

[0095] In the gripping and positioning process S32, the control unit 21 determines the midpoint of the group with the smallest priority value (or the re-set gripping position candidate) as the gripping position based on the priority assigned and further adjusted for each group in the priority assignment process S31.

[0096] The holding operation process S4 (corresponding to the holding operation unit P4) includes a holding process S41, a lifting process S42, and a transfer process S43. In the holding operation process S4, the control unit 21 operates (controls) the operator 3 to hold the sheet Z and transfer it to the next process.

[0097] In the holding process S41, the control unit 21 controls the manipulator 3 to hold the sheet Z at the holding position determined in the positioning process S3 using the end effector 31. However, since the holding position determined in the positioning process S3 is only the XY coordinates on the image, the control unit 21 converts these coordinates into coordinates corresponding to the manipulator 3. Regarding the Z coordinate of the manipulator 3, it is envisioned that it be a pre-specified value, or a value determined using the vertical height inferred from the captured image, etc. At the location specified in the above manner, the manipulator 3 holds the sheet Z using the end effector 31. Furthermore, the control unit 21 can also adjust the direction of the end effector 31 so that the direction of clamp opening is perpendicular or parallel to the edge element ZF.

[0098] In the lifting process S42, the control unit 21 lifts the sheet Z held by the end effector 31 to a predetermined height. This lifting height can be a pre-specified height or a value that is appropriately changed according to the state of the sheet Z. As a result, the edges and corners of the sheet Z become more easily exposed in the drooping state, creating a state that facilitates the execution of the next process (the operation placed with the corners of the sheet Z exposed).

[0099] In the transfer process S43, the control unit 21 transfers the lifted sheet Z from the manipulator 3, which is held in this invention, to the downstream manipulator that performs the next process. At this time, the manipulator 3 preferably holds the sheet Z near the holding position, and the exchange is performed as if it were handed over by hand. As a result, the shape of the lifted sheet Z can be maintained as much as possible.

[0100] The processing method X of this embodiment may also include a placement step as the next processing step after the holding operation step S4, in which the transferred sheet Z is placed with its edge ZH or corner exposed. Thus, by using the folding manipulator provided immediately after this invention, the corners of the sheet Z can be mechanically held, thereby facilitating the folding operation. Furthermore, the placement step requires a program P for executing the processing device 2, as well as a downstream manipulator and a movable placement stage, which can be implemented by adding to this invention.

[0101] The processing method X of this embodiment can also have the following modifications. However, the modifications shown below are only examples, and the presence or absence of each example is determined independently unless there is a special subordinate relationship.

[0102] <<Example of Change>> In color determination of color images, although the RGB color system was used in the above description, other systems can be used as long as the color system is numerically quantifiable, such as the CMYK color system, XYZ color system, Lab color system, etc. Even in these cases, it is preferable that the control unit 21 is configured as described above to be able to cope with the variations caused by individual differences and consumption of the sheet Z, and further to become a configuration and method that can be used in the inspection process S2.

[0103] As a component capable of adapting to changes, for example, the characteristic quantity F preferably uses the three-point color separation value characteristic quantity TF. The three-point color separation value characteristic quantity TF is only one example of the characteristic quantity F, and indices calculated by other methods can also be used.

[0104] The three-point color separation characteristic TF is calculated in a specified area within the image based on the maximum (MR, MG, MB, ML), minimum (mR, mG, mB, mL), and average (AR, AG, AB, AL) values ​​of each pixel's RGB color system (or other color systems) and / or brightness (hereinafter referred to as L) distribution, using the formula shown below.

[0105] R component characteristic value = (MR-mR) / AR G component characteristic value = (MG - mG) / AG B component characteristic quantity = (MB - mB) / AB L-component characteristic quantity = (ML - mL) / AL By using the above methods, the accuracy of the detection feature quantity F can be maintained regardless of the shape of the sheet Z, its positional relationship with the light source, or the deterioration of the sheet Z.

[0106] The processing device 2 and / or the detection unit P2 may also have an abnormal placement notification unit. This abnormal placement notification unit informs the user when the shape of the sheet Z prevents the detection unit P2 (and detection process S2) from detecting the feature quantity F, and requests that the sheet Z be repositioned so that the edge element ZF can be captured. The notification may be displayed on the display unit 4 via a pop-up window or by sound from a separate speaker.

[0107] The processing method X of this embodiment can also be as follows: In the image acquisition step S1, multiple cameras are used to capture images of the sheet Z and obtain the shape of the sheet Z, and the detection step S2 and the positioning step S3 are performed using the shape. At this time, everything is as described above until the detection step S2, but in the positioning step S3, especially in the priority assignment step S31, the control unit 21 can use the "height" of the bounding box B through the shape.

[0108] In the priority assignment process S31 under this condition, the control unit 21 uses the length, midpoint, and highest point of each group of bounding boxes B that were grouped in the differentiation process S25 to determine the priority order.

[0109] For example, regarding the length of a group, the control unit 21 increases the priority of groups with a length of more than a preset value, and regarding the highest point, determines the highest point within each group and further calculates the positional relationship with the intermediate point.

[0110] Accordingly, the control unit 21, in principle, raises the height of the highest point to the highest priority group, assigning priority in order to control the edge element ZF at the highest position.

[0111] However, in cases where the distance between the highest point and the middle point within a group is extremely far (the highest point is near the end of the group, and if the highest point is held, it may be impossible to hold the center of the edge element ZF), the control unit 21 uses the middle point instead of the highest point to assign priority and performs processing to determine the holding position, provided that the specified conditions are met.

[0112] The processing method X (processing system Y) of this embodiment may also have a learning process for improving the accuracy of the detection process S2 and the positioning process S3, instead of the processing process that uses a preset benchmark such as the three-point color separation value feature quantity TF.

[0113] This learning process, for example, involves the processing system Y reading teacher data, and the control unit 21 determining and optimizing the parameters used to detect feature quantity F and determine continuity. Specifically, this learning process, for images obtained from photographing sheet Z, manually determines the pixels and their continuity as feature quantity F, and uses the dataset of the labeled image and the original image as teacher data.

[0114] The learning steps shown above are merely specific examples. The processing method X (processing system Y) of the present invention can also use existing learning algorithms such as YOLO (registered trademark), YOLACT, and MASK-RCNN.

[0115] In addition, the processing system Y can also have a learning process that simultaneously processes sheet Z and feeds back data as its result, improving the accuracy of the detection process S2 and the positioning process S3. In this case, the learning process is envisioned as a feature quantity detection mode learning process L1, a continuity determination mode learning process L2, and a holding position determination mode learning process L3. Each learning method is detailed below.

[0116] The feature detection pattern learning process L1 is a learning process used to more accurately detect the parts corresponding to edge features ZF in the detection of feature F appearing in an image, especially in the allocation of segments (semantic segmentation).

[0117] Considering the actual processing scenario, due to factors such as the way light illuminates the sheet Z and the wear and tear of the sheet Z's stitching, the representation of feature quantity F in the image will vary from person to person, which may reduce the accuracy of semantic segmentation. Therefore, in the feature quantity detection pattern learning process L1, the control unit 21 takes the combination of the brightness value and RGB value (preferably the three-point color separation feature quantity TF) of each pixel along with the semantic segmentation region as the input dataset and stores it in the storage unit 22, so that it corresponds to the state when the sheet Z is lifted in the subsequent lifting process S42.

[0118] At this point, the imaging unit captures the state of sheet Z while it is in a lifted state. If the vertical length of sheet Z is greater than a predetermined value, a penalty (negative value) is associated with the dataset; if it is less than the predetermined value, a reward (positive value) is associated with the dataset. The control unit 21 repeats this operation in each series of processes, reproducing the state of the dataset close to when a reward was obtained through semantic segmentation. Thus, regardless of individual differences in sheet Z, the feature quantity F can be detected more accurately.

[0119] The continuity recognition pattern learning process L2 is a learning process used in the continuity determination process S24 to more accurately determine the continuity of the feature quantity F.

[0120] For example in Figure 7 In the image, the third set of bounding boxes B3 and the fourth set of bounding boxes B4 are very likely to be perceived as continuous by the user. In addition, there are cases where originally continuous edge features ZF are partially obscured by overlapping parts due to the way the sheet Z is rolled, and are thus perceived as discontinuous in the image.

[0121] To address this issue, the continuous deterministic pattern learning process L2 learns patterns like those described above and detects longer, continuous feature quantities F.

[0122] In the continuous pattern learning process L2, the control unit 21 stores the position and orientation of the grouped bounding box B and the information of the segments belonging to it in the storage unit 22, and further stores it in the storage unit 22 as a dataset combined with the distribution of colors (brightness values, RGB values, etc.) in the image.

[0123] The control unit 21 establishes a correlation between this dataset and the state and reward / penalty data of the same lifted sheet Z described above. The control unit 21 repeats this operation in each series of processes, reproducing the state of the dataset close to when a reward was obtained in the continuity determination step S24. Therefore, regardless of individual differences in sheet Z, the continuity of the feature quantity F can be detected more accurately, thereby enabling the determination of a more suitable holding position.

[0124] The gripping position determination pattern learning process L3 is a learning process used in the positioning process S3 to increase the probability of determining a more suitable gripping position.

[0125] In the positioning process S3, the control unit 21 determines the feature quantity F that meets the specified conditions as the holding position based on the feature quantity F that has continuity, but establishes a correlation with the data used here (e.g., a dataset that summarizes the coordinate data of the ends, midpoints, and positional relationships of the groups of bounding box B, and the data on the state and reward / penalty of the lifted sheet Z as described above).

[0126] The control unit 21 repeats this operation in each series of processes, applying changes to the coordinate data to optimize the gripping position determination process, so as to perform the positioning step S3 in a dataset state close to when the reward is obtained. Therefore, regardless of individual differences in the sheet Z, a more suitable gripping position can be determined.

[0127] When the sheet material Z processed by the present invention is a plain-colored towel or the like, as described above, its edge element ZF and characteristic quantity F correspond to the hem, selvage, and pile, and the control unit 21 in the detection step S2 detects these. At this time, the processing performed by the control unit 21 from the detection step S2 to the positioning step S3 can be as follows: Figure 8 The cases are categorized as shown in the table.

[0128] In Figure 8 When the situation classification shown is applied to this embodiment, in the positioning process S3, the part that is preferentially determined as the holding position among the folded edge, fabric edge, and pile is predetermined.

[0129] It should be noted that, Figure 8 The ○ symbol in the diagram represents a state where each feature quantity F can be identified, and the × symbol represents a state where each feature quantity F cannot be identified.

[0130] Figure 8 The first row of the table shows situations where the control unit 21 cannot detect any of the following: folded edge, fabric edge, or nap. In this case, the control unit 21 determines that further processing cannot be performed; in other words, it cannot proceed to the positioning process S3.

[0131] Afterwards, the control unit 21 performs processes such as removing the sheet Z from the conveyor line of the transfer unit 5, or notifying the user of an error and requesting that the sheet Z be reloaded to a state where a certain characteristic quantity F can be detected. However, it is preferable that the control unit 21 can operate the operator 3 to change the curling mode (loading state) of the sheet Z.

[0132] Figure 8The second, third, and fifth rows of the table show the cases where the control unit 21 detects any one of the following: folded edge, fabric edge, or nap. At this time, in the positioning process S3, the control unit 21 compares the result with the preferred gripping position. If the gripping position setting matches the detected characteristic quantity F, the control unit 21 determines the gripping position based on that characteristic quantity F.

[0133] For example, if the part that is the preferred gripping position is set as the selvage, and the feature quantity F detected by the control unit 21 is also related to the selvage, the gripping position is determined in the positioning process S3 based on the feature quantity F related to the selvage.

[0134] When the set gripping position differs from the type of detected feature quantity F, the control unit 21 determines the gripping position from the region where no feature quantity is detected based on another reference. This reference is envisioned, for example, in the brightness recognition process S21, using the widest area of ​​the same region as the gripping position.

[0135] Figure 8 Rows four, six, and seven of the table show the case where the control unit 21 detects any two of the following: folded edge, fabric edge, and nap. In this case, the control unit 21, as described above, compares the position with the preferred gripping position in the positioning process S3. If the gripping position setting matches the type of the detected feature quantity F, the control unit 21 determines the gripping position based on that feature quantity F.

[0136] When the set gripping position is different from the type of detected feature quantity F, the control unit 21 infers the position of the remaining feature quantity F or the gripping position that replaces it based on the two detected feature quantities F, thereby determining the gripping position.

[0137] Figure 8 The eighth row of the table also shows three scenarios detected by the control unit 21: folded edge, fabric edge, and nap. In these scenarios, the control unit 21 determines the gripping position based on the portion set as the preferred gripping position in the positioning process S3.

[0138] The processing method X (including variations) shown above is performed and executed by a program P that can be processed by a computer.

[0139] Hereinafter, embodiments of the processing method X of this embodiment will be described in detail using the accompanying drawings. Furthermore, the following embodiments are merely examples of the processing of this embodiment, and the configuration and order of each step may vary.

[0140] <<Example 1>> Figure 6The image is captured by the imaging unit 1 and further cropped by the control unit 21. At this time, the edge features ZF of the existing sheet Z are reflected in the image as feature quantity F. Accordingly, in the detection process S2, the control unit 21 detects the feature quantity F.

[0141] Since the edge element ZF in this embodiment is a colored line, as described above, the three-point color separation value feature quantity TF, which adopts the RGB color system, is used.

[0142] In the allocation process S23, the control unit 21 allocates bounding boxes B to the feature quantity F detected above. Here, if the control unit 21 has performed the differentiation process S25, different groups of bounding boxes B are allocated in the allocation process S23, such as the first group of bounding boxes B1 and the second group of bounding boxes B2.

[0143] Furthermore, in the continuity determination process S24, the control unit 21 combines the boundary boxes B in each of the first group of boundary boxes B1 and the second group of boundary boxes B2 to determine the continuity of each feature quantity F.

[0144] When multiple feature quantities F are reflected in the image, the control unit 21 performs the differentiation process S25. Figure 6 In the example, the shapes (visual shapes of edge elements ZF) of the feature quantities F assigned to the first group of bounding boxes B1 and the second group of bounding boxes B2 are different. Such differences in shape are detected by factors such as different colors (assignment of RGB values) of feature quantities F, different positions (parallel and continuous without identification of intersections, merging points, etc.), and different stitching patterns (one is stitched in a straight line, while the other is stitched in a jagged shape, etc.).

[0145] After identifying the first set of bounding boxes B1 and the second set of bounding boxes B2, the control unit 21 estimates the long and short sides of the sheet Z in the shape estimation process S26. Furthermore, even without performing the shape estimation process S26, the position of the edge ZH of the sheet Z is estimated in the shape estimation process S26 during the positioning process S3. After the above processing, the control unit 21 determines the position of any feature quantity F or the estimated position of the edge ZH as the holding position held by the manipulator 3.

[0146] In the priority assignment process S31, the control unit 21 assigns a priority to the bounding box group B, which serves as the holding position, based on the information of the edge ZH of the sheet Z and its length, the position and shape of the corresponding feature quantity F, through the processing of the shape estimation process S26.

[0147] As mentioned earlier, this priority is a numerical value; for example, control unit 21 to Figure 6The second set of bounding boxes B2 is assigned a priority of "1", and the first set of bounding boxes B1 is assigned a priority of "2". In this case, the control unit 21 assumes that the second set of bounding boxes B2 is closer to the edge ZH of the sheet Z than the first set of bounding boxes B1, and therefore assigns the former a priority value that is smaller.

[0148] In the holding and positioning process S32, the control unit 21 determines the group of boundary boxes B with smaller priority values ​​based on their grouping. Figure 6 In the example, the gripping position is determined for the second set of boundary boxes B2. The gripping position can be selected from any position on the second set of boundary boxes B2, or it can be selected from a position away from the second set of boundary boxes B2 in the ZH direction towards the Z edge of the sheet, away from a specified distance (e.g., 10mm).

[0149] In addition, in the gripping and positioning process S32, the control unit 21 preferably determines the gripping position near the center of the second set of boundary frames B2, and more preferably, when the corner of the sheet Z is identified at the end of the second set of boundary frames B2, the gripping position is determined from the center of the second set of boundary frames B2 in the opposite direction to the corner.

[0150] In this way, even if the end effector 31 is within the edge of the sheet Z, it can easily hold the area near the midpoint of the edge, thereby suppressing the height at which the sheet Z is lifted by the manipulator 3 and minimizing the space required for operation. Furthermore, it can suppress the possibility of the lifted and drooping sheet Z contacting or snagging with surrounding devices and obstructing the operation.

[0151] <<Example 2>> Figure 7 The diagram shows that the first group of bounding boxes B1, the second group of bounding boxes B2, the third group of bounding boxes B3, and the fourth group of bounding boxes B4 have been assigned to each feature quantity F and the states of each continuity have been determined.

[0152] Especially Figure 7 In the state shown, the ends of each of the second group of bounding boxes B2 to the fourth group of bounding boxes B4 are close to each other, and appear continuous from the user's perspective.

[0153] In this case, it is preferable that the control unit 21 determines the continuity between groups based on the condition that the ends of the groups are less than or equal to a predetermined length (number of pixels). Alternatively, it is preferable that in the priority assignment process S31, the control unit 21 assigns a priority to each of the second group boundary boxes B2 to the fourth group boundary boxes B4 using consecutive numbers.

[0154] However, in the above situation, control unit 21 assigns the lowest priority to the group located in the center. That is, in Figure 7In this state, the control unit 21 assigns priority "1" to the third group of bounding boxes B3, priority "2" to the fourth group of bounding boxes B4, and priority "3" to the second group of bounding boxes B2.

[0155] In this way, during the positioning process S3, the control unit 21 can easily select the vicinity of the center of the continuous edge element ZF as the holding position.

[0156] Hereinafter, the processing method X of the second embodiment of the present invention will be described in detail with reference to the accompanying drawings. However, for content common to the first embodiment, the same reference numerals will be used and descriptions will be omitted.

[0157] <<Implementation Method Two>> like Figure 9 As shown, the processing system Y of this embodiment includes two operators 3. Correspondingly, in the positioning process S3, the control unit 21 also determines two gripping positions, i.e., sets of coordinates.

[0158] In the gripping and positioning process S32 of this embodiment, the control unit 21 determines the gripping positions of two points from a high-priority group. At this time, the gripping positions (a set of coordinates representing the points) are determined such that the two points to be gripped are spaced apart by a predetermined straight-line distance. The "predetermined straight-line distance" here is a length determined based on the size of the end effector 31, the size of the clamp of the downstream manipulator that grips the sheet Z in the next process of the present invention, the stretching length required by other subsequent processes, etc.

[0159] In the gripping and positioning process S32, there are cases where the length of the group selected according to priority is shorter than the "specified straight distance" used to determine the gripping position.

[0160] In such cases, the control unit 21 first changes the group used to the next priority. If the gripping position still cannot be determined even so, the control unit 21 uses the information related to the Z-shape of the sheet obtained in the detection process S2 (especially the brightness recognition process S21 and the shape estimation process S26) to determine the gripping position as a group of points (e.g., two points) with a specified distance along the surface in any group (outside the prohibited area) (presumably using an estimated value, but preferably a measured value if it can be measured). This distance along the surface is the distance between two points spaced at a specified distance along the surface of the sheet.

[0161] When the number of operators 3 in the processing system Y of this embodiment is increased to 3 or more, the configuration becomes such that the gripping position to be determined also increases accordingly with the number of operators 3. Regarding the processing of determining each point (coordinate group) of the gripping position in the gripping position determination process S32, the control unit 21 also performs the processing described above.

[0162] Explanation of reference numerals in the attached figures X processing method Y Processing System 1 Filming Department 2 Processing device 3 operators 4 Display Unit 5Transportation Department P program P1 Image Acquisition Unit P2 Feature Detection Department P3 Positioning Unit P4 Control Operation Section S1 Image Acquisition Process S2 Inspection Process S3 Positioning Process S4 Handling Operation Procedure F feature quantity B Boundary box Z-sheet ZF edge elements ZH edge

Claims

1. A processing method for causing a computer to execute a process of determining a proper gripping position in a sheet in an arbitrary state, wherein include: The image acquisition process involves photographing the sheet and acquiring image data. The detection process involves detecting the feature quantities contained in the image data; as well as In the positioning process, the grasping position is determined using the aforementioned feature quantity.

2. The processing method according to claim 1, wherein, In the aforementioned inspection process, the characteristic quantity is used to determine the segments of the edge elements possessed by the sheet. In the positioning process, the holding position is determined based on the segment.

3. The processing method according to claim 2, wherein, The edge features are provided along the edge of the sheet. In the detection process, the continuity of multiple segments is determined, and the segments that are determined to have continuity are grouped together. In the positioning process, the elements of the group are used to determine the holding position.

4. The processing method according to claim 2, wherein, The sheet material has edge elements that can be identified on a per-edge basis. In the detection process, each edge element is identified and grouped based on the feature values. In the positioning process, based on the feature quantities corresponding to each group, a point that meets the specified conditions is determined as the holding position.

5. The processing method according to claim 3, wherein, In the positioning process, when multiple gripping positions are determined, in the segment that is determined to have continuity, a group of points spaced a specified straight-line distance apart from each other is determined as the gripping position.

6. The processing method according to claim 3, wherein, In the positioning process, when multiple gripping positions are determined, in the segment that is determined to have continuity, a group of points spaced apart by a specified surface distance from each other are determined as the gripping positions.

7. A program, wherein, The computer functions as an image acquisition unit, detection unit, and positioning unit, determining a suitable holding position within the sheet material in any state. The image acquisition unit photographs the sheet and acquires image data. The detection unit detects the feature quantities contained in the image data. The positioning unit uses the feature quantity to determine the holding position of the sheet.

8. A processing system in which a computer performs a process to determine a suitable gripping position in a sheet in any state, wherein, The processing system includes: The image acquisition unit captures images of the sheet and acquires image data. The detection unit detects the feature quantities contained in the image data; and The positioning unit uses the aforementioned feature quantity to determine the gripping position.

Images