Item posture estimation method and program, bulk picking system

Abstract

A method and program for estimating the posture of an item, and a bulk picking system are provided that enable improved responsiveness. [Solution] A method for estimating the posture of an article 2, in which markers 14 are attached to the article 2 in a predetermined area suitable for a grasping operation to identify the posture of the article 2 at the grasping point, includes an imaging process of imaging multiple articles 2 that are stacked in bulk so that the markers 14 are included in the imaging range, a marker detection process of inputting the image of the article 2 imaged in the imaging process into a trained model 24 and obtaining a bounding box bb that includes the area where the markers 14 of the article 2 are present as an output result of the trained model 24, and an inclination angle calculation process of calculating the inclination angle θ of the marker 14 with respect to a predetermined reference line based on the vertical and horizontal lengths of the bounding box bb, and is executed by a computer.

Description

[Technical Field]

[0001] The present invention relates to a posture estimation method and program, and a bulk picking system for handling bulk items. [Background technology]

[0002] Traditionally, businesses engaged in the rental of various items have devoted a considerable amount of time and effort to inspecting returned items. For example, scaffolding rental businesses that rent out temporary scaffolding for construction sites inspect the components of returned scaffolding. While such inspection work is sometimes performed visually by rental company employees, efforts are also being made to automate and reduce labor using mechanical means. For example, as disclosed in Patent Document 1, a technology for automatically sorting long components such as single-tube pipes of different lengths is known. Furthermore, as disclosed in Patent Document 2, the present applicant has proposed a technology for counting small components such as clamps using optical detection means. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 49-63153 [Patent Document 2] Japanese Patent Application Publication No. 11-96329 [Patent Document 3] Japanese Patent Application Laid-Open No. 2005-273365 [Patent Document 4] Japanese Patent Application Laid-Open No. 2010-120141 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the objects to be inspected often have complex shapes and configurations. For example, in recent years, the construction industry has promoted the widespread use of so-called next-generation scaffolding, which incorporates a distinctive handrail component called a leading handrail, as shown in Patent Document 3, in order to improve the safety of construction site workers. The leading handrail that constitutes next-generation scaffolding is primarily composed of a cross member and a pair of support members that are movably attached to both ends of the cross member via link members, and the cross member and each support member are long cylindrical bodies such as single-tube pipes. Such leading handrails have a movable mechanism similar to a three-section stick and can take on a variety of shapes and configurations.

[0005] Furthermore, the items subject to inspection are often handled in a loose pile. For example, when the preceding handrail is returned in a loose pile, the support members at both ends are configured to be movable (both ends are movable), so the cross members, support members, and other components become entangled, and the postures of the components (direction, shape, etc.) are not uniformly determined.

[0006] When inspecting such bulk items by mechanical means, it is possible to apply a known bulk picking technique that includes a camera and a robot unit that are communicatively connected to a computer terminal, as shown in Patent Document 4. This robot unit is equipped with a gripper (grasping unit) that is an end effector for holding, transporting, etc., an object at the end of a manipulator, such as an articulated robot.

[0007] The gripper can have any suitable mechanism, such as a grasping gripper that holds an object with a multi-fingered hand or a non-grasping gripper that uses an electromagnet or air suction. Regardless of the type of gripper used, the robot unit contacts the gripper in a predetermined area suitable for grasping the object based on information detected by a camera, lifts the object, and transfers it. To ensure the reliability of this picking process, it is important to accurately estimate the orientation of the member (e.g., the cross member in the case of a leading handrail) to which the gripping point where the gripper contacts belongs, and to transfer the object reliably and in a balanced manner.

[0008] However, as described above, various items to be picked often have complex shapes and are often piled up in a random, tangled state. In such cases, accurately identifying the orientation of the items may require a complex calculation process, such as pre-storing item detection patterns for a wide variety of orientations and performing pattern matching or other calculations each time.

[0009] This increases the amount of calculation required for detecting an item, lengthening the cycle time from detecting the target item to the robot unit controlling the gripping operation, resulting in a problem of the system lacking responsiveness.

[0010] In view of the above problems, an object of the present invention is to provide an article orientation estimation method and program, and a bulk picking system that can improve responsiveness. [Means for solving the problem]

[0011] In order to solve the above problem, the posture estimation method according to claim 1 of the present invention is a method for estimating the posture of an article, wherein the article is provided with a marker in a predetermined area suitable for a grasping operation to identify the posture of the article at the grasping point, and includes: an imaging process of imaging a plurality of articles that are stacked in bulk so that the marker is included in the imaging range; a marker detection process of inputting the image of the article imaged in the imaging process into a trained model and acquiring, as an output result of the trained model, a bounding box that is partitioned to include the area where the marker of the article exists; and an inclination angle calculation process of calculating the inclination angle of the marker with respect to a predetermined reference line based on the length and width of the bounding box, and is executed by a computer.

[0012] In addition, in the posture estimation method according to claim 2, the tilt angle calculation step further includes a virtual marker setting step of setting a virtual marker of the same size as the marker at an internal reference position of the bounding box, a virtual marker rotation step of rotating the virtual marker by the tilt angle around its central coordinate as an axis, and an image feature amount calculation step of calculating image feature amounts in an existence area of ​​the virtual marker after the rotation.

[0013] In addition, the posture estimation method according to claim 3 further includes a candidate angle calculation step of calculating a plurality of patterns of tilt angles as candidate angles, the tilt angles being calculated according to an aspect ratio of the bounding box, the image feature amount being calculated including the degree of variation thereof; an image feature amount comparison step of performing the virtual marker rotation step and the image feature amount calculation step for each candidate angle and comparing the degree of variation of each image feature amount at each candidate angle; and a tilt angle determination step of uniquely determining, based on the result of the comparison, the candidate angle corresponding to a case where the degree of variation is relatively small, as the tilt angle.

[0014] In a posture estimation method according to claim 4, the marker is attached to a center of gravity area of ​​the article.

[0015] In a posture estimation method according to claim 5, the markers are attached all around the center of gravity area.

[0016] In a posture estimation method according to claim 6, the article includes a cylindrical portion, and the marker is attached to the cylindrical portion, so that the marker has a rectangular shape in a plan view.

[0017] In a posture estimation method according to claim 7, the tilt angle is an angle formed between a base line of the bounding box, which is the predetermined reference line, and a side of the marker in the longitudinal direction.

[0018] In the posture estimation method according to claim 8, the marker has visual characteristics including at least any one of a predetermined color, a predetermined pattern, and a predetermined texture for each type of the article.

[0019] In addition, in the posture estimation method described in claim 9, the trained model has been trained by associating the visual features of the marker with type information of the article, and in the marker detection process, the type information is output by the trained model.

[0020] A posture estimation program according to claim 10 causes a computer to execute each of the above methods.

[0021] In addition, the bulk picking system described in claim 11 comprises an information processing device configured to be able to execute the program, an imaging device, and a robot unit communicatively connected to the imaging device and the information processing device, and the robot unit includes a manipulator which is a multi-joint robot, and a gripping unit which is an end effector located at the end of the manipulator, and is configured to control the gripping unit in a posture suitable for contact action based on the tilt angle. [Effects of the Invention]

[0022] According to the inventions of claims 1 to 11, it is possible to improve the responsiveness of article posture estimation or bulk picking. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a diagram showing the configuration of a bulk picking system according to the present invention. [Figure 2] FIG. 10 is a diagram showing details of each part of the leading handrail. [Figure 3] FIG. 1 is a block diagram conceptually illustrating a hardware configuration of an information processing device. [Figure 4] FIG. 2 is a block diagram conceptually showing functions realized by the information processing device. [Figure 5] 10 is a flowchart showing the flow of a learning process. [Figure 6] 10 is a flowchart showing the flow of a process of estimating the posture of a leading handrail and picking up the handrail. [Figure 7] FIG. 1 is a diagram schematically illustrating the contents of an inclination angle calculation algorithm. [Figure 8] FIG. 10 is another diagram schematically illustrating the contents of the inclination angle calculation algorithm. [Figure 9] FIG. 2 is a diagram showing details of a robot unit. [Figure 10] 10A and 10B are diagrams illustrating the relationship between the operation of the robot unit and the tilt angle. [Figure 11] FIG. 2 is a diagram showing details of each part of the loosening machine. [Figure 12] FIG. 10 is a diagram showing details of a robot unit in a modified example. [Figure 13] 10A and 10B are diagrams illustrating the relationship between the operation and tilt angle of a robot unit in a modified example. DETAILED DESCRIPTION OF THE INVENTION

[0024] A posture estimation method according to the present invention is a method for estimating the posture of an article, in which the article is a target of a mechanical grasping operation. A marker for identifying the posture of the article at a grasping point (a point where a contact action is applied in the grasping operation) is attached to the article in a predetermined area suitable for the grasping operation (for example, an area of ​​the center of gravity of the article). To identify the posture of the article using the marker, it is preferable that the apparent shape of the marker differs depending on at least the rotation state of the article around a vertical axis.

[0025] The method also includes an imaging process of imaging multiple loosely stacked items so that the markers are included in the imaging range, a marker detection process of inputting the image of the items imaged in the imaging process into a trained model created using an object detection algorithm or the like, and obtaining a bounding box, which is a rectangular area defined to include the area where the markers of the items are present, as an output result of the trained model, and an inclination angle calculation process of calculating the inclination angle of the marker with respect to a predetermined reference line (the bottom side of the bounding box at the time of detection) based on the length and width of the bounding box (aspect ratio), and the method is executed by a computer.

[0026] As described above, the markers used have an apparent shape that varies at least depending on the orientation of the article. In other words, the aspect ratio of the marker when defined by a bounding box changes depending on the orientation of the article. To achieve this, for example, the distance between the outer edge and the center point of the marker needs to be different at some point. A variety of marker shapes are considered to generally satisfy this requirement, with the exception of extreme cases such as a perfect circle. However, a marker shape in which the aspect ratio of the bounding box defined by the marker changes significantly depending on the orientation of the article is more preferable. Even if a marker alone is unlikely to significantly change its aspect ratio depending on the orientation of the article, such as a perfect circle, it does not deviate from the spirit of the present invention as long as multiple markers are detected and some parameter representative of the orientation of the article can be obtained by appropriate calculation processing.

[0027] This makes it possible to accurately estimate the posture of an item even if the posture or shape of the items being stacked is uneven, for example, if the items are tangled and not uniform, by representing the posture of the detected item with the inclination angle of the marker, while reducing the amount of calculation required during detection.

[0028] Preferably, the tilt angle calculation step further includes a virtual marker setting step of setting a virtual marker of the same size as the marker at an internal reference position of the bounding box, a virtual marker rotation step of rotating the virtual marker by the tilt angle around its central coordinate as an axis, and an image feature calculation step of calculating image features in the area where the rotated virtual marker exists. The image features may include hue distribution, brightness, luminance, saturation, visual features, and various other quantitative and qualitative data. This makes it possible to verify whether the marker represents the orientation of the article with a required accuracy.

[0029] Preferably, the method further includes a candidate angle calculation step of calculating a plurality of patterns of tilt angles as candidate angles, the tilt angles being calculated according to the aspect ratio of the bounding box, including the degree of variation of the image features; an image feature comparison step of performing the virtual marker rotation step and the image feature calculation step for each candidate angle and comparing the degree of variation of each image feature at each candidate angle; and a tilt angle determination step of uniquely determining, based on the result of the comparison, the candidate angle corresponding to a case where the degree of variation is relatively small, as the tilt angle.

[0030] As described above, the aspect ratio of the bounding box that defines the marker is a parameter that indicates the posture of the object. Among the postures of the object represented by the marker, the rotational state of the object around the vertical axis is assumed to be a particularly important factor in the grasping operation, and this is understood as the tilt angle of the marker. However, since bulk-stacked objects can assume various rotational states not only around the vertical axis but also around the horizontal axis, the apparent shape of the marker at the time of detection varies depending on various factors, such as the shape of the marker, the degree of circumferential bias of the marker's affixed location, and the tilt state of the object's surface to which the marker is affixed. Therefore, it is natural to think that multiple (two or more) candidate tilt angle patterns may be calculated for the same value of the aspect ratio of a specific bounding box.

[0031] This situation is due to the fact that the method of the present invention does not directly detect the actual posture of the marker itself, but rather aims to reduce the amount of calculation processing by determining (estimating) the posture of the marker through the aspect ratio of the bounding box. Therefore, the applicant came up with the idea of ​​representing various existence patterns such as the posture and state of the marker according to the aspect ratio of the bounding box as virtual markers, and by determining the variation in image features in the area occupied by this virtual marker, it is possible to, so to speak, check whether the posture and state of the actual marker are appropriately represented.

[0032] As a result, in the more preferred embodiment of the present invention based on this idea, it is possible to verify whether the marker represents the orientation of the article with the required accuracy. In addition, even when multiple candidate angles compete and the differences in image features at each candidate angle are not particularly clear, the tilt angle can be uniquely determined by relative comparison, thereby improving flexibility during orientation estimation.

[0033] Preferably, the marker is attached to the center of gravity area of ​​the article, which enables posture estimation that is more suitable for grasping the article in a balanced manner.

[0034] Preferably, the markers are placed around the entire circumference of the center of gravity area. This makes the apparent shape of the marker less susceptible to the rotation state of the article around the horizontal axis, compared to when the markers are placed unevenly around the circumference. Therefore, the aspect ratio of the bounding box more accurately represents the tilt angle of the article (in other words, with a reduced number of possible patterns).

[0035] Preferably, the article includes a cylindrical portion, and the marker is attached to the cylindrical portion so that the marker has a rectangular shape in plan view regardless of the rotation state of the article about the longitudinal axis of the cylindrical portion. Here, the cylindrical portion includes, for example, a cylindrical or rectangular elongated body, and is a specific portion or component of the article corresponding to these. The diameter of the cylindrical portion (or the distance from the longitudinal axis to the outer periphery) is substantially equal around the entire circumference. When a marker is attached to such a cylindrical portion, the size of the marker in the short direction is determined according to the diameter of the cylindrical portion. Therefore, the size of the marker can be used as known information to smoothly set a virtual marker. More preferably, the length of the marker in the width direction is equal around the entire circumference, which allows the size of the marker to be uniquely determined with higher accuracy.

[0036] Preferably, the tilt angle is the angle between the base of the bounding box, which is the predetermined reference line, and the longitudinal side of the marker. This makes it clear that the tilt angle of the marker represents the longitudinal orientation of the article, which is more useful for optimizing the gripping operation.

[0037] Preferably, the marker has visual characteristics including at least one of a predetermined color, a predetermined pattern, and a predetermined texture for each type of the object, thereby enabling the type of object to be identified by identifying the visual characteristics, thereby enabling accurate identification of the type of object while reducing the computational load during detection.

[0038] Each of the above methods is described as a computer program and implemented in an information processing device of a bulk picking system. This bulk picking system includes an imaging device and a robot unit communicatively connected to the imaging device and the information processing device. The robot unit includes a manipulator that is an articulated robot and a gripper that is an end effector located at the end of the manipulator, and is configured to control the gripper in a posture suitable for contact based on the tilt angle. The gripper is configured to grip and transport the object to be picked for picking processing by gripping with a hand mechanism, suction by air or electromagnet, or other appropriate gripping method.

[0039] More preferably, in the above-mentioned bulk picking system, the specific configuration for controlling the gripper in a posture suitable for contact action may be, for example, the following configuration A or B, which allows the gripper to apply the contact action of the gripper to the gripping point of the item with as little loss as possible, enabling more stable picking processing. A) A configuration applicable to an item that includes a cylindrical portion and has a marker of the same width attached around the entire circumference of the center of gravity area of ​​the cylindrical portion, the gripper includes a pair of claws, and controls the gripper in a posture in which the longitudinal direction of the marker is perpendicular to the direction of the contact action of the claws. B) A configuration applicable to an item that includes a cylindrical portion and has a marker of the same width attached around the entire circumference of the center of gravity area of ​​the cylindrical portion, the gripper includes a rectangular electromagnet, and controls the gripper in a posture in which the electromagnet and the marker fit together (a posture in which one side of the electromagnet and the longitudinal direction of the marker are perpendicular or parallel). [Example]

[0040] An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram showing the configuration of a bulk picking system 1 (hereinafter sometimes referred to as "this system") according to the present invention, which is applied to a picking process including grasping and transferring a leading handrail 2 in this embodiment. This system 1 includes a two-dimensional camera (imaging device) 3, a TOF camera (height detection device) 4, a robot unit 5, and an information processing device 6 communicatively connected to these via wired or wireless communication. The robot unit 5 is composed of a multi-joint manipulator 51 and a gripper (grasping unit) 52, which is an end effector provided at the end of the manipulator. Details of the robot unit 5 will be described later.

[0041] The system 1 is equipped upstream in the picking process with a loosening device 7 that untangles (disentangles) multiple leading handrails 2 that become tangled when returned. The loosening device 7 can be, for example, a known chain conveyor, a vibration transport mechanism (vibrator), or various strikers. The returned leading handrails 2 are sequentially fed to the loosening device 7, and after being loosened during the transport process, are discharged toward the detection area 8 below the two-dimensional camera 3 and the TOF camera 4 and piled up in bulk. Details of the loosening device 7 will be described later.

[0042] The system also includes a light-shielding hood 9 that covers the detection area 8, and a light source 10 that illuminates the interior of the hood 9. The hood 9 houses a loosely stacked leading handrail 2, a two-dimensional camera 3, and a TOF camera 4. The TOF camera 4 acquires a distance image of the subject using a time-of-flight (TOF) method. The hood 9 blocks ambient light, such as sunlight, which can be a disturbance, thereby suppressing adverse effects on the measurement accuracy of the TOF camera 4. The light source 10 maintains the required brightness of the detection area 8 inside the light-shielded hood 9. In this embodiment, LED lighting is used as the light source 10, which is less likely to flicker, thereby suppressing disturbances during detection.

[0043] FIG. 2 is a diagram showing the details of each part of the leading handrail 2. The leading handrail 2, which is an item to be picked by the system 1, is mainly composed of a cross member 11 and a pair of support members 12. The cross member 11 and the support members 12 are pipe members (long cylindrical bodies) of an appropriate size, and the base ends of the support members 12 are pivotally attached to both ends of the cross member 11 so as to be able to move freely via link mechanisms 13. In addition, locking hooks 111 are provided on both side portions of the cross member 11, and wedge members 121 are provided at the end ends of the support members 12. When the leading handrail 2 is installed (not shown), the pair of support members 12 are crossed, and the locking hooks 111 and the wedge members 121 can be fitted and fixed to perforated flange portions (not shown) of support pipes (not shown) located on upper floors.

[0044] The leading handrails 2 are piled in bulk in a mixture of multiple types (classifications) according to various specifications such as the size of each component. In this embodiment, there are two types, type A and type B, according to the overall length of the cross member 11. There may be three or more types of leading handrails 2. The specifications and types of the leading handrails 2 are not limited to this embodiment, and the details are determined appropriately according to various standards and the product lineup of each temporary equipment manufacturer.

[0045] A marker 14 is attached to the cross member 11 of each leading handrail 2 in the longitudinal center portion. The longitudinal center portion of the cross member 11 where the marker 14 is attached is roughly the center of gravity area of ​​the leading handrail and is suitable for grasping by the robot unit 5, which will be described later. The markers 14 are painted in appropriate colors depending on the type of leading handrail 2. In this embodiment, the markers 14 are colored red and yellow depending on the type, A, or B, of each leading handrail 2. The markers 14 are attached around the entire circumference of the cross member 11 (pipe material) and have an appropriate length in the longitudinal direction. Note that the markers 14 may be made of vinyl tape, colored spray paint, or other appropriate paint of various colors. In addition to the central portion, supplementary markers 14 of the same color or a different color may be attached to both ends of the cross member 11.

[0046] FIG. 3 is a block diagram conceptually illustrating the hardware configuration of the information processing device 6. The information processing device 6 is configured by a computer having a processor 15 such as a CPU or GPU, a memory 16 such as RAM, and an auxiliary storage device 17 such as a hard disk drive or SSD. Computer programs for executing the object detection process and robot unit control process described below are installed in the auxiliary storage device 17. The information processing device 6 is also communicatively connected to the imaging device (two-dimensional camera) 3, height detection device (TOF camera) 4, and robot unit 5 via an input / output device 18. The robot unit 5 is communicatively connected to the information processing device 6 via a PLC 19, but in the following description, the functions of the PLC 19 are treated as being included in the information processing device 6 and are treated as a single entity. The means for communicatively connecting the components of the information processing device 6 may be wired or wireless.

[0047] Some or all of the functions of the information processing device 6 may be implemented on a cloud server. For example, the information processing device 6 may execute various arithmetic processes, program and data reading processes, storage processes, etc., using a virtual machine on a server physically separated from the present system 1.

[0048] 4 is a block diagram conceptually showing the functions realized by the information processing device 6. The information processing device 6 includes a detection unit 20, a calculation unit 21, a control unit 22, and a learning unit 23. The functions of these units are realized by the computer serving as the information processing device 6 operating in accordance with a computer program stored in an auxiliary storage device 17 or the like.

[0049] The detection unit 20 is a processing unit that further includes a trained model 24 and detects the markers 14 of the leading handrail 2 contained in image data captured by the two-dimensional camera 3, based on the image data of the leading handrail 2. When image data of the leading handrail 2 captured and including the markers 14 is input to the trained model 24, it outputs a rectangular bounding box that defines the area where the markers 14 exist. Each side of the bounding box is output so as to be in contact with the outer edge of the area where the markers 14 exist, and the trained model 24 has been trained by machine learning in advance by the learning unit 23 so as to obtain this output result. Details of the machine learning will be described later together with a separate flowchart.

[0050] The output result of the trained model 24 of the detection unit 20 includes information on the center coordinates (two-dimensional coordinates) of the rectangular area occupied by the bounding box, as well as the length and width. The trained model 24 also outputs the color discrimination result of the marker 14, along with a reliability score that numerically represents the likelihood of the color discrimination result and the partition accuracy of the bounding box, between 0 and 1. Furthermore, height information of the object within the detection area 8 acquired by the TOF camera 4 is input to the detection unit 20. The height information is the vertical distance (depth) from the TOF camera 4 to the object, and is measured based on the time delay between irradiating the object with light from the TOF camera 4 and capturing the reflected light.

[0051] The calculation unit 21 is a processing unit that calculates control parameters for the robot unit 5 based on various information acquired by the detection unit 20 in accordance with a predetermined algorithm. The calculation unit 21 calculates the tilt angle of each marker 14 with respect to a predetermined reference line (in this embodiment, the bottom side of the bounding box) based on the aspect ratio of the bounding box acquired by the detection unit 20, and sends this as a control parameter to the control unit 22. The details of the calculation algorithm for this tilt angle will be described later.

[0052] The calculation unit 21 also links the center coordinates of each marker 14 with the height information acquired by the TOF camera 4, and calculates three-dimensional coordinates that are candidates for the grasping point (pickup position) where the robot unit 5 makes contact with the leading handrail 2. Furthermore, the calculation unit 21 compares the coordinates of the calculated candidate grasping points, selects the grasping point that is located at the highest point (the shallowest point), and sends the coordinates to the control unit 22 as control parameters. Note that the grasping point of the leading handrail 2 that is the target of the picking process corresponds, in principle, to the coordinates of the cross member 11 stacked on top, but if there are multiple competing candidates for the grasping point, it is advisable to select the coordinate with the highest reliability score as the grasping point.

[0053] The control unit 22 is a processing unit that controls the operation of the robot unit 5 based on control parameters such as the tilt angle and the three-dimensional coordinates of the gripping point calculated by the calculation unit 21. The control unit 22 controls the robot unit 5 based on the tilt angle corresponding to the selected gripping point. This control includes rotationally driving the movable parts of the robot unit 5 so that the robot unit 5 can reliably grip the gripping point of the leading handrail 2 in a state and positional relationship that allows the robot unit 5 to stably transmit force.

[0054] The learning unit 23 is a processing unit that executes creation of the trained model 24. The learning unit 23 performs appropriate machine learning based on predetermined data related to the leading handrail 2, and creates the trained model 24. In this example, supervised machine learning is performed based on an image of the leading handrail 2 including the markers 14, and a training dataset including various kinds of label information, and the specific execution procedure will be explained below using a flowchart.

[0055] 5 shows the flow of the learning process by the learning unit 23. The capital letter S in parentheses indicates a step. First, a marker 14 is attached to the center of the longitudinal direction of the cross member 11 of the leading handrail 2 (S1).

[0056] In this example, there are two types of leading handrails 2, with longitudinal lengths of the cross member 11 of 1800 mm (Type A) and 1500 mm (Type B), and they are colored according to type, with Type A being colored with red markers 14 and Type B being colored with yellow markers 14. For coloring, paint is applied to the entire circumference of the long cylindrical body (pipe material) that makes up the cross member 11. When viewed horizontally in a plan view (when the cross member 11 is placed horizontally and viewed from directly above), the short-side length of each type of marker 14 is just under 30 mm (approximately 28 mm), and the longitudinal length is approximately 150 mm.

[0057] In this way, by having the markers 14 have a suitable length in the longitudinal direction, the difference in the aspect ratio of the bounding box after the marker 14 is more likely to be a characteristic, which is advantageous in the subsequent calculation of the inclination angle. The length of each marker 14 in the longitudinal direction of the cross member 11 is appropriately determined so that it is at least sufficiently longer (in this embodiment, at least five times longer) than the length of the marker 14 in the transverse direction, which is determined according to the diameter of the cross member 11. According to tests conducted by the inventors, when the longitudinal length of the marker 14 attached to the same leading handrail 2 as above was 100 mm or less, the accuracy of estimating the inclination angle significantly decreased. Therefore, it is considered desirable that the longitudinal length of the marker 14 be at least three times the length in the transverse direction.

[0058] Next, learning image data of the leading handrail 2 is acquired (S2). Specifically, the various leading handrails 2 are imaged with the two-dimensional camera 3 to acquire learning image data, including the markers 14 of each color. At this time, disturbances from natural light can be suppressed by performing the image capture within the hood 9, similar to the environment during detection. It is desirable to acquire a reasonable amount of learning image data for multiple types of leading handrails 2 that are piled up in a wide variety of shapes and postures, as is expected during actual detection. Furthermore, if necessary, a single leading handrail 2 in a piled up state may be extracted, and data imaged in various shapes and postures may be used as learning image data.

[0059] Next, the training image data is annotated (S3). Specifically, the presence area of ​​each color marker 14 included in the training image data is defined by a rectangular bounding box. Annotation may be performed manually or automatically. During annotation, the bounding box is defined so that each side of the bounding box is in contact with the outer edge of the presence area of ​​the marker 14, so that each side of the bounding box output from the trained model 24 also contacts or is very close to the presence area of ​​the marker 14, just as during training. This improves the reliability of various data used as the basis for control parameters, such as the aspect ratio and center coordinates of the rectangular region of the bounding box.

[0060] Next, image processing is performed on the annotated learning image data (S4). Image processing includes inversion, rotation, etc. This allows the learning image data to be expanded to include a wider variety of shapes and postures of the leading handrail 2, improving the versatility of the trained model 24 created based on that data. Note that the order of S3 and S4 may be reversed, and S4 may be omitted as appropriate.

[0061] Next, the color information of the markers 14 is further linked as a label to the annotated learning image data to create a learning dataset (S5). In this embodiment, color information is linked as a label to each image data, for example, (image 1, red), (image 2, yellow), (image 3, ...). At this time, as described above, the color information of the markers 14 is associated with the type of the preceding handrail 2, such as (red, type A) or (yellow, type B), so the type of the preceding handrail 2 can be determined by distinguishing the color of the markers 14.

[0062] Furthermore, because the leading handrail 2 with floating ends has a wide variety of shapes and postures when piled up (at the time of detection), if type discrimination is performed directly based on the external shape alone, the required amount of learning image data and the amount of calculation required during detection may become excessive. On the other hand, color discrimination makes it easier to ensure detection accuracy when a trained model 24 is created using the same amount of learning image data, and the amount of calculation required during detection is relatively reduced. Therefore, in this embodiment, by replacing type discrimination with color discrimination, the amount of learning image data and the amount of calculation required during detection can be reduced, thereby contributing to a shortened cycle time until detection data is sent to the control unit 22 and improving the responsiveness of the present system 1.

[0063] Next, the created learning dataset is input into a predetermined machine learning algorithm to perform supervised machine learning (S6). In this embodiment, the object detection algorithm YOLO (You Only Look Once) is used for the machine learning. In addition, SSD (Single Shot MultiBox Detector) or various other object detection algorithms using deep learning, convolutional neural networks, etc. may also be used.

[0064] Finally, the estimation accuracy of the trained model 24 after machine learning processing is confirmed (S7). Specifically, image data (without annotation) of the leading handrail 2 for verification is input into the trained model 24 after machine learning processing, and it is verified whether the bounding box properly defines the area in which the marker 14 exists in the image. When the verification results are good and a trained model 24 with sufficient estimation accuracy is obtained, the various learning parameters at that stage are adopted and the trained model 24 is finalized. The trained model 24 created and finalized through the above steps S1 to S7 is incorporated into the detection unit 20 of the information processing device 6.

[0065] 6 shows the flow of the picking process and the posture estimation of the leading handrail 2. In the following, a series of procedures for posture estimation of the leading handrail 2 and bulk picking by this system 1 will be described using the flowchart.

[0066] First, various leading handrails 2 (with markers 14 of each color attached) returned from a construction site or the like are fed into the loosening device 7 (S101). As a result, the various leading handrails 2 that are tangled in various forms are given kinetic energy such as vibration, impact, and other external forces, and are discharged into the detection area 8 in the hood 9 in a loosened (loosened) state compared to before they were fed in.

[0067] Next, the two-dimensional camera 3 captures an image of the leading handrail 2 in the detection area 8, including the marker 14, to obtain image data (S102). The TOF camera 4 also obtains height information for each point in the detection area 8, including the leading handrail 2, particularly the marker 14 (S103). The image data and height information data are sent to the detection unit 20.

[0068] Next, the image data is input to the trained model 24 of the detection unit 20 (S104), and a bounding box that defines the area where each marker 14 exists and a color discrimination result are obtained as output results (S105). The acquired information includes the length and width of the bounding box and the center coordinates, and is sent to the calculation unit 21 together with the color discrimination result. Note that the color discrimination result (type of leading handrail 2) is information used in the sorting process after the picking process, and it is not necessarily necessary to use the color discrimination result for operation control of the robot unit 5 (for example, optimization of the gripping operation according to the type).

[0069] Next, the calculation unit 21 calculates the tilt angle, which is a control parameter for the robot unit 5, in accordance with the following calculation algorithm (S106). In explaining the tilt angle calculation algorithm, steps S106-1 to S106-6 will be explained separately, and the schematic diagrams in Figs. 7 and 8 will be referred to as appropriate.

[0070] As shown in FIG. 7(a), each side of the bounding box bb contacts the outer edge of the area where the marker 14 exists. By obtaining the bounding box bb, the vertical length L, horizontal length W, and center coordinates (X, Y) of the rectangular area are also obtained. Furthermore, by obtaining the color discrimination result, type information of the leading handrail 2 is obtained. A reliability score (not shown) is also obtained. The vertical length s and horizontal length t of the marker 14 in a horizontal plan view are known values.

[0071] The tilt angle θ is the angle between the longitudinal direction of the marker 14 and the direction of the aforementioned reference line, with the base of the bounding box bb as the reference line (the tilt of the longitudinal direction of the marker 14 with respect to the reference line). Note that the center coordinates of the bounding box bb match the center coordinates of the marker 14, or fall within an error range that allows them to be considered to be approximately the same.

[0072] The first step of the calculation algorithm is to calculate a plurality of candidate angles for the tilt angle based on the aspect ratio L / W of the bounding box bb (S106-1).

[0073] The tilt angle (candidate angle) can be calculated by, for example, using polar coordinates and trigonometric ratios to express the ratio of geometric quantities including the length and width L and W of the bounding box bb as a function of angle, and substituting the detected values ​​of the actual length and width L and W. In this embodiment, when the values ​​of the length and width s and t, which are known size information of the marker 14 in a horizontal planar view, are further used, the tilt angle (candidate angle) is calculated as θ=arctan{(sW-tL) / (sL-tW)}.

[0074] As shown in Figure 7(b), when the angle calculated based on the length ratio is θ, there are two candidate angles. In the following description, for convenience, the two candidate angles will be referred to as θa and θb. In this case, the actual existence pattern of the marker 14 can be one of two: a right-upper pattern corresponding to θa, and a left-upper pattern corresponding to θb.

[0075] In the step following S106-1, a virtual marker 25 of the same size as the marker 14 is set to its initial position (the internal reference position of the bounding box) (S106-2). The virtual marker in its initial position is in the state shown in Fig. 7(c). The marker 14 present in the background is omitted.

[0076] The size of the virtual marker 25 in this embodiment is approximately 28 mm x 150 mm (s x t), similar to the size of the marker 14 described above. The center coordinates of the virtual marker 25 and the center coordinates of the bounding box are the same. As previously explained, the short-side length s of the virtual marker 25 is determined based on the pipe diameter of the cross member 11, and the long-side length t is set to a predetermined multiple of the short-side length s. Note that the apparent shape and size of the marker 14 vary depending on the vertical tilt of the marker 14 (it is not necessarily an accurate rectangle). Therefore, the size of the virtual marker 25 is determined based on the rectangular shape of the marker 14 in a horizontal planar view (when the cross member 11 is placed horizontally in the detection area 8 and viewed from directly above), and this size is used as a known value for various calculation processes.

[0077] At its initial position, the virtual marker 25 is contained in a rectangular area defined by a bounding box bb. At the initial position, the base of the bounding box, which serves as the reference line, is parallel to the longitudinal direction of the virtual marker. That is, the tilt angle θ of the virtual marker 25 at the initial position is 0°.

[0078] In the step following S106-2, the virtual marker 25 is rotated from its initial position by an amount equivalent to the assumed tilt angle θ of the actual marker 14 (S106-3). As shown in FIG. 8(d), in this embodiment, the rotation is performed in two patterns, a counterclockwise rotation pattern (θa) and a clockwise rotation pattern (θb), around the central coordinates of the virtual marker 25 (which coincide with the central coordinates of the bounding box). Note that the marker 14 present in the background is omitted from the figure. The thick arrow in the figure indicates the rotation direction of the virtual marker 25.

[0079] Next, the hue distribution (image feature amount) of the area occupied by the rotated virtual marker 25 is calculated (S106-4). In this embodiment, the hue distribution of the virtual area α occupied by the virtual marker 25 in the counterclockwise rotation pattern (θa) of Fig. 8(d) <C> and the hue distribution of the virtual area β occupied by the virtual marker 25 in the clockwise rotation pattern (θb) of Fig. 8(d) are calculated. In the figure, the outer edge of the virtual marker 25 is represented by a dashed line, and the virtual areas α and β occupied by the virtual marker 25 are represented by hatching.

[0080] Next, the standard deviations of the hue distributions of the virtual regions α and β occupied by the rotated virtual marker 25 are compared (S106-5). In this embodiment, the standard deviation value α' of the hue distribution of the virtual region α corresponding to the counterclockwise rotation pattern (θa) and the standard deviation value β' of the hue distribution of the virtual region β corresponding to the clockwise rotation pattern (θb) are calculated, and the magnitudes of these values ​​are compared.

[0081] In this embodiment, among the patterns shown in FIG. 7(b), the actual presence pattern of the marker 14 is assumed to be the right-top pattern. Below, we will switch to FIG. 8(e), which does not omit the actual presence pattern of the marker 14. As shown in FIG. 8(e), the virtual region α corresponding to θa is superimposed so as to nearly coincide with the actual presence region of the marker 14. This minimizes color variation in the virtual region α, and the standard deviation α' of the hue distribution is significantly low. On the other hand, as shown in <H>, the virtual region β corresponding to θb intersects with the actual presence region of the marker 14. This results in a significant variation in color between the overlapping common region and the non-overlapping background region, and a significantly high standard deviation β' of the hue distribution. Therefore, the standard deviations α' and β' of the hue distribution serve as indicators of whether each candidate tilt angle is consistent with the actual presence pattern of the marker.

[0082] In the step following S106-5, the tilt angle corresponding to the pattern with the lowest standard deviation value of the hue distribution as a result of the above comparison is uniquely determined as the picking angle to be used in the picking process (S106-6). In the example of Figure 7, since the relationship α'<β' holds, θa corresponding to the pattern (A) with the lower standard deviation value of the hue distribution and the upper right presence of the marker is determined (adopted) as the picking angle.

[0083] If the cross member 11 is tilted in the height direction, the actual area of ​​the marker 14 on the image will not necessarily be rectangular due to the fact that it is projected onto a plane, and the apparent shape may be distorted (it may be trapezoidal or some other shape). In such cases, even the hue distribution corresponding to the actual pattern of the marker 14 may vary to some extent. However, by comparing the standard deviation values, the angle with the least variation can be adopted as the picking angle, improving the versatility of the present system 1.

[0084] It is assumed that the tilt angle has been calculated (confirmed as the picking angle) in each of steps 106-1 to 106-6. In the next step, the calculation unit 21 associates the height information corresponding to each coordinate acquired by the TOF camera 4 with each of the two-dimensional coordinates (X, Y) that are the common center coordinates of each bounding box and each marker 14, and acquires the three-dimensional coordinates (X, Y, Z) of the center point of each marker 14 as a candidate for the gripping point (S107). The three-dimensional coordinates of the candidate for the gripping point are associated with each picking angle and sent to the control unit 22.

[0085] The control unit 22 selects the center coordinates of the marker 14 corresponding to the coordinates of the highest point (largest Z value) among the candidate gripping points as the gripping point of the leading handrail 2 (S108). Next, the control unit controls the robot unit 5 based on the picking angle linked to the coordinates of the selected gripping point (S109). In this embodiment, the position coordinates of the gripping point of the highest leading handrail 2 to be picked are selected, and the operation of the robot unit 5 is controlled based on the picking angle corresponding to the coordinates so that the angle is such that the robot unit 5 can reliably grip the leading handrail 2.

[0086] First, the detailed configuration of the robot unit 5 that realizes the above-mentioned operation will be described. As shown in Fig. 9, the robot unit 5 employed in this embodiment is a six-axis articulated robot, and is composed of a manipulator 51 and a gripper 52, which is an end effector for gripping operation provided at the end of the manipulator 51. The operation of the robot unit 5 consisting of the manipulator 51 and the gripper 52 is controlled by the control unit 22 based on control parameters such as the picking angle calculated by the calculation unit 21.

[0087] The manipulator 51 includes a manipulator base 511 fixed to an installation surface, a body 512, a first arm A1, a second arm A2, and a third arm A3. The arms A1 to A3 are connected by a first joint J1, a second joint J2, a second joint J3, a third joint J3, a fourth joint J4, a fifth joint J5, and a sixth joint J6 so that each can rotate independently.

[0088] Specifically, the first joint J1 rotates the body 512 about a vertical axis, the second joint J2 rotates the first arm A1 about a horizontal axis, the third joint J3 rotates the second arm A2 about a horizontal axis, the fourth joint J4 rotates the second arm A2 about its longitudinal axis, the fifth joint J5 rotates the third arm A3 about an axis perpendicular to the rotation direction of the fourth joint J4, and the sixth joint J6 rotates the gripper 52 provided at the end thereof about the longitudinal axis of the third arm A3.

[0089] As described above, the gripper 52 for performing the gripping operation is connected to the sixth joint J6 located at the end of the manipulator 51. In this embodiment, the gripper 52 is a gripping hand having a gripper base 521 and a pair of claws 523 that are movably connected to the gripper base 521 via a movable connecting portion 522.

[0090] In this way, the manipulator 51 is configured to be able to independently rotate each of the arms A1 to A3 by each of the joints J1 to J6, thereby guiding the gripper 52 located at the end of the manipulator 51 to the required position where the leading handrail 2, which is the object to be picked, is located, and perform picking processes such as grasping and transferring. Note that the operating modes of the robot unit described above are merely examples, and the configuration such as the number of axes (degrees of freedom) of the manipulator is not limited to this embodiment, and any appropriate configuration can be adopted as long as the above object can be achieved.

[0091] Then, in controlling the robot unit 5 (S109) after determining the gripping point of the leading handrail 2, the control unit 22 controls the rotation of each of the joints J1 to J6, which are rotatable in each direction, in conjunction with each other, to move the gripper 52 at the end of the manipulator 51 (end of the third arm A3) in the required position and posture to the vicinity of the gripping point of the leading handrail 2, which is the object to be picked. Next, the control unit 22 brings the opposing claws 522 closer together (or apart as necessary) to bring the gripper 52 into contact with the gripping point (center coordinates of the marker 14) of the leading handrail 2, and grips it with the required force.

[0092] 10(a) and 10(b) show the positional relationship between the gripper 52 and the leading handrail 2 during a picking process including a gripping operation. Note that each component of the gripper 52 is shown by a dashed line (imaginary line), and other components of the robot unit 5 are omitted.

[0093] 10(a) shows the positional relationship between the gripper 52 and the leading handrail 2 that is suitable for gripping operation. As shown in the figure, in a plan view, the gripper 52 is positioned above the longitudinal center (area marked with marker 14) of the cross member 11 that constitutes the leading handrail 2, and the opposing direction of the pair of claws 523 of the gripper 52 is perpendicular to the longitudinal axis direction of the cross member 11, which is a suitable state for the contact action of the gripper 52. In this state, the longitudinal axis direction of the cross member 11 extends perpendicular to the moving direction of the claws 523, and the gripping force of the gripper 52 of the robot unit 5 is efficiently transmitted to the cylindrical cross member 11. This allows the leading handrail 2 to be securely gripped.

[0094] Figure 10(b) shows the positional relationship between the gripper 52 and the leading handrail 2 during actual bulk picking. In the figure, two types of leading handrails 2, 2' (including cross members 11, 11' of different lengths) are piled on top of each other in bulk, with the 1800 mm (type A) leading handrail 2 at the top and the 1500 mm (type B) leading handrail 2' at the bottom. The two types of leading handrails 2, 2' are colored according to type, with type A indicated by a red marker 14 and type B indicated by a yellow marker 14', and the center coordinates of the red marker 14 located at the highest point are selected as the gripping point, with the picking angle (tilt angle) being θa.

[0095] Therefore, the control unit 22 controls the drive of the robot unit 5 so that the center coordinates of the marker 14, which is the gripping point, are set as the target position and the robot unit 5 assumes a posture that matches the picking angle θa. That is, starting from various initial positions of the robot unit 5, a series of controls are executed with the position state shown in Fig. 10(b) as the target.

[0096] Specifically, for example, first, each arm A1 to A3 of the manipulator 51 is rotated via each joint J1 to J6 to align the two-dimensional coordinates (X, Y coordinates) of the center of the marker 14 with the center position of the gripper 52. The gripper 52 is then lowered to a position near the target object that corresponds to the height (Z coordinate) of the marker 14. The gripper 52 is then rotated via the sixth joint J6 by an amount equivalent to the picking angle (tilt angle) θa of the marker 14, so that the facing direction of the claws 523 of the gripper 52 is perpendicular to the long side of the marker 14. Finally, the connecting portion 522 is driven to narrow the gap between the claws 523, and the claws 523 come into contact with and grip the leading handrail 2. In this way, the control unit 22 controls the drive of the robot unit 5 so that the leading handrail 2 can be gripped in the appropriate position and posture according to the picking angle (tilt angle) θa. The specific mode of control is determined by calculation processing each time depending on the state of the pile of the leading handrail 2, the initial position of the robot unit 5 at the time of detection, and the like.

[0097] At this time, the gripping point where the gripper 52 makes contact is in the center of gravity area marked with the marker 14, and the gripper 52 can make contact at the gripping point included in this area at a picking angle θa that matches the posture of the leading handrail. As a result, the force is transmitted to the leading handrail 2 with as little loss as possible (efficiently) to grip, and the picking process can be performed reliably by holding and transferring the leading handrail 2 while maintaining balance.

[0098] In this embodiment, the tilt angle in the height direction among the three dimensions is not included in the detection target, so a certain amount of error may occur in the height direction, but if the error is at the level caused by the overlap of the preceding handrail 2, it can be grasped without any problems.

[0099] After the robot unit 5 grasps the leading handrail 2 through the above control (S110), it continues to hold the leading handrail 2 with the gripper 52 and transfers it to a predetermined position (S111). In the transfer process, for example, based on the type of leading handrail 2 linked to the color information of the marker 14, the robot unit 5 transfers various leading handrails 2 to an appropriate position and sorts and selects them. This completes a series of inspection tasks, including the picking process of the leading handrail 2, by this system 1.

[0100] 11 is a plan view showing the details of each part of the loosening device 7. One embodiment of the loosening device 7 will be described with reference to FIG.

[0101] In this embodiment, the disentangling device 7 comprises a support section 30, a pair of rail members 31 suspended over the support section 30, and a chain unit 32 with a transport function that is provided along each rail member 31, and is configured to sequentially supply multiple leading handrails 2 onto the chain unit 32, place the leading handrails 2 so that they are suspended in the short direction of the chain unit 32, and be able to send out in the long direction of the chain unit 32. As a result, leading handrails 2 that have become tangled when returned from the site are untangled in the process of being transported to the disentangling device 7 compared to before they were supplied, making it relatively easy to estimate the posture of the leading handrail 2 in the subsequent processing steps and reducing the risk of the manipulator failing to grasp it.

[0102] The present invention described above is not limited to the above-described embodiments, and some of the elements appearing in the above-described embodiments may be deleted or combined without departing from the spirit of the present invention.

[0103] For example, the gripping section of the manipulator 51 of this embodiment is configured to grip an object by providing a gripper 52, which is a gripping hand, at the end of the manipulator 51. However, the gripping section may also be configured to include one or more electromagnets at the end of the manipulator 51 to attract an object.

[0104] FIG. 12 shows details of a modified robot unit 5 in which an electromagnet is used as the end effector. The difference from FIG. 9 is that the electromagnet unit 52', which is the gripping section, comprises a gripper base 521' and a pair of rectangular electromagnets 524 connected to the gripper base 521'. The electromagnet 524 is configured so that it can grip / release an object at the required timing by being switched between conductive and non-conductive by the control unit 22. The electromagnet unit 52' is used when the leading handrail 2 is made of a magnetic material such as iron. Note that the other configuration of the robot unit 5 in this modified example is the same as that in FIG. 9, so a description thereof will be omitted.

[0105] 13(a) and 13(b) show the positional relationship between the gripper 52' ​​and the leading handrail 2 in a picking process including a gripping operation in a modified example. Note that each component of the gripper 52' ​​is shown by a dashed line (imaginary line), and other components of the robot unit 5 are omitted.

[0106] 13(a) shows the positional relationship between the electromagnet unit 52' and the leading handrail 2 that is suitable for gripping operation. As shown in the figure, in a plan view, the electromagnet unit 52' is positioned above the longitudinal center (area marked with marker 14) of the horizontal member 11 that constitutes the leading handrail 2, and the opposing direction of the pair of electromagnets 524 of the electromagnet unit 52' is parallel to the longitudinal axis direction of the horizontal member 11, which is the optimum state for the contact action of the electromagnet unit 52'. In this state, particularly when a rectangular electromagnet 524 is used, the shape of the electromagnet 524 fits perfectly with the longitudinal axis direction of the horizontal member 11, so the gripping force of the gripper 52 of the robot unit 5 is efficiently transmitted to the horizontal member 11, which is a long body. This allows the leading handrail 2 to be gripped reliably.

[0107] Figure 13(b) shows the positional relationship between the electromagnet unit 52' and the leading handrail 2 during actual bulk picking, and is the same as Figure 10(b) except that the gripping operation is performed by the direction of action of the gripping part (parallel to the longitudinal axis direction of the cross member 11) and the conduction of the electromagnet 524, so explanation will be omitted.

[0108] Furthermore, the gripping unit may be an air suction mechanism, adhesive tape, or other mechanism as appropriate. Therefore, the picking process including the gripping operation in the technology of the present application encompasses various embodiments that can be realized by these various types of gripping units. Even when various gripping units are used, there is a suitable positional relationship between the gripping unit and the leading handrail 2 that is suitable for the picking process, and it is considered effective to control the robot unit 5 to move to such a suitable position based on the tilt angle (picking angle) that represents the posture of the leading handrail.

[0109] In addition, for example, in this embodiment, a TOF camera is used as the height detection device, but LiDAR or other optical measurement devices may also be used as the height detection device.

[0110] In this embodiment, the two-dimensional camera and the TOF camera are separately provided, but by adopting a three-dimensional camera, a point cloud camera (3D laser scanner), or the like, the functions of the imaging device and the height detection device may be integrated into one device.

[0111] In this embodiment, the marker is attached to the longitudinal center of the horizontal member of the leading handrail. However, the marker may be attached to any suitable area suitable as a gripping point, such as the center of gravity area of ​​the horizontal member or the entire leading handrail. Markers may be of various shapes and patterns, as well as colors, as long as they can be easily identified by an object detection algorithm.

[0112] In this embodiment, for the sake of convenience of explanation, two types of leading handrails 2 are distinguished. However, it is also possible to distinguish more types of leading handrails 2. For example, in accordance with the lineup of a certain temporary construction equipment manufacturer, leading handrails 2 with five different sizes of cross members 11, 600 mm, 900 mm, 1200 mm, 1500 mm, and 1800 mm in length, may be targeted, with markers colored differently for each type, and the system may then perform the distinction and picking process.

[0113] Furthermore, the cross members and support members that make up the leading handrail are not limited to cylindrical pipe members, but may be rectangular pipes or other elongated members with an appropriate cross-sectional shape. [Industrial Applicability]

[0114] The present invention can be used to inspect long, composite objects with floating ends, such as leading handrails, which require quick response during inspection work due to the large volume of leased items that are handled. [Explanation of symbols]

[0115] 1...Bulk picking system (main system) 2...Items (preceding handrail) 3...2D camera (imaging device) 4...TOF camera (height detection device) 5...Robot unit 51... Manipulator 52... Gripper (grasping part) 6... Information processing device 7... Loosening equipment (chain conveyor) 8...Detection area 9...Hood 10...Light source 11...Horizontal member 12...Support material 14...Marker 20...Detection unit 24...Trained model

Claims

1. A method for estimating the posture of an article, comprising: a marker for identifying the posture of the article at a gripping point is attached to the article in a predetermined area suitable for a gripping operation; an imaging step of capturing an image of the plurality of bulk-stacked articles so as to include the marker in an imaging range; a marker detection step of inputting the image of the article captured in the imaging step into a trained model and acquiring, as an output result of the trained model, a bounding box that includes and partitions the area where the marker of the article is present; an inclination angle calculation step of calculating an inclination angle of the marker with respect to a predetermined reference line based on the length and width of the bounding box; Including, computer-implemented, Pose estimation method.

2. In the tilt angle calculation step, a virtual marker setting step of setting a virtual marker having the same size as the marker at an internal reference position of the bounding box; a virtual marker rotation step of rotating the virtual marker by the tilt angle around its central coordinate as an axis; an image feature amount calculation step of calculating an image feature amount in a region where the virtual marker exists after rotation; Further comprising: The pose estimation method of claim 1 .

3. The image feature amount is calculated including the degree of variation thereof, a candidate angle calculation step of calculating a plurality of patterns of tilt angles as candidate angles, the tilt angles being calculated according to the aspect ratio of the bounding box; an image feature amount comparison step of performing the virtual marker rotation step and the image feature amount calculation step for each candidate angle and comparing the degree of variation of each image feature amount at each candidate angle; a tilt angle determination step of uniquely determining the candidate angle corresponding to a case where the degree of variation is relatively small based on the result of the comparison as the tilt angle; Further comprising: The pose estimation method according to claim 2.

4. The marker is attached to a center of gravity area of ​​the article. The pose estimation method of claim 1 .

5. The marker is attached around the entire periphery of the center of gravity area. The pose estimation method according to claim 4.

6. the article includes a tubular portion; The marker is attached to the cylindrical portion so that the marker has a rectangular shape in a plan view. The pose estimation method according to claim 5.

7. The inclination angle is an angle formed between the base line of the bounding box and a side of the marker in the longitudinal direction, the base line being the predetermined reference line. The pose estimation method according to claim 6.

8. The marker has visual characteristics including at least any one of a predetermined color, a predetermined pattern, or a predetermined texture for each type of the article. The pose estimation method of claim 1 .

9. The trained model is The visual features of the markers have been learned in association with type information of the articles, In the marker detection step, the type information is output by the trained model. The pose estimation method according to claim 8.

10. A method for causing a computer to execute the method according to any one of claims 1 to 9, Pose estimation program.

11. an information processing device configured to be able to execute the program according to claim 10; Imaging equipment and a robot unit communicably connected to the imaging device and the information processing device; Equipped with The robot unit comprises: A manipulator that is an articulated robot; a gripping unit that is an end effector located at the end of the manipulator; Including, The gripping portion is controlled to have a posture suitable for contact action based on the inclination angle. Bulk picking system.

Images