Information processing method, information processing device, control method, robot system, article manufacturing method, program, and recording medium

Abstract

To suppress a robot from holding a linear member getting entangled with the other linear member.SOLUTION: A robot system comprises: a robot; a photographing part that can photograph a plurality of linear members; a processing part that can execute processing of obtaining a first image having the plurality of linear members projected thereon, from the photographing part, obtaining a second image by applying image processing to the first image, on the basis of a predetermined model, and identifying a superior linear member out of at least two linear members crossing each other, of the plurality of linear members, on the basis of the second image; and a control part that controls the robot so that the robot holds the superior linear member.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to robot technology and image processing technology.

Background Art

[0002] In a production line, operations such as taking out or assembling linear members such as harnesses are difficult processes, and conventionally, such operations have often been performed manually. Therefore, automation of these operations using mechanical devices using cameras and recognition algorithms has been explored.

[0003] As a system for automating such operations, Patent Document 1 discloses measuring the position and orientation of the tip of a linear member based on an image captured by a camera, and using the measurement result to cause a robot to grip the tip of the linear member.

[0004] Furthermore, Patent Document 2 discloses that when a plurality of linear members are mixed and overlapping, considering the shape of the robot's hand, selecting the linear member to be gripped by the hand so that the part and the hand do not interfere and the hand does not grip two linear members.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, when a plurality of linear members cross each other, it may be difficult to determine the crossed state in an image obtained by imaging these plurality of linear members, and the linear member may be held by the robot in a state of being entangled with another linear member.

[0007] The present invention aims to prevent linear members from being held by a robot while entangled with other linear members. [Means for solving the problem]

[0008] The robot system of the present invention is characterized by comprising: a robot; an imaging unit capable of imaging a plurality of linear members; a processing unit capable of acquiring a first image in which the plurality of linear members are captured from the imaging unit, performing image processing on the first image based on a predetermined model to acquire a second image, and performing a process to identify a higher linear member from among the plurality of linear members that intersect each other, based on the second image; and a control unit that controls the robot to hold the higher linear member.

[0009] The robot control method of the present invention is characterized by: applying image processing to a first image in which a plurality of linear members are captured based on a predetermined model to obtain a second image; identifying a higher linear member from among the plurality of linear members that intersect each other, at least two of which are identified, based on the second image; and controlling the robot to hold the higher linear member.

[0010] The present invention is characterized by comprising a processing unit capable of performing image processing on a first image in which a plurality of linear members are captured, based on a predetermined model, to obtain a second image, and then using the second image to identify the uppermost linear member from among the plurality of linear members, of which at least two linear members intersect each other.

[0011] The present invention is characterized by performing image processing on a first image in which a plurality of linear members are captured, based on a predetermined model, to obtain a second image, and then, based on the second image, identifying the uppermost linear member from among the plurality of linear members, at least two linear members that intersect each other. [Effects of the Invention]

[0012] According to the present invention, it is possible to suppress the state in which the linear member is held by the robot in a state of being entangled with other linear members.

Brief Description of the Drawings

[0013] [Figure 1] It is an explanatory diagram showing a schematic configuration of a robot system according to an embodiment. [Figure 2] It is an explanatory diagram of an image processing apparatus according to an embodiment. [Figure 3] It is a block diagram of a computer system in a robot system according to an embodiment. [Figure 4] It is an explanatory diagram of an assembly operation according to an embodiment. [Figure 5] It is an explanatory diagram of an assembly operation of a comparative example. [Figure 6] (a) is a flowchart of a pre - preparation stage process according to an embodiment. (b) is a flowchart of an assembly operation stage process according to an embodiment. [Figure 7] It is an explanatory diagram of a pre - preparation stage according to an embodiment. [Figure 8] (a) is an explanatory diagram showing an example of an input image according to an embodiment. (b) is an explanatory diagram showing an example of an output image according to an embodiment. (c) is an explanatory diagram showing an image of a reference example. [Figure 9] It is an explanatory diagram of an assembly operation stage according to an embodiment. [Figure 10] (a) to (c) are explanatory diagrams of a method for extracting intersection candidates according to an embodiment. [Figure 11] (a) and (b) are explanatory diagrams of a method for extracting intersection candidates according to an embodiment. [Figure 12] (a) and (b) are explanatory diagrams of an intersection determination process according to an embodiment. [Figure 13] (a) to (d) are explanatory diagrams of an example of three - dimensional measurement according to an embodiment. [Figure 14] (a) to (f) are explanatory diagrams of an example of a control method according to an embodiment.

Modes for Carrying Out the Invention

[0014] Hereinafter, embodiments for implementing the present invention will be described in detail with reference to the drawings. FIG. 1 is an explanatory diagram showing a schematic configuration of a robot system 10 according to an embodiment. The robot system 10 includes a robot 100, an image processing device 200, a robot controller 300 which is an example of a control device, and an imaging device 400 which is an example of an imaging unit. The robot 100 is an industrial robot, which is arranged on a production line and used for manufacturing an assembled product W20 which is an example of an article. The assembled product W20 is, for example, a final product or an intermediate product in the equipment itself or parts of the equipment. Examples of the equipment include electronic equipment, electrical equipment, optical equipment, etc.

[0015] The robot 100 is a manipulator. The robot 100 is fixed to a pedestal (not shown). An intermediate product W10 is arranged around the robot 100. The intermediate product W10 has two linear members W1 and W2 which are an example of a plurality of linear members, and a main body W0 to which the linear members W1 and W2 are connected. The linear member W1 is an example of a first linear member, and the linear member W2 is an example of a second linear member. Each of the linear members W1 and W2 is a flexible linear member, and examples thereof include electric wires such as signal lines and power lines, optical fibers, cables, wire harnesses, etc. The tip of each of the linear members W1 and W2 is a free end, and the base end of each of the linear members W1 and W2 is a fixed end fixed to the main body W0. Therefore, the robot 100 can determine the tip of each of the linear members W1 and W2 to an arbitrary position and posture by holding and operating a predetermined portion of each of the linear members W1 and W2.

[0016] The tip of the linear member W1 is constituted by, for example, a connector A1, and the tip of the linear member W2 is constituted by, for example, a connector A2. The main body W0 has a connector A3 which is an assembly target to which the connector A1 is assembled, and a connector A4 which is an assembly target to which the connector A2 is assembled. By assembling the connector A1 to the connector A3 and the connector A2 to the connector A4 by the robot 100, the assembled product W20 is manufactured.

[0017] The explanation will use the example of two linear members W1 and W2, but it is not limited to this; there may be three or more linear members. Also, while it will be explained that each linear member W1 and W2 has connectors A1 and A2, they do not have to have connectors. Furthermore, the color, material, and wire diameter of the multiple linear members may be the same or different.

[0018] The robot 100 and the robot controller 300 are connected by wiring to enable communication. The robot controller 300 and the image processing device 200 are connected by wiring to enable communication. The imaging device 400 and the image processing device 200 are connected by wired or wireless means to enable communication.

[0019] The robot 100 includes a robot arm 101 and a robot hand 102, which is an example of an end effector, i.e., an example of a holding mechanism. The robot arm 101 is, for example, a vertically articulated robot arm. The robot hand 102 is supported by the robot arm 101. The robot hand 102 is attached to a predetermined part of the robot arm 101, for example, the tip of the robot arm 101. The robot hand 102 is configured to hold each linear member W1, W2. Although the case where the holding mechanism is the robot hand 102 is described, it is not limited to this, and for example, the holding mechanism may be a suction mechanism that can hold each linear member W1, W2 by suction. In this embodiment, the robot hand 102 is configured to grip each linear member W1, W2.

[0020] The imaging device 400 is a stereo camera that includes a camera 401, which is an example of a first imaging unit, and a camera 402, which is an example of a second imaging unit. Each camera 401 and 402 is a digital camera that captures an image of a subject and outputs the captured image generated by the capture to the image processing device 200. Here, the image is digital image data in this embodiment.

[0021] The imaging device 400 is fixed to a frame (not shown). Each camera 401, 402 is positioned to image an area containing multiple linear members W1, W2. In other words, the intermediate product W10 is transported by a transport mechanism (not shown) to a predetermined position around the robot 100 such that the multiple linear members W1, W2 are included in the field of view of the imaging device 400. Thus, the imaging device 400 is capable of imaging the multiple linear members W1, W2.

[0022] Each camera 401 and 402 captures images of multiple linear members W1 and W2, which are the subjects, thereby generating two-dimensional images I11 and I12 as examples of the first image. Each camera 401 and 402 may be a color camera that generates a color image, or a monochrome camera that generates a monochrome image. Furthermore, although the description describes the case where the imaging device 400 includes two cameras, it may also include one camera or three or more cameras.

[0023] The image processing device 200 sends imaging commands to each camera 401 and 402 to cause each camera 401 and 402 to perform imaging. The image processing device 200 acquires each image I11 and I12 generated by each camera 401 and 402 and performs image processing on each acquired image I11 and I12. By performing image processing on each image I11 and I12, the image processing device 200 calculates the position and orientation of predetermined parts of each linear member W1 and W2 in the coordinate system of the robot 100. Based on the position and orientation of predetermined parts of each linear member W1 and W2 calculated by the image processing device 200, the robot controller 300 causes the robot 100 to selectively hold one of the predetermined parts of the linear members W1 and W2. The robot controller 300 controls the operation of the robot 100 so that the connector of the linear member held by the robot 100 is assembled to the corresponding connector. Multiple linear members W1 and W2 are sequentially assembled onto the main body W0 to manufacture the assembled product W20.

[0024] In this embodiment, the image processing device 200 is configured as a computer. Figure 2 is an explanatory diagram of the image processing device 200 according to this embodiment. The image processing device 200 includes a main unit 201, a display 202 which is an example of a display device connected to the main unit 201, and a keyboard 203 and a mouse 204 which are examples of input devices connected to the main unit 201.

[0025] In this embodiment, the robot controller 300 shown in Figure 1 is configured as a computer. The robot controller 300 is configured to control the movement of the robot 100, that is, the posture of the robot 100.

[0026] Figure 3 is a block diagram of the computer system in the robot system 10 according to the embodiment. The main body 201 of the image processing device 200 includes a CPU (Central Processing Unit) 251, which is an example of a processor. The CPU 251 is an example of a processing unit. The main body 201 also includes a ROM (Read Only Memory) 252, a RAM (Random Access Memory) 253, and an HDD (Hard Disk Drive) 254 as storage units. The main body 201 also includes a recording disk drive 255 and an input / output interface 256. The CPU 251, ROM 252, RAM 253, HDD 254, recording disk drive 255, and interface 256 are connected to each other by a bus so that they can communicate with one another.

[0027] ROM252 stores the basic programs related to the operation of the computer. RAM253 is a memory device that temporarily stores various data, such as the results of calculations performed by CPU251. HDD254 records the results of calculations performed by CPU251 and various data acquired from external sources, as well as program 261, which causes CPU251 to perform various processes described later. Program 261 is application software that enables CPU251 to perform various processes described later. Therefore, by executing program 261 recorded in HDD254, CPU251 can perform image processing described later. Recording disk drive 255 can read various data and programs recorded on recording disk 262.

[0028] In this embodiment, the non-temporary recording medium readable by the computer is the HDD254, and the program 261 is recorded on the HDD254, but this is not the only possible representation. The program 261 may be recorded on any non-temporary recording medium readable by the computer. Examples of recording media that can be used to supply the program 261 to the computer include flexible disks, hard disks, optical disks, magneto-optical disks, magnetic tapes, non-volatile memory, and the like.

[0029] The robot controller 300 includes a CPU 351, which is an example of a processor. The CPU 351 is an example of a control unit. The robot controller 300 also includes a ROM 352, RAM 353, and HDD 354 as storage units. Furthermore, the robot controller 300 includes a recording disk drive 355 and an input / output interface 356. The CPU 351, ROM 352, RAM 353, HDD 354, recording disk drive 355, and interface 356 are connected to each other by a bus so that they can communicate with one another.

[0030] ROM352 stores the basic programs related to the operation of the computer. RAM353 is a memory device that temporarily stores various data, such as the results of calculations performed by CPU351. HDD354 records the results of calculations performed by CPU351 and various data acquired from external sources, as well as program 361, which causes CPU351 to perform various processes. Program 361 is application software that enables CPU351 to perform various processes described later. Therefore, CPU351 can perform control processing by executing program 361 recorded in HDD354, thereby controlling the operation of robot 100 in Figure 1. Recording disk drive 355 can read various data and programs recorded on recording disk 362.

[0031] In this embodiment, the non-temporary recording medium readable by the computer is the HDD 354, and the program 361 is recorded on the HDD 354, but this is not the only possible representation. The program 361 may be recorded on any non-temporary recording medium readable by the computer. Examples of recording media that can be used to supply the program 361 to the computer include flexible disks, hard disks, optical disks, magneto-optical disks, magnetic tapes, non-volatile memory, and the like.

[0032] In this embodiment, image processing and control processing are performed by multiple computers, i.e., multiple CPUs 251, 351, but this is not limited to this configuration. Image processing and control processing may also be performed by a single computer, i.e., a single CPU. In this case, the single CPU can be configured to function as both the processing unit and the control unit.

[0033] In such a robot system 10, the case in which linear members W1 and W2 are assembled to the main body W0 in this order will be explained with reference to Figure 4. The robot controller 300 controls the robot arm 101 so that the robot hand 102 moves above the holding position based on the three-dimensional coordinate information indicating the holding position of the linear member W1 calculated by the image processing device 200. The robot controller 300 then causes the robot hand 102 to hold the linear member W1. Next, the robot controller 300 controls the robot arm 101 to assemble the connector A1 of the linear member W1 to the connector A3 of the main body W0. After the assembly of connector A1 to connector A3 is completed, the robot controller 300 repeats the same processing operation for the linear member W2. This completes the assembly process and manufactures the assembled product W20.

[0034] Here, flexible linear members are flexible and their orientation can easily change. For this reason, at least two of the multiple linear members, for example, two linear members W1 and W2 as shown in Figure 5, may be arranged around the robot 100 at an intersection. At the point where linear members W1 and W2 intersect, linear member W2 may be positioned higher than linear member W1. Here, the higher linear member is the linear member that is positioned highest among the at least two linear members at the point where the at least two linear members intersect each other. The lower linear member is the linear member that is positioned below the highest linear member among the at least two linear members at the point where the at least two linear members intersect each other. In the example in Figure 5, at the point where linear members W1 and W2 intersect before assembly, linear member W2 is the higher linear member and linear member W1 is the lower linear member.

[0035] In this case, if the robot 100 holds the linear member W1 before the linear member W2 and assembles connector A1 to connector A3, the linear member W2 may become entangled with the linear member W1. In this state, it may be difficult for the robot 100 to then hold the linear member W2, or even if the robot 100 is able to hold the linear member W2, it may be difficult to assemble connector A2 to connector A4. Furthermore, the linear members are flexible and easily change their orientation, and the linear members themselves may be small (i.e., have a thin wire diameter). Therefore, a method for accurately measuring the intersection state of multiple linear members in 3D using a spatial coding method with a projector could be considered, but this measurement method requires time for measurement processing, which reduces the productivity of the assembled product W20.

[0036] In this embodiment, the CPU 251 of the image processing device 200 determines the intersection state of linear members W1 and W2 using at least one of the two-dimensional images I11 and I12 captured by the imaging device 400, in order to avoid situations in which linear members become entangled during assembly work.

[0037] Figures 6(a) and 6(b) are flowcharts of the processes according to the embodiment. The flowcharts include each process of the control method for the robot 100, including the image processing method. The flowchart shown in Figure 6(a) shows the preparation stage, and the flowchart shown in Figure 6(b) shows the assembly stage. Here, the CPU 251 performs the following processes by executing program 261. Also, the CPU 351 performs the following processes by executing program 361.

[0038] First, the preparation stage will be explained using the flowchart in Figure 6(a). Figure 7 is an explanatory diagram of the preparation stage according to the embodiment. In the preparation stage, a trained model 260 is created as an example of a predetermined model.

[0039] The subject prepared in this preliminary stage shall be an intermediate product with the same configuration as the intermediate product W10 that will be used in the assembly stage. That is, in the assembly stage, the assembled product W20 is manufactured from the intermediate product W10, but a different intermediate product from the intermediate product W10 used in the assembly stage may be prepared in the preliminary stage, as long as it has the same configuration as the intermediate product W10 used in the assembly stage. Hereafter, the intermediate product used in the preliminary stage will be described using the same reference numerals as the intermediate product W10 used in the assembly stage, since it has the same configuration as the intermediate product W10 used in the assembly stage.

[0040] Furthermore, it is preferable that the exposure conditions, such as the imaging time and lens aperture of the imaging device 400, during the preparation phase be set to the same settings as during the assembly phase. Also, it is preferable that the position where the imaging device 400 is installed and the operating environment, such as the illumination light shining on the multiple linear members W1 and W2, be set to the same settings as during the assembly phase. While the preparation phase will describe the use of the imaging device 400 used in the assembly phase, it is not limited to this, and a different imaging device may be used. Additionally, since no assembly work is performed during this preparation phase, the robot 100 does not need to be present around the multiple linear members W1 and W2.

[0041] In step S11, the CPU 251 causes the imaging device 400 to image multiple linear members W1 and W2, and acquires the image I1A from the imaging device 400. The image I1A acquired in this step S11 process becomes part of the training data D1 used for learning. Therefore, it is preferable that the number of images I1A acquired in step S11 be as large as possible.

[0042] In step S11, the CPU 251 causes the imaging device 400 to image several patterns of linear members W1 and W2 with different intersection states, thereby acquiring multiple images I1A, and stores these images I1A in a storage unit such as the HDD 254. The orientation of the linear members W1 and W2 may be changed by an operator or by a mechanism such as a robot.

[0043] Note that among the multiple images I1A, there may be images in which the linear members W1 and W2 do not intersect, or images in which either of the linear members W1 or W2 is not visible. Figure 8(a) illustrates an example of one of the multiple images I1A. The image I1A shown in Figure 8(a) is a grayscale image. Image I1A includes images IW1A and IW2A corresponding to the linear members W1 and W2, and image IB1A corresponding to the background.

[0044] Next, in step S12, the CPU 251, under the operator's instructions, creates training data D1 based on the image I1A obtained in step S11. In this embodiment, a machine learning algorithm is used in the learning process, and in particular, a "supervised learning" algorithm is used. Therefore, in step S12, training data D1 to be used for "supervised learning" is created. Training data D1 includes multiple images I1A and multiple images I2A that have a one-to-one correspondence with the multiple images I1A. A trained model 260 is created by machine learning so that each image I2A is output from each image I1A.

[0045] Each image I1A is an example of an input image. Each image I2A is an example of an output image. Each image I2A is created by an operator operating the image processing device 200 according to the following steps 1 to 3, based on the corresponding image I1A.

[0046] Step 1: Create a background image by filling the image corresponding to the background of elements other than the linear elements W1 and W2 (elements other than the linear elements) with a single color. Step 2: Create linear images by filling the images corresponding to linear members W1 and W2 with a single color. However, the color should be different from the background image. Step 3: At the intersection of the linear image corresponding to linear member W1 and the linear image corresponding to linear member W2, the linear image corresponding to the lower linear member shall be discontinuous. The linear image corresponding to the upper linear member shall be continuous. The colors of the linear images and the linear images shall be different from the background image color, and they may be the same color or different colors.

[0047] Figure 8(b) shows an example of image I2A. Image I2A shown in Figure 8(b) was created based on image I1A shown in Figure 8(a). In image I2A, the linear image L1A corresponding to the upper linear member and the linear image L2A corresponding to the lower linear member are distinguished and represented depending on whether they are continuous or interrupted at the intersection C1A. Here, when the linear image L2A becomes discontinuous, or interrupted, at the intersection C1A, it means that the linear image L2A is divided into two parts with the linear image L1A in between, so that the linear image L2A does not come into contact with the linear image L1A. In this way, at the intersection C1A, the linear image L2A corresponding to the lower linear member becomes intermittent. In image I2A shown in Figure 8(b), the linear images L1A and L2A are a first color, for example, white, and the background image B1A is a second color different from the first color, for example, black.

[0048] Furthermore, if a computer performs image processing on image I1A, such as binarization, the resulting processed image may not necessarily be image I2A as shown in Figure 8(b). For example, in the processed image, there may be no boundary between two intersecting linear images, and the two linear images may appear as a single continuous line. In this embodiment, in step 3, the operator may create image I2A directly from image I1A. Alternatively, in step 3, the computer may first perform image processing on image I1A, such as binarization, and then the operator may create image I2A by modifying the processed image.

[0049] By following steps 1-3 above, a pair of one image I1A and one image I2A is created. By performing steps 1-3 above for all images I1A, multiple pairs of images I1A and I2A are created. These multiple pairs constitute the training data D1.

[0050] Next, in step S13, the CPU 251 performs machine learning to associate image I1A with image I2A. The machine learning algorithm used in step S13 is, for example, Pix2Pix (Pixel to Pixel). Pix2Pix is ​​a type of "supervised learning" algorithm that infers an output value for each pixel of the input image by performing machine learning based on training data D1. If the learning progresses well, linear images corresponding to higher linear members at intersections will be filled with a continuous solid color, while linear images corresponding to lower linear members at intersections will be filled in a broken state at the intersection.

[0051] Pix2Pix is ​​used in this type of learning because it can automatically extract features from the grayscale information on the image of linear members in an intersecting state to determine whether the linear members are continuous or discontinuous. These features include, for example, the shading formed on the linear image corresponding to the lower linear member at the intersection, and the continuity of color in the linear image corresponding to the upper linear member at the intersection. Therefore, in this embodiment, if the vertical relationship between two intersecting linear members does not appear as a grayscale feature in the captured image, as shown in Figure 8(c), then that captured image is excluded from the training data D1. For example, if the linear members are illuminated uniformly from all directions by dome lighting, and imaging is performed with equal light intensity, no shadow may be formed near the intersection, and a state like that shown in Figure 8(c) may occur. Therefore, images obtained by such imaging are excluded.

[0052] Furthermore, the algorithms used for machine learning are not limited to Pix2Pix; any algorithm other than Pix2Pix may be used as long as it has the function of extracting the aforementioned features.

[0053] The trained model 260 obtained through the learning process in step S13 is stored in the memory unit of the image processing device 200, for example, the HDD 254.

[0054] Next, the assembly work stage will be explained according to the flowchart in Figure 6(b). Figure 9 is an explanatory diagram of the assembly work stage according to the embodiment. Intermediate products W10 are sequentially transported to the vicinity of the robot 100. In step S21, the CPU 251 causes the imaging device 400 to image the linear members W1 and W2 of the intermediate product W10 that have been transported to the vicinity of the robot 100. As described above, it is preferable that the imaging conditions such as exposure time and lens aperture, as well as the operating environment, are the same as in step S11.

[0055] In step S21, the imaging operation causes the CPU 251 to acquire an image I1 from the imaging device 400 as shown in Figure 9. Image I1 is an example of a first image. Image I1 is a grayscale image. In this embodiment, since the imaging device 400 is a stereo camera, two images I11 and I12 are obtained by the imaging operation, but the image I1 shown in Figure 9 is one of the two images I11 and I12. As shown in Figure 9, two linear members W1 and W2 are captured in image I1. That is, image I1 includes images IW1 and IW2 corresponding to the linear members W1 and W2, and image IB1 corresponding to the background. In this embodiment, each linear member W1 and W2 is imaged so that the tip, which is an example of a predetermined part to be held by the robot 100, is captured. In image I1, the endpoint IE1 of image IW1 corresponds to the tip of linear member W1, and the endpoint IE2 of image IW2 corresponds to the tip of linear member W2.

[0056] Next, in step S22, the CPU 251 infers the intersection state of linear members W1 and W2 from the image I1 captured in step S21. The pre-trained model 260 stored in the HDD 254 is used for inference. That is, in step S22, the CPU 251 generates image I2 by performing image processing on image I1 based on the pre-trained model 260. In other words, the CPU 251 takes image I1 as the input image and processes image I1 based on the pre-trained model 260 to generate image I2 as the output image. Image I2 is an example of a second image.

[0057] Image I2 contains linear images L1 and L2 corresponding to linear members W1 and W2. In Figure 9, linear images L1 and L2 are a first color, for example, white, and background image B1 is a second color different from the first color, for example, black. At the intersection of the two linear images L1 and L2, the linear image corresponding to the upper linear member becomes a continuous monochrome image, while the linear image corresponding to the lower linear member becomes a broken monochrome image. That is, at the intersection, the linear image corresponding to the lower linear member is discontinuous, sandwiched between the linear image corresponding to the upper linear member.

[0058] In the example shown in Figure 9, image I2 contains multiple intersections C1 and C2. At intersection C1, the first linear image corresponding to the upper linear member is linear image L1, and the second linear image corresponding to the lower linear member is linear image L2. Similarly, at intersection C2, the first linear image corresponding to the upper linear member is linear image L2, and the second linear image corresponding to the lower linear member is linear image L1.

[0059] The following describes a method for detecting intersections C1 and C2 in image I2. In steps S23 to S24, the CPU 251 extracts candidate intersections from image I2, performs a predetermined calculation on the candidate intersection, and performs an intersection determination process to determine whether the candidate intersection is an intersection based on the result of the predetermined calculation.

[0060] The process in step S23 will now be explained. First, the CPU 251 extracts crossover candidates in image I2. The method for extracting crossover candidates will be explained in detail below using Figures 10(a) to 11(b).

[0061] First, as shown in Figure 10(a), the CPU 251 extracts line segment images L11, L12, L21, and L22 from image I2. Next, as shown in Figure 10(b), the CPU 251 extracts the endpoints of the respective regions for each of the line segment images L11, L12, L21, and L22. Various endpoint extraction algorithms are known for extracting endpoints, such as the Harrisno corner detection method, but the method is not particularly limited. The endpoints extracted in this way are shown as endpoints L11-a, L11-b, L12-a, L21-a, L21-b, and L22-a in Figure 10(b). Next, the CPU 251 defines a predetermined region containing each endpoint and sets this as the endpoint region for each endpoint. Figure 10(c) shows the endpoint regions T11-a, T11-b, T12-a, T21-a, T21-b, and T22-a. Each endpoint region is, for example, a rectangular region of a predetermined size, with each endpoint located at the center of each endpoint region. Note that the shape of each endpoint region is not limited to a rectangle, but may be various shapes, such as a circle.

[0062] Then, if at least two endpoint regions exist in image I2, the CPU 251 determines whether the positional relationship of the line segment images contained within the endpoint regions is within a certain threshold for all combinations of endpoint regions. Based on this determination, if it is within the certain threshold, the region containing the endpoint regions of that combination is extracted as an intersection candidate.

[0063] The following will provide a detailed explanation using two line segment images L21 and L22 as examples. Line segment image L21 is an example of a first line segment image, and line segment image L22 is an example of a second line segment image. Also, endpoint L21-b of line segment image L21 is an example of a first endpoint, and endpoint L22-a of line segment image L22 is an example of a second endpoint. Endpoint region T21-b, which includes endpoint L21-b, is an example of a first region, and endpoint region T22-a, which includes endpoint L22-a, is an example of a second region.

[0064] As shown in Figure 11(a), the CPU 251 calculates the regression line LR1 along the line segment image L21 in the endpoint region T21-b, and the regression line LR2 along the line segment image L22 in the endpoint region T22-a. For the calculation of the regression line LR1, the portion of the line segment image L21 included in the endpoint region T21-b is used. For the calculation of the regression line LR2, the portion of the line segment image L22 included in the endpoint region T22-a is used.

[0065] CPU251 calculates the interval Da between the perpendicular Lp1 drawn from regression line LR1 to endpoint L21-b of line segment image L21, and the perpendicular Lp2 drawn from regression line LR2 to endpoint L22-a of line segment image L22. CPU251 also calculates the intersection angle θb between regression line LR1 and regression line LR2.

[0066] In this embodiment, the CPU 251 determines whether the predetermined conditions are met, namely that the interval Da is less than or equal to the threshold THa and the intersection angle θb is less than or equal to the threshold THb. If the predetermined conditions are met between line segment image L21 and line segment image L22, the CPU 251 extracts an area encompassing endpoint region T21-b and endpoint region T22-a as intersection candidate C11. Intersection candidate C11 is, for example, the smallest rectangular area encompassing endpoint region T21-b and endpoint region T22-a. Note that the shape of the area indicating the intersection candidate is not limited to a rectangle, but may be various shapes, such as a circle.

[0067] In Figure 11(b), the region containing the combination of endpoint regions T21-b and T22-a satisfies the predetermined conditions and becomes intersection candidate C11, and the region containing the combination of endpoint regions T11-b and T12-a satisfies the predetermined conditions and becomes intersection candidate C12. Thus, it is possible to infer the broken region between the two line segment images. Note that in Figure 11(c), for example, endpoint regions T11-a and T21-a do not satisfy the predetermined conditions and are not intersection candidates.

[0068] Next, at intersection candidate C11, the CPU 251 interpolates line segment images L21 and L22 using a predetermined interpolation method. Linear interpolation is preferred for this predetermined interpolation. By interpolating line segment images L21 and L22 using linear interpolation, a linear interpolation region R1 shown in Figure 11(a) is obtained. Similarly, an interpolation region is obtained at intersection candidate C12 shown in Figure 11(b).

[0069] Through the above interpolation process, interpolation region R1 for intersection candidate C11 and interpolation region R2 for intersection candidate C12 are obtained, as shown in Figure 12(a). Interpolation region R1 is the pixel region connecting line segment image L21 and line segment image L22. Interpolation region R2 is the pixel region connecting line segment image L11 and line segment image L12. Note that the method is not limited to linear interpolation, as long as it is possible to infer the gap between the two line segment images.

[0070] Next, in step S24, the CPU 251 determines whether the line segment image L11 and the interpolation region R1 overlap at intersection candidate C11. If the line segment image L11 and the interpolation region R1 overlap, the CPU 251 determines intersection candidate C11 as intersection portion C1. Similarly, the CPU 251 determines whether the line segment image L22 and the interpolation region R2 overlap at intersection candidate C12. If the line segment image L22 and the interpolation region R2 overlap, the CPU 251 determines intersection candidate C12 as intersection portion C2. Depending on the state of the linear members W1 and W2, there may be one intersection portion, or there may be multiple intersection portions C1 and C2 as shown in Figure 9. Also, if a line segment image and the interpolation region do not overlap at an intersection candidate, that intersection candidate is determined not to be an intersection portion.

[0071] As shown in Figure 12(b), in the intersection C1, the continuous line segment image L11 is the linear image L1 corresponding to the linear member W1, and the line segment images L21 and L22 connected in the interpolation region R1 are the linear image L2 corresponding to the linear member W2. In the intersection C1, the CPU 251 identifies the linear member W1 corresponding to the linear image L1 as the upper linear member WU, and the linear member W2 corresponding to the linear image L2 as the lower linear member WL.

[0072] Next, in step S25, the CPU 251 determines the position of a predetermined part of the upper linear member WU to be held by the robot 100, i.e., the holding position. In this embodiment, the CPU 251 calculates the holding position by 3D measurement using the image captured by the imaging device 400. As shown in Figure 1, the imaging device 400 has a plurality of cameras 401 and 402 arranged at intervals from each other. The parallax between camera 401 and camera 402 enables 3D measurement of the linear members W1 and W2, which are the subjects.

[0073] Figures 13(a) to 13(d) are explanatory diagrams of an example of three-dimensional measurement according to the embodiment. In step S21 described above, the CPU 251 acquires image I11 shown in Figure 13(a) generated by the imaging operation of camera 401 and image I12 shown in Figure 13(a) generated by the imaging operation of camera 402. Each image I11 and I12 is a grayscale image. Image I11 includes image IW11 corresponding to linear member W1 and image IW21 corresponding to linear member W2. The endpoint IE11 of image IW11 corresponds to the tip of linear member W1, and the endpoint IE21 of image IW21 corresponds to the tip of linear member W2. Image I12 includes image IW12 corresponding to linear member W1 and image IW22 corresponding to linear member W2. The endpoint IE12 in image IW12 corresponds to the tip of the linear member W1, and the endpoint IE22 in image IW22 corresponds to the tip of the linear member W2.

[0074] Then, in step S22 described above, the CPU 251 generates images I21 and I22, shown in Figure 13(b), from images I11 and I12 by inference based on the trained model 260. Images I21 and I22 are binarized images. Image I21 includes linear member W1, i.e., linear image L11 corresponding to image IW11, and linear member W2, i.e., linear image L21 corresponding to image IW21. Image I22 includes linear member W1, i.e., linear image L12 corresponding to image IW12, and linear member W2, i.e., linear image L22 corresponding to image IW22. Then, in steps S23 and S24 described above, the CPU 251 performs cross-rejection processing. As a result, in image I21, the intersection C11 between linear image L11 and linear image L21 is determined, and in image I22, the intersection C12 between linear image L12 and linear image L22 is determined. At intersection C11, it is determined that linear image L11 corresponds to the upper linear member WU, and linear image L21 corresponds to the lower linear member WL. Furthermore, at intersection C12, it is determined that linear image L12 corresponds to the upper linear member WU, and linear image L22 corresponds to the lower linear member WL.

[0075] Next, in each image I21 and I22, the endpoints E11 and E12 of each linear image L11 and L12 corresponding to the upper linear member WU are determined. Here, it is assumed that the position where the main body W0 is installed around the robot 100 and the direction in which the linear members W1 and W2 extend relative to the main body W0 are predetermined. This determines the approximate region in each image I21 and I22 where each endpoint E11 and E12 corresponding to each connector A1 and A2, which are the tips of each linear member W1 and W2, are located. As shown in Figure 13(c), the CPU 251 scans each image I21 and I22 in an interlaced manner, starting from a position opposite to where the main body W0 is located, to search for each linear image L11 and L12 corresponding to the upper linear member WU. Then, in each image I21 and I22, the CPU 251 determines the endpoints E11 and E12 as the first points encountered.

[0076] Even if the linear member W1 is a linear member without a connector, the same method can be used to search for each endpoint E11, E12. Furthermore, the holding position of the part of the linear member to be held by the robot 100 may be a position offset from the tip by a predetermined length from the tip to the base end of the linear member. In this case, as shown in Figure 13(d), after detecting each endpoint E11, E12, the CPU 251 determines points F11, F12 that are offset by a predetermined length from each endpoint E11, E12 along each linear image L11, L12.

[0077] Based on the positions of each endpoint E11, E12 or each point F11, F12 on the respective images I21, I22 obtained through the above processing, the three-dimensional position of the holding position can be measured by the principle of triangulation. The CPU 251 transmits the holding position information to the CPU 351 of the robot controller 300. In step S26, the CPU 351 of the robot controller 300 controls the robot 100 to hold a predetermined part of the linear member W1, which is the upper linear member WU, based on the received holding position information. The CPU 351 then controls the robot 100 to assemble the connector A1 of the linear member W1 to the connector A3. This prevents the linear members W1 and W2 from being held by the robot 100 in an entangled state, and allows the assembly work to be performed smoothly.

[0078] Although the description has focused on the case where the imaging device 400 is a stereo camera, the explanation is not limited to this. For example, the imaging device 400 may be a monocular camera. In this case, the holding position may be measured using a separate device.

[0079] The above example described the case where there is one intersection point between the linear members W1 and W2, but there may also be cases where there are multiple intersection points between the linear members W1 and W2. In such cases, the robot 100 may be operated to eliminate the intersection of the linear members W1 and W2. The processing in such cases will be explained with reference to Figures 14(a) to 14(f). Note that Figures 14(a) to 14(f) only show and explain images I1 obtained from one of the multiple cameras 401 and 402 shown in Figure 1, or images I2 obtained by processing image I1. Images obtained from the other camera are processed in the same way as images I1 obtained from the other camera, so the explanation is omitted.

[0080] First, the CPU 251 acquires image I1 from the imaging device 400, as shown in Figure 14(a). In image I1, the two images IW1 and IW2, corresponding to the two linear members W1 and W2, are twisted so as to intersect twice. Also, in image I1, the endpoint IE1 of image IW1 corresponds to the tip of linear member W1, and the endpoint IE2 of image IW2 corresponds to the tip of linear member W2.

[0081] The CPU 251 generates image I2 shown in Figure 14(b) from image I1 based on the trained model 260 and performs intersection detection processing. As a result, multiple intersection points, such as two C1 and C2, are detected between linear image L1 corresponding to linear member W1 and linear image L2 corresponding to linear member W2.

[0082] Next, as shown in Figure 14(c), the CPU 251 selects the single intersection CC to focus on from the intersections C1 and C2. The twisted linear members W1 and W2 need to be untwisted from the tip. Therefore, the CPU 251 searches for intersections by scanning in an interlaced manner in image I2 from a position opposite to the position where the main body W0 is located, and selects the first intersection C1 encountered as the intersection CC to focus on. Alternatively, the intersection CC to focus on could be the first intersection C1 encountered by tracing the linear image L1 from its endpoint E1, which is the first intersection encountered by scanning in an interlaced manner.

[0083] In this way, once the intersection CC of interest is uniquely determined, the process described above defines the upper linear member WU and the lower linear member WL at that intersection CC, as shown in Figure 14(d). Thus, the CPU 251 identifies the upper linear member WU at the first intersection point reached by tracing from the respective ends of the linear members W1 and W2. At the intersection C1, the linear image L1 corresponds to the upper linear member WU, and the linear image L2 corresponds to the lower linear member WL. Then, by scanning in an interlaced manner, the endpoint E1 is searched for and the gripping position is determined.

[0084] Since there are two intersection points C1 and C2, the CPU 351 controls the robot 100 to hold the upper linear member WU, which is the linear member W1, at intersection point C1. Subsequently, the CPU 351 controls the robot 100 so that the upper linear member WU no longer intersects with the lower linear member WL. The position and location where the upper linear member WU is retracted is determined based on the positional relationship between the linear members W1 and W2 on the 2D image related to intersection point C1. By repeating the above operations, the number of intersection points is reduced one by one.

[0085] The CPU 251 again causes the imaging device 400 to image the linear members W1 and W2, and processes the captured images based on the trained model 260 to generate a new image I2 as shown in Figure 14(e). Then, the CPU 251 performs the intersection determination process again. In the example shown in Figure 14(e), only one intersection C2 is detected. At intersection C2, the linear image L2 corresponds to the upper linear member WU, and the linear image L1 corresponds to the lower linear member WL. Then, as shown in Figure 14(f), the endpoint E2 of the linear image L2 is searched by scanning in an interlaced manner, and the gripping position is determined. After that, the linear members W2 and W1 are assembled to the main body W0 in the order of W2 and W1. Through the above operations, the entanglement of the linear members W1 and W2 is resolved, and the assembly of each linear member W1 and W2 can be performed smoothly.

[0086] As described above, according to this embodiment, the overlapping state of linear members W1 and W2 can be determined with high accuracy by the image I1 captured of the linear members W1 and W2, so that the robot 100 can hold the upper of the two linear members W1 and W2. In other words, it is prevented from the robot 100 holding a linear member in an entangled state with other linear members. For example, when assembling connector A2 to connector A4 and connector A1 to connector A3, the entangled situation of linear members W1 and W2 as shown in Figure 5 can be avoided. Therefore, the productivity of the assembled product W20 is improved.

[0087] Furthermore, since the image I2 is generated using the trained model 260 to determine the intersection state of the linear members W1 and W2, there is no need to apply a high-resolution camera to the imaging device 400. Also, there is no need to prepare a separate projector to determine the holding position of the linear members, nor is there a need to prepare a high-precision 3D measurement system for measuring the thickness of thin wires. In this embodiment, since the intersection state of multiple linear members W1 and W2 can be determined using the image I2 generated based on the trained model 260, it is possible to hold the linear members in the robot 100 without entanglement using a simple and lightweight configuration. Thus, the productivity of the assembled product W20 is improved.

[0088] It should be noted that the present invention is not limited to the embodiments described above, and many modifications are possible within the technical concept of the present invention. Furthermore, the effects described in the embodiments are merely a list of the most preferred effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0089] In the above embodiment, the case in which the image I2A used for training data D1 is a binarized image was described, but the invention is not limited to this. In image I2A, each of the multiple linear images corresponding to the multiple linear members may be colored differently from each other. In this case, in image I2 generated based on the trained model 260, the multiple linear images will be colored differently from each other.

[0090] Furthermore, although the above-described embodiment described the case where the robot arm 101 is a vertically articulated robot arm, it is not limited to this. The robot arm 101 may be various types of robot arms, such as a horizontally articulated robot arm, a parallel link robot arm, or a Cartesian robot. Alternatively, instead of a robot arm, it may be applied to a machine that can automatically perform movements such as extension and retraction, bending and straightening, vertical movement, horizontal movement, or rotation, or combinations thereof, based on information stored in a memory device provided in the control device.

[0091] Furthermore, although the above embodiment described a case in which the imaging device 400 is fixed to a frame other than the robot 100, the invention is not limited to this, and the imaging device 400 may be fixed to the robot 100, for example.

[0092] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]

[0093] 10…Robot system, 100…Robot, 200…Image processing device, 251…CPU (processing unit), 351…CPU (control unit), 400…Imaging device (imaging unit)

Claims

1. A processing unit comprising: Acquire an image in which a plurality of linear members can be detected; identifying a top linear member from among at least two mutually intersecting linear members based on a portion in the image where the linear members are discontinuous; 23. An information processing method comprising:

2. The portion where at least two linear members intersect in the image is discontinuous.

2. The information processing method according to claim 1,

3. The image includes a first linear image corresponding to an upper linear member and a second linear image corresponding to a lower linear member intersecting the upper linear member, The processing unit, When the second linear image is divided into two parts with the first linear image sandwiched therebetween so as not to come into contact with the first linear image, it is determined that the linear member corresponding to the second linear image is discontinuous; identifying a linear member corresponding to the second linear image that is discontinuous as a lower linear member, and identifying a linear member corresponding to the first linear image that is not discontinuous as a higher linear member; 3. The information processing method according to claim 1, wherein the first and second inputs are input to the information processing apparatus.

4. The processing unit, extracting an area including a first area in which a first end point of a first line segment image exists and a second area in which a second end point of a second line segment image exists as a crossing candidate if a predetermined condition is satisfied between the first area and the second area in the image; if an area obtained by interpolating the first line segment image and the second line segment image by a predetermined interpolation overlaps with another line segment image, the intersection candidate is determined to be an intersection portion.

4. The information processing method according to claim 1, wherein the first and second inputs are input to the first and second inputs.

5. The specified condition is a condition in which the distance between a perpendicular line drawn from a first regression line along the first line segment image to the first endpoint and a perpendicular line drawn from a second regression line along the second line segment image to the second endpoint is within a threshold range, and an intersection angle between the first regression line and the second regression line is within a threshold range.

5. The information processing method according to claim 4.

6. The predetermined interpolation is a linear interpolation.

6. The information processing method according to claim 4, wherein:

7. Each tip of the linear member is a free end, taking an image of the linear members so that the tips of the linear members can be detected from the image; 7. The information processing method according to claim 1,

8. The at least two linear members include a first linear member and a second linear member that intersect with each other, When there are a plurality of points where the first linear member and the second linear member intersect, the processing unit identifies a linear member that is higher than the first intersecting point by tracing from the tip of each of the first linear member and the second linear member.

8. The information processing method according to claim 1,

9. The processing unit: having the robot retain the first detected end point of the linear member identified above the first intersection point; 9. The information processing method according to claim 8,

10. The image is binarized by making the color of the linear members different from the color of the background of the linear members.

10. The information processing method according to claim 1,

11. A first image in which a linear member is captured and a second image in which the shading of the first image is binarized are obtained, and a trained model is generated by performing machine learning using the first image as an input image and the second image as an output image as training data; The processing unit, converting the first image into the image for detecting linear members using the trained model; 11. The information processing method according to claim 1,

12. The second image used as the training data can be modified by a user.

12. The information processing method according to claim 11.

13. The processing unit, controlling the robot to hold the upper linear member, and then controlling the robot to assemble a tip end of the upper linear member to an assembly target; 13. The information processing method according to claim 1,

14. The processing unit, controlling the robot to hold the upper linear member, and then controlling the robot to prevent the upper linear member from crossing the lower linear member; 13. The information processing method according to claim 1,

15. The images are acquired by a stereo camera.

12. The information processing method according to claim 1,

16. The processing unit, Acquire an image in which a plurality of linear members can be detected; identifying a top linear member from among at least two mutually intersecting linear members based on a portion in the image where the linear members are discontinuous; 23. An information processing apparatus comprising:

17. A method for controlling a robot, comprising: Acquire an image in which a plurality of linear members can be detected; Identifying a top linear member from among at least two mutually intersecting linear members based on a portion where the linear members are discontinuous in the image; Controlling the robot so as to cause the robot to hold an upper linear member; A control method comprising:

18. A robot system comprising a robot and a processing unit, The processing unit, Acquire an image in which a plurality of linear members can be detected; Identifying a top linear member from among at least two mutually intersecting linear members based on a portion where the linear members are discontinuous in the image; Controlling the robot so as to cause the robot to hold an upper linear member; A robot system comprising:

19. A method for manufacturing an article by manipulating a linear member using the information processing method according to any one of claims 1 to 15.

20. A program for causing a computer to execute the information processing method according to any one of claims 1 to 15 or the control method according to claim 17.

21. A recording medium readable by a computer having the program described in claim 20 recorded thereon.

Images