Control method, program, and component supply device for a component supply device.

Abstract

To provide a method for controlling a component supply device that can enhance the efficiency of pickup operation for components, a program, and the component supply device.SOLUTION: A method for controlling a component supply device includes processing (1) to (3). The processing (1) includes photographing a component mounted on a pick table from a plurality of different view points. The processing (2) includes extracting feature points of the component based on image information obtained through the photography from the plurality of different view points. The processing (3) includes comparing the extracted feature points of the component with feature points extracted from a 3D model of the component which is previously registered to estimate a position and an attitude of the component.SELECTED DRAWING: Figure 9

Description

Technical Field

[0001] The present invention relates to a method for controlling a component supply device, a program, and a component supply device.

Background Art

[0002] In recent years, a component supply device has been proposed that takes out a small number of components from a stacked group of components and supplies them to a predetermined location. Also, when picking up a component loaded on a pick table, the component is photographed by a camera to recognize the position and orientation of the component.

[0003] As a conventional technique for recognizing components, for example, there is one described in Patent Document 1. Patent Document 1 describes extracting a shielding contour line of a subject in a multi-viewpoint image from multi-viewpoint images of the subject photographed by a plurality of cameras and camera parameters of each camera. Then, from the camera parameters, a corresponding tangent line is obtained at each point constituting the shielding contour line, the position of the contact point with the surface of the subject on the calculated tangent line is estimated, and the contact point is used as a feature point on the surface of the subject to restore the three-dimensional shape of the subject that restores the shape of the surface of the subject.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the technique described in Patent Document 1 aims to restore the details of the three-dimensional shape of the subject, so it requires shooting from a large number of viewpoints. As a result, the technique described in Patent Document 1 has a problem that the time for photographing components on the pick table becomes long and the productivity as a component supply device decreases.

[0006] In view of the above-mentioned conventional problems, the present invention aims to provide a control method, program, and component supply device for a component supply device that can improve the efficiency of component picking operations. [Means for solving the problem]

[0007] To solve the above problems and achieve the objectives of the present invention, the control method for the parts supply device includes the following processes (1) to (3). (1) A process of photographing parts loaded on a picking table from multiple different viewpoints. (2) A process to extract feature points of a part based on image information taken from multiple different viewpoints. (3) A process to estimate the position and orientation of a part by comparing the feature points of the extracted part with the feature points extracted from a 3D model of the part that was registered in advance.

[0008] Furthermore, the program of the present invention causes a computer to perform the following steps. A procedure for photographing parts stacked on a picking stand from multiple different viewpoints. A procedure for extracting characteristic points of the part based on image information taken from multiple different viewpoints. A procedure for estimating the position and orientation of a part by comparing the extracted feature points of the part with feature points extracted from a 3D model of the part that has been registered in advance.

[0009] Furthermore, the parts supply device of the present invention comprises a picking table on which parts are stacked, a supply unit that picks up the parts stacked on the picking table and supplies them to a predetermined location, a camera that photographs the parts stacked on the picking table, and a control unit that determines the state of the parts stacked on the picking table based on the image information captured by the camera. The control unit extracts feature points of the parts based on image information of the parts stacked on the picking table taken from multiple different viewpoints, and estimates the position and orientation of the parts by comparing the extracted feature points of the parts with feature points extracted from a 3D model of the parts that has been registered in advance. [Effects of the Invention]

[0010] According to the control method, program, and component supply device of the component supply device configured as described above, the efficiency of component picking operations can be improved. [Brief explanation of the drawing]

[0011] [Figure 1] This is a perspective view of a component supply device according to an embodiment of the present invention. [Figure 2] This is a top view of a parts supply device according to an embodiment of the present invention. [Figure 3] This is a side view of a parts supply device according to an embodiment of the present invention. [Figure 4] This is a side view of the supply unit in a parts supply device according to an embodiment of the present invention. [Figure 5] This is a perspective view of the hand of the supply unit in a parts supply device according to an embodiment of the present invention. [Figure 6] This is a perspective view of a pick stand in a parts supply device according to an embodiment of the present invention. [Figure 7] This is a block diagram showing an example of the configuration of a control system in a component supply device according to an embodiment of the present invention. [Figure 8] This figure illustrates the parts supply operation of a parts supply device according to an embodiment of the present invention. [Figure 9] This flowchart shows an example of a pickup operation in a parts supply device according to an embodiment of the present invention. [Figure 10] This is an explanatory diagram illustrating the component region extraction process. [Figure 11] This flowchart shows the interference detection operation for the first example. [Figure 12] This is an explanatory diagram showing the interference detection operation for the first example. [Figure 13] This flowchart shows the interference detection operation for the second example. [Figure 14] This is an explanatory diagram showing the interference detection operation for the second example. [Figure 15]It is a flowchart showing the interference determination operation according to the third example. [Figure 16] It is an explanatory diagram showing the first photographing method. [Figure 17] It is a flowchart according to the first photographing method. [Figure 18] It is an explanatory diagram showing the second photographing method. [Figure 19] It is a flowchart according to the second photographing method. [Figure 20] It is an explanatory diagram showing the third photographing method. [Figure 21] It is a flowchart showing the method of setting the first photographing position. [Figure 22] It is a flowchart showing the method of setting the second photographing position. [Figure 23] It is a flowchart showing the method of setting the third photographing position. [Figure 24] It shows the method of setting the third photographing position and is an explanatory diagram showing the result of physical simulation. [Figure 25] It is a flowchart showing the method of setting the first feature point. [Figure 26] It is an explanatory diagram showing the feature points on the 3D model. [Figure 27] It is an explanatory diagram showing the feature points on the 3D model. [Figure 28] It is a flowchart showing the method of setting the second feature point. [Figure 29] It is a flowchart showing the method of setting the third feature point.

Embodiments for Carrying Out the Invention

[0012] Hereinafter, embodiments for carrying out the present invention will be described with reference to FIGS. 1 to 29. In each figure, common members are denoted by the same reference numerals. Further, the present invention is not limited to the following embodiments.

[0013] 1. Embodiment Example 1-1. Configuration of Component Feeding Device First, the configuration of the component supply device according to the embodiment will be described with reference to Figures 1 to 3. Figure 1 is a perspective view of the parts supply device. Figure 2 is a top view of the parts supply device. Figure 3 is a side view of the parts supply device.

[0014] As shown in Figure 1, the parts supply device 1 comprises a frame 2, storage sections 3A and 3B, a supply section 4, picking tables 5A and 5B, placing tables 6A and 6B, and a control board 7. The storage sections 3A and 3B, the supply section 4, the picking tables 5A and 5B, the placing tables 6A and 6B, and the control board 7 are mounted on the frame 2. The parts supply device 1 places the parts stored in the storage sections 3A and 3B onto the placing tables 6A and 6B in the same orientation and supplies them to the device for the next process.

[0015] Frame 2 is formed in a roughly rectangular parallelepiped shape and has width, depth, and height. In Figures 1 to 3, the X-axis direction indicates the width direction of Frame 2, the Y-axis direction indicates the depth direction of Frame 2, and the Z-axis direction indicates the height direction of Frame 2. The X-axis and Y-axis directions correspond to the two horizontal axes, which are two axes parallel to the horizontal plane, and the Z-axis direction corresponds to the vertical direction, which is perpendicular to the horizontal plane. Frame 2 is composed of a horizontal member extending in the X-axis direction or the Y-axis direction and a vertical member extending in the Z-axis direction.

[0016] The storage sections 3A and 3B are located on one side of the frame 2 in the Y-axis direction. The storage sections 3A and 3B face each other with an appropriate distance between them in the X-axis direction. The storage sections 3A and 3B are formed in a roughly box shape with an open top. The storage sections 3A and 3B are provided with a lifting mechanism that moves the bottom in the Z-axis direction. This allows each storage section 3A and 3B to change its storage capacity and the height position of the stored components.

[0017] For example, a first component is stored in storage section 3A, and a second component, different from the first component, is stored in storage section 3B. In this case, the component supply device 1 supplies the first component and the second component to the device for the next process. Alternatively, the first component may be stored in storage sections 3A and 3B during a first period, and the second component may be stored in storage sections 3A and 3B during a second period different from the first period. In this case, the component supply device 1 supplies the first component to the device for the next process during the first period, and supplies the second component to the device for the next process during the second period.

[0018] The supply unit 4 is located approximately in the center of the upper part of the frame 2. The supply unit 4 grasps one or more parts from the large quantity of first parts or large quantity of second parts stored in the storage units 3A and 3B and drops them onto the picking tables 5A and 5B for supply. As a result, the first parts or second parts are placed on the picking tables 5A and 5B. The supply unit 4 also grasps the first parts or second parts placed on the picking tables 5A and 5B one by one and supplies them to the placing tables 6A and 6B. The configuration of the supply unit 4 will be explained later with reference to Figures 4 and 5.

[0019] The picking tables 5A and 5B are located on both sides of the supply unit 4 in the X-axis direction. Furthermore, the picking tables 5A and 5B are adjacent to the storage units 3A and 3B, respectively, in the Y-axis direction. The picking tables 5A and 5B are located above the storage units 3A and 3B.

[0020] In the Z-axis direction, a portion of the pick stand 5A overlaps with the storage section 3A. As a result, parts that fall from a portion of the pick stand 5A are stored (returned) to the storage section 3A. In the Z-axis direction, a portion of the pick stand 5B overlaps with the storage section 3B. As a result, parts that fall from a portion of the pick stand 5B are stored (returned) to the storage section 3B. The configurations of the pick stands 5A and 5B will be described later with reference to Figure 6.

[0021] The placing tables 6A and 6B have a belt conveyor that transports parts in the Y-axis direction. The placing tables 6A and 6B are also attached to an X-axis movement mechanism. The X-axis movement mechanism moves the placing tables 6A and 6B in the X-axis direction. The placing tables 6A and 6B transport the parts supplied from the supply unit 4 in the Y-axis direction and position them in predetermined positions. The positioned parts are then supplied to the next process device.

[0022] As shown in Figures 1 and 3, the control board 7 is mounted on the side of the frame 2. The control board 7 is equipped with a control unit 71 (see Figure 7) that controls the operation of the housing units 3A and 3B, the supply unit 4, and the place tables 6A and 6B.

[0023] 1-2. Configuration of the supply unit Next, the configuration of the supply unit 4 will be explained with reference to Figures 4 and 5. Figure 4 is a side view of the supply unit 4 in the parts supply device 1. Figure 5 is a perspective view of the hand of the supply unit 4 in the parts supply device 1.

[0024] As shown in Figure 4, the supply unit 4 comprises an arm block 41 and a hand block 42 connected to the arm block 41. The arm block 41 has a support base 411 and an arm 412 attached to the support base 411. The support base 411 is fixed to the frame 2. The support base 411 rotatably supports the arm 412.

[0025] The arm 412 moves the hand block 42 freely in the X-axis, Y-axis, and Z-axis directions. The arm 412 also rotates the hand block 42 freely around the X-axis, Y-axis, and Z-axis. The arm 412 includes a base member 413, a first link member 414, a second link member 415, and a connecting member 416.

[0026] The base member 413 is rotatably connected to the support base 411. The base member 413 rotates about the Z-axis (first axis). One end of the first link member 414 is rotatably connected to the base member 413. The first link member 414 rotates about a horizontally extending axis (second axis).

[0027] The second link member 415 has a pivot portion 415a and a swivel portion 415b connected to the pivot portion 415a. The pivot portion 415a is rotatably connected to the other end of the first link member 414. The pivot portion 415a rotates around a horizontally extending axis (third axis). The swivel portion 415b is rotatably connected to the pivot portion 415a. The swivel portion 415b rotates around an axis (fourth axis) extending in the direction of connection with the pivot portion 415a.

[0028] The connecting member 416 has a rotating portion 416a and a swivel portion 416b connected to the rotating portion 416a. The rotating portion 416a is rotatably connected to the swivel portion 415b of the second link member 415. The rotating portion 416a rotates around an axis (fifth axis) that extends horizontally. The swivel portion 416b is rotatably connected to the rotating portion 416a. The swivel portion 416b rotates around an axis (sixth axis) that extends in the direction of connection with the rotating portion 416a. The directions in which the second axis, third axis and fourth axis extend are parallel.

[0029] As shown in Figure 5, the hand block 42 has a housing 421 and a hand 422 and camera 423 attached to the housing 421. The housing 421 is connected to the swivel portion 416b of the connecting member 416 in the arm 412. The housing 421 is a roughly rectangular parallelepiped housing. On the lower surface of the housing 421, there is a hand hole 421a through which the hand 422 passes, and a lens hole 421b that exposes the objective lens of the camera 423.

[0030] The hand 422 is composed of multiple (two in this embodiment) gripping pieces 422a. Inside the housing 421 are an opening / closing mechanism for opening and closing the multiple gripping pieces 422a, and a lifting / lowering mechanism for raising and lowering the multiple gripping pieces. As the multiple gripping pieces 422a are raised and lowered by the lifting / lowering mechanism, the length that protrudes from the hand hole 421a changes. If the length that the multiple gripping pieces 422a protrude from the hand hole 421a is increased, the space for holding parts increases, and the number of parts that can be gripped increases. On the other hand, if the length that the multiple gripping pieces 422a protrude from the hand hole 421a is shortened, the space for holding parts decreases, and the number of parts that can be gripped decreases.

[0031] Multiple gripping pieces 422a can also grip a single part at their tip. The hand 422 grips one or more parts from a large number of parts stored in the storage section 3A or storage section 3B and supplies them to the picking table 5A or picking table 5B. On the other hand, the hand 422 grips one part from one or more parts placed on the picking table 5A or picking table 5B and supplies it to the placing table 6A or placing table 6B.

[0032] Furthermore, the widthwise length of the gripping piece 422a (hand fingertip width) is set to W_h. Also, the distance between the two gripping pieces 422a when they are open (finger spread width) is set to W_f. Information regarding the hand fingertip width W_h and the finger spread width W_f of the gripping piece 422a is stored in the memory unit 72, which will be described later.

[0033] Camera 423 shows one specific example of the detection unit according to the present invention. Camera 423 has an image sensor, a plurality of lenses including an objective lens, a polarizing filter, illumination, etc. Camera 423 is housed in a housing 421. The objective lens of camera 423 is exposed through a lens hole 421b of the housing 421.

[0034] The images (video) captured by camera 423 are transmitted to control unit 71, which will be described later. The control unit 71 detects information such as the positions of storage units 3A and 3B and picking tables 5A and 5B from the images captured by camera 423.

[0035] 1-3. Pick stand configuration Next, the configuration of picking stands 5A and 5B will be explained with reference to Figure 6. Figure 6 is a perspective view of the picking table 5A in the parts supply device 1.

[0036] Picking tables 5A and 5B have the same configuration. Therefore, here we will explain the configuration using picking table 5A as an example. As shown in Figure 6, picking table 5A has a tray 51 that forms the loading surface and three wall plates 52 to 54 that are continuous with the tray 51.

[0037] The tray 51 consists of a roughly rectangular plate. The plane of the tray 51 is roughly perpendicular to the Z-axis direction. The tray 51 has two sides that are roughly parallel to the X-axis direction and two sides that are roughly parallel to the Y-axis direction. The wall plate 52 protrudes roughly perpendicularly from the side of the tray 51 that is furthest from the storage section 3A (see Figure 2) of the two sides that are roughly parallel to the X-axis direction. The wall plates 53 and 54 also protrude roughly perpendicularly from the two sides of the tray 51 that are roughly parallel to the Y-axis direction.

[0038] The wall plates 52-54 prevent supplied parts from falling from the tray 51. The sides of the tray 51 without wall plates overlap the opening of the storage section 3A in the Z-axis direction. As a result, parts that fall from the sides of the tray 51 without wall plates are returned to the storage section 3A. In addition, a tilting mechanism is provided at the bottom of the pick stand 5A to tilt the pick stand 5A. The tilting mechanism tilts the pick stand 5A so that the side with wall plate 52 is higher. As a result, parts placed on the pick stand 5A fall from the sides of the tray 51 without wall plates and are collected in the storage section 3A.

[0039] 1-4. Control System Configuration Next, the configuration of the control system of the parts supply device 1 will be explained with reference to Figure 7. Figure 7 is a block diagram showing an example of the control system configuration in the parts supply device 1.

[0040] The control board 7 (see Figure 1) is equipped with a control unit 71 and a memory unit 72. The control unit 71 includes a CPU (Central Processing Unit), ROM (Read Only Memory), and RAM (Random Access Memory). Various functions of the control unit 71 are realized by the CPU executing a predetermined processing program stored in the ROM. Examples of the various functions of the control unit 71 include the control of the arm 412 by the arm control unit 712 and the control of the hand 422 by the hand control unit 713.

[0041] As shown in Figure 7, the control unit 71 includes an overall control unit 711, an arm control unit 712, a hand control unit 713, and a recognition control unit 714. The control unit 71 is a specific example of a supply adjustment unit according to the present invention.

[0042] The overall control unit 711 is connected to the arm control unit 712, the hand control unit 713, and the recognition control unit 714. The overall control unit 711 receives detection results from the recognition control unit 714, such as the position of each part including the housing units 3A and 3B and the hand 422, the size of the picking tables 5A and 5B, and the number of parts being gripped by the hand 422.

[0043] The overall control unit 711 performs overall control of the arm control unit 712 and the hand control unit 713 based on the detection results received from the recognition control unit 714 and the supply parameters 723 and characteristic information 724 stored in the storage unit 72.

[0044] The arm control unit 712 is connected to the drive unit of the arm 412. The arm control unit 712 receives control commands from the overall control unit 711. Based on the control commands received from the overall control unit 711, the arm control unit 712 generates an arm drive signal to drive the arm 412 and transmits it to the drive unit of the arm 412. As a result, the arm 412 performs the operation in accordance with the control commands from the overall control unit 711.

[0045] The hand control unit 713 is connected to the drive unit of the hand 422. The hand control unit 713 receives control commands from the overall control unit 711. Based on the control commands received from the overall control unit 711, the hand control unit 713 generates a hand drive signal to drive the hand 422 and transmits it to the drive unit of the hand 422. As a result, the hand 422 performs the operation in accordance with the control commands from the overall control unit 711.

[0046] The recognition control unit 714 is connected to the camera 423. The recognition control unit 714 controls the camera 423 to take pictures based on the shooting parameters 721 stored in the memory unit 72. The recognition control unit 714 also applies image processing to the image data received from the camera 423 based on the image processing parameters (various correction values) stored in the memory unit 72.

[0047] The recognition control unit 714 detects the positions of the storage units 3A and 3B, the picking tables 5A and 5B, and the placing tables 6A and 6B from the image data that has been processed. The recognition control unit 714 also detects the posture of the hand 422 and the number of parts grasped by the hand 422 from the image data that has been processed. Furthermore, the recognition control unit 714 detects the size (area) of the picking tables 5A and 5B, the shape of the picking tables 5A and 5B (presence or absence of wall plates), and the outline (contour) of the parts placed on the picking tables 5A and 5B from the image data that has been processed. Finally, the recognition control unit 714 transmits the detection results to the overall control unit 711.

[0048] The memory unit 72 stores the shooting parameters 721, image processing parameters 722, supply parameters 723, and characteristic information 724. The shooting parameters 721 are used when the camera 423 photographs each part (pick stand 5A, 5B, etc.). Examples of shooting parameters include exposure time, illumination intensity, and image size depending on the object being photographed. The image processing parameters 722 are various correction values ​​used when applying image processing to the image data received from the camera 423.

[0049] The supply parameters 723 are used to determine the operation of the supply unit 4 when supplying parts to the picking table 5A or the picking table 5B. The supply parameters 723 are stored in the storage unit 72 beforehand.

[0050] The characteristic information 724 is at least one of the following: the shape of the part, the weight of the part, the center of gravity of the part, the material of the part, the surface properties of the part, the surface friction coefficient of the part, and the color of the part. The characteristic information 724 is stored in the storage unit 72 in advance for each type of part. Alternatively, the control unit 71 may extract the characteristic information 724 from the 3D model data of the part. In this case, the 3D model data of the part is stored in the storage unit 72 in advance.

[0051] 2. Example of parts supply device operation 2-1. Parts supply operation Next, we will describe an example of the operation of the parts supply device 1. First, we will explain the parts supply operation with reference to Figure 8. Figure 8 is a diagram illustrating the parts supply operation of the parts supply device 1.

[0052] As shown in Figure 8, in order for the parts supply device 1 to supply parts to the next process device, the parts are first stored in the storage sections 3A and 3B (hereinafter referred to as "storage section 3"). The storage of parts in storage section 3 may be performed by the device in the previous process, or it may be performed by a person.

[0053] Next, the supply unit 4 grasps one or more parts from the large number of parts in the storage unit 3 and supplies them to the pick stand 5A or pick stand 5B (hereinafter referred to as "pick stand 5"). At this time, the supply unit 4 performs a supply operation such that the grasped parts are scattered on the pick stand 5. Hereinafter, the supply operation such that the parts are scattered on the pick stand 5 will be referred to as the "part scattering operation".

[0054] Next, camera 423 photographs the picking table 5, and the recognition control unit 714 of the control unit 71 performs an overhead recognition of the picking table 5. At this time, the recognition control unit 714 determines whether or not there are any parts that can be grasped on the picking table 5. If it is determined that there are no parts that can be grasped on the picking table 5, the supply unit 4 grasps one or more parts from the large number of parts in the storage unit 3.

[0055] When the recognition control unit 714 determines that there is a part that can be gripped on the picking table 5, it recognizes (determines) a gripping position for gripping one of the parts on the picking table 5. The supply unit 4 then grips one part and supplies it to the placing tables 6A and 6B (hereinafter referred to as "placing table 6"). The placing table 6 positions the supplied part in a predetermined position. The part positioned in the predetermined position is supplied to the device for the next process.

[0056] When the supply unit 4 supplies one part to the placing table 6, the recognition control unit 714 recognizes (determines) a gripping position for gripping one of the parts on the picking table 5. If there are no parts on the picking table 5 at this time, the supply operation to the placing table 6 is terminated. Then, the supply unit 4 grips one or more parts from the large number of parts in the storage unit 3.

[0057] 2-2. Example of pickup operation of the supply unit Next, an example of the pickup operation in the supply unit 4 will be described with reference to Figure 9. Figure 9 is a flowchart illustrating an example of the pickup operation. Figure 10 is an explanatory diagram showing the component area extraction process.

[0058] As shown in Figure 9, first, the supply unit 4 grasps (supplies) parts from the storage unit 3 (step S1). Then, the supply unit 4 loads the grasped parts onto the tray 51 of the picking table 5 (step S2). Then, the control unit 71 takes a picture of the tray 51 with the camera 423 of the supply unit 4 and acquires an overall image I0. (Step S3). Next, the control unit 71 extracts component region A(I0): a1···an from the acquired overall image I0 (Step S4). That is, as shown in Figure 10, the control unit 71 labels the component image I_f from the captured overall image I0 and extracts component region a(i)·(i=1···n).

[0059] Next, a determination is made in step S5 to determine whether the extracted part regions ai are interfering with each other (overlapping). If it is determined in step S5 that there is no interference between the parts, the process proceeds to step S8, which will be described later. If it is determined that there is no interference between the parts, the position and orientation of the parts are estimated based on the extracted part regions ai.

[0060] Furthermore, if interference between parts is determined in step S5, the control unit 71 performs a 3D measurement of the part region ai (step S6). Details of the 3D measurement will be described later. The control unit 71 then estimates the position and orientation of the parts based on the 3D measurement and feature points set in advance from a 3D model of the target parts (step S7). In the process of step S7, the control unit 71 searches for a combination that matches the relative positional relationship between the feature points of the 3D model that have been registered in advance and the feature points obtained by the 3D measurement, and estimates the position and orientation of the parts.

[0061] Next, the control unit 71 determines whether it is possible to calculate the gripping point (position at which the gripping piece 422a of the hand 422 will grip the part) based on the estimated position and orientation of the part (step S8). If it is determined in step 8 that the gripping point can be calculated, the control unit 71 controls the supply unit 4 to pick up the part (step S9).

[0062] In this way, by measuring the part area ai in 3D and estimating the position and orientation of the parts, it is possible to easily estimate the state in which parts overlap or are in contact with each other. As a result, it becomes possible to grasp the parts with the gripping piece 422a of the hand 422 without having to rearrange the overlapping or contacting parts on the picking table 5. As a result, the efficiency of the picking operation can be increased.

[0063] Furthermore, the position and orientation of the part are estimated by comparing the feature points of a pre-registered 3D model with the feature points acquired through 3D measurement. In this way, since it is sufficient to extract feature points that can be compared with the feature points of a pre-registered 3D model, the number of viewpoints from which the part is photographed can be reduced. As a result, the time required for photographing the part for 3D measurement can be shortened, and the efficiency of the pickup process can be improved.

[0064] Next, the control unit 71 determines whether or not there are any parts remaining on the picking table 5 (step S10). If the control unit 71 determines that there are still parts on the picking table 5 in step S10 (Yes determination in step S10), the control unit 71 returns to the process of step S5. If the control unit 71 determines that there are no parts on the picking table 5 in step S10 (No determination in step S10), the control unit 71 returns to the process of step S2, takes the parts out of the storage unit 3, and places the parts on the picking table 5.

[0065] In step 8, if it is determined that the gripping point cannot be calculated, the control unit 71 suspends the pickup operation of the part (step S11). After the processing in step S11 is completed, the control unit 71 determines whether all parts on the pick stand 5 are being held (step S12). In the processing in step S12, if it is determined that not all parts are being held (No determination in step S12), the control unit 71 returns to the processing in step S5.

[0066] In contrast, if it is determined in step S12 that all parts are to be held (a Yes determination in step S12), the control unit 71 drives the tilting mechanism to tilt the pick stand 5 and discard the parts on the pick stand 5 (step S13). In other words, in step S13, the parts on the pick stand 5 are collected in the storage unit 3. Then, the process returns to step S2, and parts are resupplied from the storage unit 3 to the pick stand 5.

[0067] By repeating the process described above, the supply operation of parts using the supply unit 4 is performed.

[0068] 2-3. Component Interference Detection Operation Next, the process for determining component interference (overlap) will be explained with reference to Figures 11 to 15. [Example 1] Figure 11 is a flowchart showing the interference detection operation for the first example, and Figure 12 is an explanatory diagram showing the interference detection operation for the first example.

[0069] As shown in Figure 11, first, the component to be subjected to interference detection is specified (step S21). Next, the control unit 71 labels the component image I_f from the captured overall image I0 and obtains the component region ai for interference detection by area, as shown in Figure 12 (step S22). Then, the control unit 71 calculates the area of ​​the obtained component region ai (step S23).

[0070] Next, the control unit 71 determines whether the calculated area is the area of ​​a "no contact / overlap state" (step S24). Information regarding the area of ​​a "no contact / overlap state" is stored in the memory unit 72 as characteristic information 724. If the processing in step S24 determines that the area is the area of ​​a "no contact / overlap state" (Yes determination in step S24), the control unit 71 determines that the component area ai is in a state of no contact / overlap (step S26).

[0071] In contrast, if the process in step S24 determines that the area is not in a "no contact / overlap" state (No determination in step S24), the control unit 71 determines that the component area ai is in a contact / overlap state (step S25). This completes the interference determination operation for the first example.

[0072] [Second example] Next, the interference detection operation for the second example will be explained with reference to Figures 13 and 14. Figure 13 is a flowchart showing the interference detection operation for the second example, and Figure 14 is an explanatory diagram showing the interference detection operation for the second example.

[0073] As shown in Figure 13, first, the component to be subjected to interference detection is specified (step S31). Next, the control unit 71 labels the component image I_f from the captured overall image I0 and obtains the component region ai for interference detection based on its contour, as shown in Figure 14 (step S32). Then, the control unit 71 detects the contour D1 of the obtained component region ai (step S33).

[0074] Next, the control unit 71 determines whether the detected contour D1 is a contour in a "non-contact / non-overlapping state" (step S34). Information regarding contours in a "non-contact / non-overlapping state" is stored in the storage unit 72 as characteristic information 724. The determination process in step S34 is determined, for example, by the contour shape or the length of the contour line.

[0075] In the process of step S34, if it is determined that the contour is in a state of "no contact / overlap" (Yes determination in step S34), the control unit 71 determines that the component region ai is in a state of no contact / overlap (step S36).

[0076] In contrast, if the process in step S34 determines that the contour is not in a state of "no contact / overlap" (No determination in step S34), the control unit 71 determines that the component region ai is in a state of contact / overlap (step 35). This completes the interference determination operation for the second example.

[0077] [Third example] Next, the interference detection operation for the third example will be explained with reference to Figure 15. Figure 15 is a flowchart showing the interference detection operation for the third example.

[0078] As shown in Figure 15, first, the component to be subjected to interference detection is specified (step S41). Next, the control unit 71 obtains the number of component regions n from the captured overall image I0 (step S42). Next, the control unit 71 measures the weight of the component on the pick stand 5 using a weight sensor provided on the pick stand 5 (step S43).

[0079] Next, the control unit 71 determines whether the value of "part weight ÷ number of part areas" is within the reference range (step S44). The reference range used in step S44 is stored in the storage unit 72 as characteristic information 724.

[0080] In step S44, if it is determined that the range is within the reference range (Yes determination in step S44), the control unit 71 determines that the parts on the pick stand 5 are not in contact or overlapping (step S46). Conversely, in step S44, if it is determined that the range is outside the reference range (No determination in step S44), the control unit 71 determines that the parts on the pick stand 5 are in contact or overlapping (step S45). This completes the interference determination operation according to the third example.

[0081] 2-4. 3D Measurement Methods Next, the 3D measurement method performed in step S6 shown in Figure 9 will be explained with reference to Figures 16 to 29.

[0082] [Shooting method] First, we will explain the shooting method by referring to Figures 16 to 20. <First shooting method> Figures 16 and 17 illustrate the first shooting method; Figure 16 is an explanatory diagram illustrating the first shooting method, and Figure 17 is a flowchart relating to the first shooting method.

[0083] First, as shown in Figure 17, the part to be measured in 3D is specified (step S51). Next, the control unit 71 calculates the shooting positions P1...Pm (step S52). Then, the arm control unit 712 of the control unit 71 controls the arm 412 of the supply unit 4 to move the camera 423 to the shooting position Pj calculated in step S52, as shown in Figure 16 (S53). Next, the control unit 71 takes a picture using the camera 423 and acquires the image Ij (step S54).

[0084] The control unit 71 then repeats the process from step S53 to step S54 for the number of shooting positions calculated in step S52. This allows the part to be photographed from multiple different viewpoints, and the shooting operation using the first shooting method is completed.

[0085] <Second shooting method> Next, a second imaging method will be described with reference to Figures 18 and 19. Figures 18 and 19 illustrate the second shooting method; Figure 18 is an explanatory diagram illustrating the second shooting method, and Figure 19 is a flowchart related to the second shooting method.

[0086] As shown in Figure 18, in the second shooting method, multiple fixed cameras C1, C2, C3, C4, and C5 are arranged around the pick stand 5. In the second shooting method, shooting is performed using the multiple fixed cameras C1, C2, C3, C4, and C5, rather than the camera 423 installed in the housing 421 of the supply unit 4. Furthermore, the multiple fixed cameras C1, C2, C3, C4, and C5 are each positioned at different shooting locations. The positions of the multiple fixed cameras C1, C2, C3, C4, and C5 are stored in the storage unit 72 as shooting parameters 721.

[0087] In this second shooting method, as shown in Figure 19, first, the part to be measured in 3D is specified (step S61). Next, the control unit 71 selects the shooting position P1...Pm from the positions of the fixed cameras C1...Ci (step S62). Next, the control unit 71 acquires the image Ij taken at the camera Cj designated as the shooting position (step S63). Then, the control unit 71 repeats the process in step S63 for a predetermined number of shooting positions. This allows the part to be photographed from multiple different viewpoints, and the shooting operation using the second shooting method is completed.

[0088] <Third shooting method> Next, the third imaging method will be explained with reference to Figure 20. Figure 20 is an explanatory diagram showing the third imaging method.

[0089] As shown in Figure 20, in the third shooting method, a fixed camera 423F is positioned vertically above the tray 51 of the picking table 5. Alternatively, a camera 423 installed in the housing 421 of the supply unit 4 may be used instead of the fixed camera 423F. In addition, mirrors 57 are installed at the upper ends of the wall plates 52, 53, and 53 of the picking table 5. The mirrors 57 are positioned so that the reflected image of the parts M loaded on the tray 51 is captured by the camera. Therefore, the fixed camera 423F captures images of the parts M not only from one vertical viewpoint, but also from multiple angles due to the reflection by the mirrors 57. This makes it possible to photograph parts from multiple different viewpoints with a single fixed camera 423F.

[0090] Furthermore, the first, second, and third shooting methods described above may be used in combination as the shooting method.

[0091] [How to set the shooting position] Next, we will explain how to set the shooting position with reference to Figures 21 to 24. <How to set the first shooting position> Figure 21 is a flowchart showing the method for setting the first shooting position.

[0092] As shown in Figure 21, the part to be measured in 3D is specified (step S71). Next, the control unit 71 takes a picture of the tray 51 with the camera 423 of the supply unit 4 and acquires an overall image I0 (step S72). Next, the control unit 71 calculates combinations of shooting positions that allow each part region a1...an to be photographed and are as far away as possible (step S73).

[0093] Next, the control unit 71 determines whether there is a combination of viewpoints within the acceptable cycle time range (step S74). Here, the acceptable cycle time range is the time required for the pickup operation of the supply unit 4 in the parts supply operation by the parts supply device 1, and is registered in advance. Then, in the process of step S74, the acceptable cycle time range is compared with the time required when taking images from two or more viewpoints.

[0094] If the control unit 71 determines that there is "no combination" in step S74, it terminates the process. Conversely, if the control unit 71 determines that there is "a combination" in step S74, it selects the specified number of shooting positions (step S75). In step S75, the combination of viewpoints that is furthest from among multiple viewpoint combinations is selected as the shooting position. This makes it possible to set two or more viewpoints that are as far apart as possible, and to extract more feature points of the parts from the captured image. As a result, the first method for setting the shooting position is completed.

[0095] <How to set the second shooting position> Figure 22 is a flowchart showing the method for setting the second shooting position.

[0096] As shown in Figure 22, the part to be measured in 3D is specified (step S81). Next, the control unit 71 uses the camera 423 of the supply unit 4 to photograph the tray 51 and acquire an overall image I0 (step S82). Then, the control unit 71 selects multiple pre-set shooting positions based on the expected orientation of the part (step S83). In the process of step S83, the pre-set shooting positions are set to an arbitrary number of points that are likely to yield more feature points based on the shape of the target part. This information is then stored for each part as shooting parameters 721 in the storage unit 72.

[0097] Next, the control unit 71 determines whether there is a combination of viewpoints within the acceptable cycle time range from among the multiple shooting positions selected in step S83 (step S84). If the control unit 71 determines that there is "no combination" in the process of step S84, it terminates the process. On the other hand, if the control unit 71 determines that there is "a combination" in the process of step S84, it selects a specified number of shooting positions (step S85). This allows for the observation of a larger number of feature points, taking into account the orientation of the part that can be expected from the shape of the part. As a result, the method for setting the second shooting position is completed.

[0098] <How to set the third shooting position> Figure 23 is a flowchart showing how to set the third shooting position, and Figure 24 is an explanatory diagram showing the physical simulation results, also showing how to set the third shooting position.

[0099] As shown in Figure 23, the part to be measured in 3D is specified (step 91). Next, the control unit 71 acquires the physical simulation results of the part specified in step S91 (step S92). That is, as shown in Figure 24, a physical simulation is performed in which the previously specified part M is randomly stacked on the pick stand 5, and an arbitrary number of viewpoints that are likely to acquire more feature points are set. These physical simulation results and the number of points of the set viewpoints are stored for each part as shooting parameters 721 in the memory unit 72.

[0100] Next, the control unit 71 determines whether there is a combination of viewpoints within the acceptable cycle time range from among the multiple shooting positions acquired in step S92 (step S93). If the control unit 71 determines that there is "no combination" in the process of step S93, it terminates the process. On the other hand, if the control unit 71 determines that there is "a combination" in the process of step S93, it selects the specified number of shooting positions (step S84). This allows a larger number of feature points to be observed through physical simulation. As a result, the third method for setting shooting positions is completed.

[0101] The control unit 71 performs three-dimensional measurement of the part based on image information captured from multiple different viewpoints based on the shooting position set by the method described above. The control unit 71 then extracts feature points that identify the position and orientation of the part from the three-dimensional measurement. The shooting position is set to a position where the part can be measured three-dimensionally, and a shooting position is set in which at least four feature points can be observed.

[0102] [How to set feature points] Next, we will explain how to set feature points used when performing 3D measurements, referring to Figures 25 to 29. <First setup method> First, we will explain how to set the first feature point by referring to Figures 25 to 27. Figure 25 is a flowchart showing how to set the first feature point, and Figures 26 and 27 are explanatory diagrams showing feature points on the 3D model. The white circles in Figures 26 and 27 indicate feature point Q1 of the 3D model N.

[0103] As shown in Figure 25, the 3D model N of the part to be measured in 3D is specified (step S101). Next, the corner radius (R) is specified (step S102), and the corner on the 3D model N is specified (step S103). Then, it is determined whether the corner radius specified in step S103 is less than a threshold (step S104). In the process of step S104, if it is determined that the corner radius is less than the threshold (Yes determination in step S104), that point is set as feature point Q1 (step S105). Then, the process from steps S103 to S105 is repeated for all corners on the 3D model N. In this way, the feature points Q1 of the part to be measured in 3D can be registered in advance.

[0104] The feature points Q1 set in this process are then registered in the memory unit 72 as characteristic information 724 for each part. The memory unit 72 also stores the relative positional relationship (distance, direction) between the feature points Q1. The corners of a 3D model are points that can be measured with fewer viewpoints. As a result, by registering points that can be measured with fewer viewpoints as feature points Q1, the number of viewpoints required for 3D measurement can be reduced.

[0105] <Second setup method> Next, we will explain how to set the second feature point, referring to Figure 28. Figure 28 is a flowchart showing how to set the second feature point.

[0106] As shown in Figure 28, a 3D model N of the part to be measured in 3D is specified (step S111). Next, convex points are detected on the 3D model N using a convex hull algorithm (step S112). Then, each convex point detected in step S112 is set as a feature point Q1 (step S113). This allows the feature points Q1 of the part to be measured in 3D to be registered in advance. In this second setting method as well, feature points Q1 can be set from the 3D model N as shown by the white circles in Figures 26 and 27.

[0107] <Third setting method> Next, we will explain how to set the third feature point, referring to Figure 29. Figure 29 is a flowchart showing the method for setting the third feature point.

[0108] As shown in Figure 29, a 3D model N of the part to be measured in 3D is specified (step S121). Next, a random stacking physical simulation is performed as shown in Figure 24 (step S122). Then, simulations are performed from multiple viewpoints (step S123). Next, points on the 3D model that were observed from multiple viewpoints are designated as feature point candidates (step S124). Furthermore, the process from steps S122 to S124 is repeated any number of times.

[0109] Then, a number of feature point candidates that were observed frequently in multiple simulations are selected from the top and designated as feature points (step S125). This completes the method for setting feature points for the parts to be measured in 3D.

[0110] The example described above illustrates how feature points are automatically set from a 3D model N, but this is not the only way to do so. For example, the user may manually set feature points from the 3D model. As a result, it becomes possible to set feature points that can be observed from a greater number of viewpoints, taking into account the part's orientation as expected from its shape.

[0111] The embodiments, including their effects, have been described above. However, the invention is not limited to the embodiments described above, and various modifications can be made without departing from the gist of the invention as described in the claims.

[0112] Furthermore, some or all of the above-mentioned components, functions, and processing units may be implemented in hardware, for example, by designing an integrated circuit. Alternatively, the above-mentioned components and functions may be implemented in software by having the processor interpret and execute programs that realize each function. Information such as programs, tables, and files that realize each function can be stored in memory, hard disks, SSDs (Solid State Drives), or other recording media such as IC cards, SD cards, and DVDs.

[0113] In this specification, although terms such as "parallel" and "orthogonal" are used, these do not mean only strictly "parallel" and "orthogonal," but may also refer to states that are "approximately parallel" or "approximately orthogonal," which include "parallel" and "orthogonal" and are within a range in which they can perform their functions. [Explanation of Symbols]

[0114] 1...Parts supply device, 2...Frame, 3,3A,3B...Storage section, 4...Supply section, 5,5A,5B...Picking table, 6,6A,6B...Placement table, 7...Control board, 41...Arm block, 42...Hand block, 51...Tray, 52,53...Wall plate, 71...Control unit, 72...Storage unit, 423...Camera (detection unit), 411...Support base, 412...Arm, 413...Base member, 414...First link member, 415...Second link member, 416...Connecting member, 421...Housing, 422...Hand, 422a...Gripping piece, 711...Overall control unit, 712...Arm control unit, 713...Hand control unit, 714...Recognition control unit, 721...Shooting parameters, 722...Image processing parameters, 723...Supply parameters 724...Characteristic Information

Claims

1. The process involves photographing the parts loaded on the picking platform from multiple different viewpoints, A process to extract characteristic points of the part based on image information taken from multiple different viewpoints, A process to estimate the position and orientation of the part by comparing the extracted feature points of the part with feature points extracted from a 3D model of the part that was registered in advance, A process to set the position for photographing the aforementioned part according to the allowable range of the cycle time required for the pickup operation of the aforementioned part, A control method for a parts supply device, including the device itself.

2. This process includes moving the camera used to photograph the aforementioned component to a different shooting position. A control method for a parts supply device according to claim 1.

3. Multiple cameras positioned at different locations are used to photograph the parts from multiple different viewpoints. A control method for a parts supply device according to claim 1.

4. Multiple mirrors are installed on the aforementioned pick stand. The camera that photographs the aforementioned part captures images from multiple different angles. A control method for a parts supply device according to claim 1.

5. The position from which the aforementioned component is photographed is set to a location where at least four feature points can be observed from image information taken from multiple different viewpoints. A control method for a parts supply device according to claim 1.

6. Based on the expected orientation of the component, multiple pre-set shooting positions are selected. From these selected shooting positions, a combination of viewpoints within the acceptable cycle time for the component's pickup operation is selected to determine the position for photographing the component. A control method for a parts supply device according to claim 1.

7. Based on the results of the physical simulation of the aforementioned part, the position for photographing the part is set. A control method for a parts supply device according to claim 1.

8. The procedure for photographing parts loaded on a picking stand from multiple different viewpoints, A procedure for extracting characteristic points of the part based on image information taken from multiple different viewpoints, A procedure for estimating the position and orientation of the part by comparing the extracted feature points of the part with feature points extracted from a 3D model of the part that was registered in advance, A procedure for setting the position for photographing the aforementioned component according to the allowable range of the cycle time required for the pickup operation of the aforementioned component, A program that causes a computer to execute something.

9. A picking table on which parts are loaded, A supply unit that picks up the parts loaded on the picking table and supplies them to a predetermined location, A camera for photographing the parts loaded on the picking platform, The system includes a control unit that determines the state of the parts loaded on the pick stand based on image information captured by the camera, The control unit, Based on image information obtained by photographing the parts loaded on the picking platform from multiple different viewpoints, feature points of the parts are extracted, and the position and orientation of the parts are estimated by comparing the extracted feature points of the parts with feature points extracted from a 3D model of the parts that has been registered in advance. The position for photographing the component is set according to the allowable range of the cycle time required for the component's pickup operation. Parts supply device.

Images