Image processing device, robot control system and control method

Abstract

To provide an image processing device capable of avoiding interference between a container and a robot even if a size of the container is changed.SOLUTION: An image processing device comprises an identifying unit, a searching unit, and a determining unit. The identifying unit identifies a position and posture of a container on the basis of a 2D image obtained by image capture by a camera. The searching unit searches for the position and posture of the camera when the identification by the identifying unit is successful. The determining unit determines a movement path of an end effector for picking an object on the basis of, the position and posture of the object detected using 3D measurement results obtained by image capture by the camera. The determining unit determines the movement path to avoid interference between the container and the end effector on the basis of the position and posture of the container identified by the identifying unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, a robot control system, and a control method. [Background technology]

[0002] Conventionally, robots that pick items one by one from multiple items in a container are known at production sites. It is desirable for the robot to avoid interfering with the container when picking an object. Japanese Patent Application Laid-Open No. 2019-188516 (Patent Document 1) discloses that the position and orientation of a parts box is detected based on perception information acquired by capturing an image of the container, and the detection results are used to perform operations to avoid interference between the container and the robot. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-188516 Summary of the Invention [Problem to be solved by the invention]

[0004] At production sites, container sizes may change. With the technology described in Patent Document 1, if the container size increases, the container may not appear in the image during imaging, making it impossible to detect the container's position and orientation. As a result, interference between the container and the robot cannot be avoided.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and its purpose is to provide an image processing device, a robot control system, and a control method that can avoid interference between a container and a robot even if the size of the container is changed. [Means for solving the problem]

[0006] According to an example of the present disclosure, an image processing device processes images acquired by a camera having a first mode operating as a 3D camera and a second mode operating as a 2D camera. The camera is movably mounted on a robot having an end effector for picking an object from a container. The image processing device includes an identification unit, a search unit, and a determination unit. The identification unit identifies the position and orientation of the container based on a 2D image acquired by image capture by the camera operating in the second mode. The search unit searches for the position and orientation of the camera when identification by the identification unit is successful. The determination unit determines a movement path of the end effector for picking the object based on the position and orientation of the object detected using 3D measurement results acquired by image capture by the camera operating in the first mode. The determination unit determines a movement path to avoid interference between the container and the end effector based on the position and orientation of the container identified by the identification unit.

[0007] According to this disclosure, the position and orientation of the camera when identification by the identification unit is successful is searched for, and the position and orientation of the container is identified based on a 2D image acquired from the camera at the searched position and orientation. Then, based on the identified position and orientation of the container, a movement path of the end effector is determined so that the container and the end effector do not interfere with each other. This makes it possible to avoid interference between the container and the robot even if the size of the container changes.

[0008] In the above disclosure, the search unit starts a search in response to a failure to identify the position and orientation of a container based on a 2D image when the camera is positioned at a reference position, thereby reducing the load on the search unit.

[0009] In the above disclosure, the search unit moves the camera upward from the reference position, which makes it easier for the container to be captured within the field of view of the camera, making it easier for the identification unit to successfully identify the container.

[0010] In the above disclosure, the search unit moves the camera horizontally from a reference position, which makes it easier for the container to be captured within the field of view of the camera, making it easier for the identification unit to successfully identify the container.

[0011] In the above disclosure, the search unit executes a first movement control to move the camera upward from a reference position, and if the first movement control does not succeed in identifying the position and posture of the container, executes a second movement control to move the camera horizontally from the reference position.

[0012] According to the above disclosure, the container is more likely to be captured within the field of view of the camera, which makes it more likely that the identification unit will be able to successfully identify the container.

[0013] In the above disclosure, the reference position is the position of the camera when the object is included in the depth of field of the camera operating in the first mode.

[0014] According to the above disclosure, if the position and orientation of a container are successfully identified based on an image from a camera at a reference position, the movement path of the end effector for picking the target object can be determined by operating the camera in the first mode without moving the camera.

[0015] In the above disclosure, the search unit moves the camera until a characteristic part of the container appears in the 2D image. The identification unit identifies the position of the characteristic part based on the 2D image, and identifies the position and orientation of the container using the position of the characteristic part and shape data indicating the shape of the container.

[0016] According to the above disclosure, it is not necessary to move the camera so that the entire container is within the field of view, which reduces the time required for the search unit to perform a search.

[0017] In the above disclosure, a container contains multiple items, including a target object. The 3D measurement results indicate the positions and orientations of the multiple items. The image processing device further includes a detection unit that detects a change in the position and orientation of the container when a difference between a first 3D measurement result and a second 3D measurement result obtained after the first 3D measurement result exceeds a threshold. This allows the change in the position and orientation of the container to be recognized.

[0018] In the above disclosure, the identifying unit and the searching unit start operating in response to the detection of a change in the position and posture of the container by the detecting unit.

[0019] According to the above disclosure, when the position and orientation of a container changes, the position and orientation of the container after the change can be immediately identified.

[0020] In the above disclosure, the image processing device further includes an adjustment unit that adjusts the home position of the end effector in accordance with the position and orientation of the container identified by the identification unit. According to the above disclosure, unnecessary movement of the end effector can be suppressed.

[0021] According to an example of the present disclosure, a robot control system includes a robot, a camera, an identification unit, a search unit, and a determination unit. The robot has an end effector that picks an object from a container. The camera is movably mounted on the robot and has a first mode in which it operates as a 3D camera and a second mode in which it operates as a 2D camera. The identification unit identifies the position and orientation of the container based on a 2D image captured by the camera operating in the second mode. The search unit searches for the position and orientation of the camera when identification by the identification unit is successful. The determination unit identifies the position and orientation of the object based on a 3D measurement result captured by the camera operating in the first mode, and determines a movement path of the end effector for picking the object based on the identified position and orientation of the object. The determination unit determines a movement path based on the position and orientation of the container identified by the identification unit so as to avoid interference between the container and the end effector.

[0022] According to an example of the present disclosure, a control method for an image processing device that processes images acquired by a camera having a first mode in which the camera operates as a 3D camera and a second mode in which the camera operates as a 2D camera includes the following first to third steps. The camera is movably mounted on a robot having an end effector that picks up an object in a container. The first step is a step of identifying the position and orientation of the container based on a 2D image acquired by imaging with the camera operating in the second mode. The second step is a step of searching for the position and orientation of the camera when the position and orientation of the container are successfully identified. The third step is a step of determining a movement path of the end effector for picking the object based on the position and orientation of the object identified using 3D measurement results acquired by imaging with the camera operating in the first mode. The third step includes a step of determining a movement path based on the identified position and orientation of the container so as to avoid interference between the container and the end effector.

[0023] These disclosures make it possible to avoid interference between the container and the robot even if the size of the container is changed. [Effects of the Invention]

[0024] According to the present disclosure, even if the size of the container is changed, interference between the container and the robot can be avoided. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a schematic diagram illustrating an overall configuration of a robot control system according to an embodiment. [Figure 2] FIG. 2 is a schematic diagram showing the concept of a specific structure of the camera shown in FIG. [Figure 3] 2 is a schematic diagram illustrating an example of a hardware configuration of the image processing device illustrated in FIG. 1. FIG. [Figure 4] FIG. 1 is a diagram illustrating an example of a coordinate system used in a robot control system. [Figure 5]2 is a block diagram showing an example of a functional configuration of the image processing device shown in FIG. 1. FIG. [Figure 6] FIG. 1 illustrates an example of a container. [Figure 7] 10A and 10B are diagrams illustrating an example of a method for identifying the position and orientation of a container. [Figure 8] 10A and 10B are diagrams illustrating another example of a method for identifying the position and orientation of a container. [Figure 9] FIG. 10 is a diagram illustrating an example of a search process for a camera position. [Figure 10] 10 is a flowchart illustrating an example of a processing flow of the image processing device. [Figure 11] FIG. 10 is a block diagram showing a functional configuration of an image processing device according to a first modification. [Figure 12] FIG. 10 is a block diagram showing a functional configuration of an image processing device according to a second modification. [Figure 13] 13 is a diagram illustrating the processing of the positional deviation detection unit shown in FIG. 12. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0026] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail with reference to the accompanying drawings, in which the same or corresponding parts in the drawings are designated by the same reference numerals and the description thereof will not be repeated.

[0027] §1 Application Examples An example of a situation in which the present invention is applied will be described with reference to Fig. 1. Fig. 1 is a schematic diagram showing the overall configuration of a robot control system according to an embodiment. The robot control system 1 illustrated in Fig. 1 is incorporated into, for example, a production line, and is used to control the picking of a selected object from one or more items 2 bulk-stacked in a container 5.

[0028] As shown in FIG. 1, the robot control system 1 includes an image processing device 100, a camera 200, a robot 300, and a robot control device 400.

[0029] The camera 200 captures an image of a subject including the item 2. The camera 200 outputs image data obtained by capturing the image (hereinafter simply referred to as an “image”) to the image processing device 100.

[0030] The camera 200 has a first mode in which it operates as a 3D camera and a second mode in which it operates as a 2D camera, and captures an image of a subject in a mode corresponding to an image capture instruction from the image processing device 100. For example, when the first mode is specified, the camera 200 captures an image of the subject while projecting a projection pattern light onto the subject. The projection pattern is, for example, a pattern in which brightness changes periodically along a predetermined direction within the irradiation surface. When the second mode is specified, the camera 200 captures an image of the subject while irradiating the subject with uniform light.

[0031] The first mode is specified when measuring the position and orientation of one or more items 2 in the container 5. The range R in which the position and orientation can be measured by imaging in the first mode is determined according to the viewing angle and depth of field of the camera 200.

[0032] The second mode is specified when determining the position and orientation of the container 5. The container 5 is placed on a predetermined surface (for example, the top of a desk, the floor, etc.). Therefore, the height of the top surface of the container 5 is known. Therefore, the position and orientation of the container 5 can be detected even using an image obtained by imaging in the second mode (hereinafter referred to as a "2D image"). Furthermore, the container 5 is larger in size than the item 2. Therefore, even if the container 5 is located outside the depth of field of the camera 200 and the 2D image is blurred, the position and orientation of the container 5 can be determined by applying image processing such as edge enhancement.

[0033] The robot 300 picks up an object in the container 5. The robot 300 is, for example, a vertical articulated robot. The robot 300 illustrated in FIG. 1 includes a base 301, an articulated arm 302, a flange plate 303, a support member 304, and an end effector 305. Note that the robot 300 is illustrated schematically in FIG. 1, and the shape of each member is not limited to the example illustrated in FIG. 1.

[0034] Base 301 is installed at a fixed position in the production site. One end of articulated arm 302 is connected to base 301. A flange plate 303 is provided at the other end of articulated arm 302. An end effector 305 is attached to flange plate 303 via a support member 304. End effector 305 is a component that picks up an object, and includes, for example, a two-fingered hand, a multi-fingered hand, a suction pad, etc.

[0035] The camera 200 is movably mounted on the robot 300. In this embodiment, the camera 200 is mounted on the robot 300 so that, for example, its relative position with respect to the end effector 305 is constant. Specifically, the camera 200 is attached to a support member 304 of the robot 300.

[0036] The robot control device 400 controls the articulated arm 302 in accordance with a movement command from the image processing device 100. The movement command indicates, for example, the position and orientation of the flange plate 303. The robot control device 400 controls the articulated arm 302 so that the position and orientation of the flange plate 303 coincide with the movement command.

[0037] The image processing device 100 determines the position and orientation that the end effector 305 should take, based on the image received from the camera 200. The image processing device 100 generates a movement command for moving the end effector 305 to the determined position and orientation, and outputs the generated movement command to the robot control device 400. The image processing device 100 may also output an image capture command to the camera at the timing when the end effector 305 moves to the determined position and orientation.

[0038] As shown in FIG. 1, the image processing device 100 includes a specifying unit 11, a searching unit 12, and a determining unit 13.

[0039] The identification unit 11 identifies the position and orientation of the container 5 based on the 2D image obtained by imaging with the camera 200 operating in the second mode. For example, the identification unit 11 identifies the position and orientation of the container 5 based on the positions of characteristic parts of the container 5 (for example, corners of the top surface 501 of the container 5) that appear in the 2D image and shape data of the container 5.

[0040] The search unit 12 searches for the position and orientation of the camera 200 when the identification by the identification unit 11 is successful. In the state of the robot 300 shown in the upper part of FIG. 1 , the top surface 501 of the container 5 is not located within the field of view of the camera 200, and therefore the top surface 501 of the container 5 is not captured in the 2D image obtained from the camera 200. Therefore, the identification unit 11 cannot identify the position and orientation of the container 5 based on the 2D image obtained from the camera 200 in this state. Therefore, the search unit 12 outputs a movement command to the robot control device 400 to move the position of the camera 200 until the identification by the identification unit 11 is successful. For example, as shown in the lower part of FIG. 1 , the search unit 12 outputs a movement command to the robot control device 400 to move the camera 200 in a direction away from the container 5, and searches for the position and orientation of the camera 200 such that the top surface 501 of the container 5 is included in the field of view. As a result, the top surface 501 of the container 5 appears in the 2D image obtained from the camera 200, and the position and orientation of the container 5 are successfully identified by the identification unit 11.

[0041] The determination unit 13 measures the position and orientation of one or more items 2 in the container 5 based on the 3D measurement results obtained by imaging by the camera 200 operating in the first mode. The determination unit 13 selects one of the items 2 whose position and orientation has been measured as a target object, and determines a movement path of the end effector 305 for picking up the target object. Based on the position and orientation of the container 5 identified by the identification unit 11, the determination unit 13 determines a movement path that avoids interference between the container 5 and the end effector 305.

[0042] According to this embodiment, the position and orientation of the camera 200 when the identification by the identification unit 11 is successful is searched for, and the position and orientation of the container 5 is identified based on the 2D image obtained from the camera 200 at the searched position and orientation. As a result, a movement path of the end effector 305 is determined based on the identified position and orientation of the container 5 so that the container 5 does not interfere with the end effector 305. As a result, even if the size of the container 5 is changed, interference between the container 5 and the robot 300 can be avoided.

[0043] §2 Specific examples <Camera configuration> Fig. 2 is a schematic diagram showing the concept of the specific structure of the camera shown in Fig. 1. As shown in Fig. 2, the camera 200 has a projection unit 21 and an imaging unit 22. The projection unit 21 projects any projection pattern light onto a subject in accordance with instructions from the image processing device 100. The imaging unit 22, in accordance with instructions from the image processing device 100, images the subject with the projection pattern light projected onto it.

[0044] The projection unit 21 has, as its main components, a light source 210 such as an LED (Light Emitting Diode) or a halogen lamp, a filter 211, and an optical system 212. The projection unit 21 is configured as an assembly formed by assembling together the light source 210, the filter 211, the optical system 212, a lens barrel (not shown), and a support structure (not shown).

[0045] The light source 210 irradiates light of a predetermined wavelength toward the filter 211. The filter 211 arbitrarily changes the in-plane light transmittance in accordance with a command from the image processing device 100. The filter 211 includes, for example, a photomask, a liquid crystal, a DMD (Digital Mirror Device), etc. When the filter 211 receives an image capture command in a first mode from the image processing device 100, it generates a projection pattern light required for measuring a three-dimensional shape. When the filter 211 receives an image capture command in a second mode from the image processing device 100, it makes the in-plane transmittance uniform. The optical system 212 includes one or more lenses. The light that has passed through the filter 211 is irradiated to the outside via the optical system 212. As a result, a projection pattern light according to the state of the filter 211 is irradiated to the external space.

[0046] The imaging unit 22 has an imaging element 220 as an optical device, an optical system 221, and a lens barrel and support structure (not shown). That is, the imaging unit 22 is configured as an assembly in which the imaging element 220, the optical system 221, the lens barrel, and the support structure are assembled together. The imaging unit 22 captures an image of an object with light projected by the projection unit 21. More specifically, an image is obtained when the imaging element 220 receives light that has passed through the optical system 221. In the first mode, the imaging unit 22 captures an image of an object with a projection pattern light required for measuring a three-dimensional shape projected onto the object, and in the second mode, the imaging unit 22 captures an image of an object with uniform light projected onto the object. In this way, the imaging unit 22 is shared in both the first mode and the second mode.

[0047] The viewing angle of the camera 200 is determined by the optical system 221. In this embodiment, the camera 200 has an imaging unit 22 that is shared by both the first mode and the second mode. Therefore, the viewing angle of the camera 200 is the same in both the first mode and the second mode. If the viewing angle of the camera 200 is widened, the resolution decreases. The camera 200 is primarily used to measure the position and orientation of one or more articles 2. Therefore, the viewing angle of the camera 200 is set so as to obtain the resolution required for 3D measurement based on the image obtained in the first mode.

[0048] It should be noted that by changing the specifications of the optical system 221, it is possible to obtain a wide field of view while maintaining high resolution; however, in this case, the size and weight of the optical system 221 will increase. As described above, the camera 200 is preferably small and lightweight because it is mounted on the robot 300 and moves. Therefore, the field of view of the camera 200 is limited to a range that does not affect the movement of the camera 200. As a result, if the size of the container 5 is large, there is a possibility that the top surface 501 of the container 5 will not be included in the field of view of the camera 200 at the position of the camera 200 when the range R in which the position and orientation can be measured by imaging in the first mode includes the article 2 in the container 5 (see FIG. 1 ).

[0049] <Hardware configuration of image processing device> Image processing device 100 is typically a computer having a general-purpose architecture, and executes a pre-installed program (instruction code) to perform the processing according to this embodiment. Such a program is typically distributed in a state stored on various recording media, or is installed in image processing device 100 via a network, etc.

[0050] When using such a general-purpose computer, an OS (Operating System) for executing basic computer processing may be installed in addition to an application for executing processing according to the present embodiment. In this case, the program according to the present embodiment may execute processing by calling necessary modules from among program modules provided as part of the OS in a predetermined sequence at a predetermined timing. In other words, the program according to the present embodiment itself may not include the above-mentioned modules, and may execute processing in cooperation with the OS. The program according to the present embodiment may also be in a form that does not include some of these modules.

[0051] Furthermore, the program according to the present embodiment may be provided by being incorporated into a part of another program. In this case, the program itself does not include the modules included in the other program to be combined as described above, and executes processing in cooperation with the other program. In other words, the program according to the present embodiment may be in a form incorporated into such other program. Note that some or all of the functions provided by the execution of the program may be implemented as dedicated hardware circuits.

[0052] Fig. 3 is a schematic diagram showing an example of the hardware configuration of the image processing device shown in Fig. 1. As shown in Fig. 3, the image processing device 100 includes a CPU (Central Processing Unit) 101, which is an arithmetic processing unit, a main memory 102 and a hard disk 103, which are storage units, a camera interface 104, an input interface 105, a display controller 106, a communication interface 107, and a data reader / writer 108. These units are connected to each other via a bus 109 so as to be able to communicate data with each other.

[0053] CPU 101 loads programs (codes) installed on hard disk 103 into main memory 102 and executes them in a predetermined order to perform various calculations. Main memory 102 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory), and stores images captured by camera 200 in addition to programs read from hard disk 103. Furthermore, hard disk 103 stores various data, as will be described later. Note that in addition to hard disk 103, or instead of hard disk 103, a semiconductor storage device such as a flash memory may be used.

[0054] Camera interface 104 mediates data transmission between CPU 101 and camera 200. In other words, camera interface 104 is connected to camera 200. Camera interface 104 issues an image capture command to camera 200 in accordance with an internal command generated by CPU 101. The image capture command includes projection conditions and image capture conditions. Camera interface 104 includes image buffer 104a for temporarily storing images from camera 200. When a predetermined number of images have been stored in image buffer 104a, camera interface 104 transfers the stored images to main memory 102.

[0055] The input interface 105 mediates data transmission between the CPU 101 and the input device 150. That is, the input interface 105 accepts input information input by a user to the input device 150. The input device 150 includes a keyboard, a mouse, a touch panel, and the like.

[0056] The display controller 106 is connected to a display 160 and controls the screen of the display 160 so as to notify the user of the processing results of the CPU 101 and the like.

[0057] The communication interface 107 mediates data transmission between the CPU 101 and external devices such as the robot control device 400. The communication interface 107 is typically implemented by Ethernet (registered trademark) or USB (Universal Serial Bus).

[0058] Data reader / writer 108 mediates data transmission between CPU 101 and memory card 700, which is a recording medium. That is, memory card 700 is distributed in a state in which programs to be executed by image processing device 100 and the like are stored, and data reader / writer 108 reads the programs from memory card 700. Furthermore, in response to an internal command from CPU 101, data reader / writer 108 writes images captured by camera 200 and / or processing results in image processing device 100 to memory card 700. Note that memory card 700 is formed from a general-purpose semiconductor storage device such as SD (Secure Digital), a magnetic storage medium such as a flexible disk, an optical storage medium such as a CD-ROM (Compact Disk Read Only Memory), or the like.

[0059] Furthermore, the image processing device 100 may be provided with an interface that can be connected to at least one of an RFID (Radio Frequency Identification) reader and a barcode reader.

[0060] <Coordinate system> The CPU 101 calculates the positions and orientations of the article 2, the camera 200, and the robot 300 using coordinate values ​​in multiple coordinate systems.

[0061] 4 is a diagram showing an example of coordinate systems used in a robot control system 1. As shown in FIG. 4, the robot control system 1 uses a robot base coordinate system, a flange coordinate system (also called a tool coordinate system), a camera coordinate system, and a workpiece coordinate system.

[0062] The robot base coordinate system is a coordinate system based on the base 301 of the robot 300. The robot base coordinate system is defined by an origin Ob set on the base 301 and base vectors Xb, Yb, and Zb.

[0063] The flange coordinate system is a coordinate system based on a flange plate 303 provided at the other end of the articulated arm 302 of the robot 300. An end effector 305 and a camera 200 are attached to the flange plate 303 via a support member 304. The flange coordinate system is defined by an origin Of set on the flange surface of the flange plate 303 and basis vectors Xf, Yf, and Zf.

[0064] The camera coordinate system is a coordinate system based on the imaging unit 22 of the camera 200. The camera coordinate system is defined by an origin Oc set in the imaging unit 22 and base vectors Xc, Yc, and Zc. The base vector Zc is on the optical axis of the imaging unit 22.

[0065] The workpiece coordinate system is a coordinate system based on the article 2. The workpiece coordinate system is defined by an origin Ow set on the article 2 and base vectors Xw, Yw, and Zw.

[0066] The position and orientation of the article 2 detected based on the image received from the camera 200 is, for example, a coordinate transformation matrix that transforms the camera coordinate system into the work coordinate system. C H W The coordinate transformation matrix is ​​given by C H W represents the position of the origin Ow of the work coordinate system in the camera coordinate system and the base vectors Xw, Yw, and Zw.

[0067] The camera 200 is fixed to the flange plate 303 via a support member 304. Therefore, the relative positional relationship between the camera 200 and the flange plate 303 is constant. This relative positional relationship can be expressed, for example, by a coordinate transformation matrix F H C The coordinate transformation matrix is ​​given by F H Crepresents the position of the origin Oc of the camera coordinate system in the flange coordinate system and the base vectors Xc, Yc, and Zc. F H C is expressed as a fixed value and is obtained by a calibration performed in advance. For the calibration, for example, a known hand-eye calibration may be adopted in which an image of a marker installed at a fixed position is captured by the imaging unit 22 of the camera 200 while the robot 300 is operating.

[0068] The movement command output from the image processing device 100 to the robot control device 400 is, for example, a coordinate transformation matrix for transforming the robot base coordinate system into the flange coordinate system. B H F The coordinate transformation matrix is ​​given by B H F represents the origin Of and the base vectors Xf, Yf, and Zf of the flange coordinate system in the robot base coordinate system.

[0069] The position and orientation of the camera 200 in the robot base coordinate system can be calculated using, for example, a coordinate transformation matrix that transforms the robot base coordinate system into the camera coordinate system: B H C The coordinate transformation matrix is ​​given by B H C teeth, B H C = B H F · F H C As mentioned above, therefore, the coordinate transformation matrix F H C is expressed as a fixed value. Therefore, the CPU 101 can calculate the movement command to be output to the robot control device 400 from the position and orientation of the camera 200 at the movement destination.

[0070] The position and orientation of the article 2 in the robot base coordinate system can be calculated using, for example, a coordinate transformation matrix that transforms the robot base coordinate system into the work coordinate system. B H W The coordinate transformation matrix is ​​given by B H W teeth, B H W = B H F · F H C · C H W As mentioned above, the coordinate transformation matrix C H W indicates the position and orientation of the article 2 in the camera coordinate system, and is detected based on the image received from the camera 200. Therefore, the CPU 101 can calculate the position and orientation of the article 2 in the robot base coordinate system using the movement command output to the robot control device 400 and the measurement results based on the image.

[0071] <Example of functional configuration of image processing device> Fig. 5 is a block diagram showing an example of the functional configuration of the image processing device shown in Fig. 1. As shown in Fig. 5, the image processing device 100 includes a storage unit 10, an identification unit 11, a search unit 12, a determination unit 13, a shape data acquisition unit 14, an image data acquisition unit 15, and a measurement unit 16. The storage unit 10 is realized by the main memory 102 and the hard disk 103 shown in Fig. 3. The identification unit 11, the search unit 12, the determination unit 13, the shape data acquisition unit 14, the image data acquisition unit 15, and the measurement unit 16 are realized by the CPU 101 shown in Fig. 3 executing a program stored in the hard disk 103.

[0072] (Storage part) The storage unit 10 stores calibration data 10a, a plurality of template data 10b, container shape data 10c, container position data 10d, at least one gripping point data 10e, and home position data 10f.

[0073] The calibration data 10a is data that defines the relative positional relationship between the camera 200 and the flange plate 303. The calibration data 10a is generated in advance by, for example, known hand-eye calibration, and is stored in the storage unit 10.

[0074] The plurality of template data 10b indicate the coordinates of points on the surface of a model having the same shape as the article 2 when the model is viewed from a plurality of virtual viewpoints in a virtual space. The plurality of template data 10b are created in advance based on data indicating the three-dimensional shape of the article 2 and stored in the storage unit 10.

[0075] The container shape data 10c indicates the three-dimensional shape of the container 5. The container shape data 10c is acquired by the shape data acquisition unit 14 and stored in the storage unit 10.

[0076] Fig. 6 is a diagram showing an example of a container. As shown in Fig. 6, the container 5 is a rectangular box in plan view, and has a rectangular frame-like top surface 501. The container shape data 10c indicates the width W, depth D, height H, sidewall thickness T, depth H2, etc. of the container 5.

[0077] One or more marks 502 may be formed on the top surface 501. In the example shown in Fig. 6, the marks 502 are formed at two positions symmetrical with respect to the center of the container 5. When one or more marks 502 are formed on the top surface 501, the container shape data 10c may further indicate the relative relationship between the positions of the corners 504 to 507 of the top surface 501 and the equation of a straight line including the position of the mark 502 and an edge 508 near the mark 502.

[0078] The container position data 10d indicates the position and posture of the container 5. Specifically, the container position data 10d indicates, for example, the range occupied by the container 5 in the space of the robot base coordinate system. The container position data 10d is generated by the identification unit 11 and stored in the storage unit 10.

[0079] The gripping point data 10e indicates the relative position and orientation of the end effector 305 with respect to the item 2 when picking up the item 2. The gripping point data 10e is created in advance using, for example, a model of the item 2 and a model of the end effector 305, and is stored in the storage unit 10.

[0080] The home position data 10f indicates the home position of the flange plate 303. The home position is expressed, for example, in a robot base coordinate system. The home position is, for example, above the center of the container 5 when the container 5 is placed in an ideal placement location, and is a position within the container 5 within a range R in which the position and orientation can be measured by imaging in the first mode. The home position data 10f is created in advance according to the ideal placement location of the container 5 and stored in the storage unit 10.

[0081] (shape data acquisition section) The shape data acquisition unit 14 acquires the container shape data 10c according to, for example, one of the following acquisition methods a to e.

[0082] [Acquisition method a] The shape data acquisition unit 14 displays a screen on the display 160 prompting the user to input the shape of the container 5 (e.g., width W, depth D, height H, side wall thickness T, depth H2, etc.), and generates container shape data 10c in accordance with the input to the input device 150.

[0083] [Acquisition Method b] The shape data acquisition unit 14 acquires CAD data indicating the three-dimensional shape of the container 5 from an external device, analyzes the CAD data, and generates the container shape data 10c.

[0084] [Acquisition Method c] An RFID tag that stores container shape data 10c is attached to the container 5. The shape data acquisition unit 14 acquires the container shape data 10c from the RFID tag via an RFID reader.

[0085] [Acquisition method d] A barcode associated with container shape data 10c is printed on the container 5. The shape data acquisition unit 14 stores the barcode and container shape data 10c in association with each other for each type of container 5. The shape data acquisition unit 14 acquires the container shape data 10c corresponding to the barcode read by the barcode reader.

[0086] [Acquisition method e] The position of camera 200 is manually adjusted so that the outer surface of the side wall of container 5 is included in the field of view. The height H of container 5 is acquired by 3D measurement using the image obtained by imaging in the first mode. The position of camera 200 is also manually adjusted so that the inner surface of the side wall of container 5 is included in the field of view. The depth H2 of container 5 is acquired by 3D measurement using the image obtained by imaging in the first mode. The position of camera 200 is also manually adjusted so that the top surface 501 of container 5 is included in the field of view. The width W, depth D, and side wall thickness T of container 5 are acquired by 2D images obtained by imaging in the second mode.

[0087] (Image data acquisition section) The image data acquisition unit 15 outputs an image capture instruction to the camera 200 and acquires an image from the camera 200. When identifying the position and orientation of the container 5, the image data acquisition unit 15 outputs an image capture instruction to capture an image in the second mode. When measuring the position and orientation of the item 2, the image data acquisition unit 15 outputs an image capture instruction to capture an image in the first mode.

[0088] (Specific part) The identification unit 11 identifies the position and orientation of the container 5 using the 2D image obtained by imaging in the second mode and the container shape data 10c. As will be described later, it is preferable that the 2D image be obtained by imaging when the optical axis of the imaging unit 22 of the camera 200 is parallel to the vertical direction (in other words, the optical axis of the imaging unit 22 is perpendicular to the top surface 501 of the container 5).

[0089] 7 is a diagram illustrating an example of a method for identifying the position and posture of a container. The identification unit 11 identifies a virtual plane 90 including the top surface 501 of the container 5 based on the height H indicated by the container shape data 10c. The virtual plane 90 is expressed in a robot base coordinate system. The identification unit 11 stores in advance the position of the surface on which the container 5 is placed (such as the top surface of a desk or the floor), and identifies the virtual plane 90 based on the position of that surface and the height H of the container 5.

[0090] The determination unit 11 calculates the coordinates of points on the virtual plane 90 that are captured in the upper right pixel 81, the lower right pixel 82, the lower left pixel 83, and the upper left pixel 84 of the 2D image 80, based on the position and orientation of the camera 200 and the viewing angle θ of the camera 200. The determination unit 11 calculates the coordinates of points on the virtual plane 90 that are captured in the upper right pixel 81, the lower right pixel 82, the lower left pixel 83, and the upper left pixel 84 of the 2D image 80, based on the position and orientation of the flange plate 303 when the 2D image 80 is captured and the coordinate transformation matrix indicated by the calibration data 10a. F H C The position and orientation of the camera 200 can be calculated using the above equation. The identification unit 11 stores a viewing angle θ that is set in advance according to the specifications of the camera 200. The coordinates of the points on the virtual surface 90 that are captured in the upper right pixel 81, the lower right pixel 82, the lower left pixel 83, and the upper left pixel 84 are expressed in, for example, a robot base coordinate system.

[0091] The identification unit 11 identifies a plurality of pixels in the 2D image 80 in which a plurality of characteristic parts of the container 5 are captured. For example, the identification unit 11 identifies pixels 85 to 88 in the 2D image 80 that depict corners (characteristic portions) of the outer contour of the container 5. Note that, because the 2D image 80 may be blurred, the identification unit 11 may identify the pixels 85 to 88 after performing known image processing such as contour enhancement processing. The pixels 85 to 88 depict one of the corners 504 to 507 (see FIG. 6 ) of the top surface 501 of the container 5. In other words, the pixels 85 to 88 depict one of the corners 504 to 507 of the container 5 located on the virtual surface 90. Therefore, for each of the pixels 85 to 88, the identification unit 11 identifies the coordinates of the point on the virtual surface 90 that is depicted in that pixel based on its positional relationship with the upper right pixel 81, the lower right pixel 82, the lower left pixel 83, and the upper left pixel 84 in the 2D image 80. The coordinates are expressed, for example, in the robot base coordinate system. The coordinates of the identified four points indicate the positions of the four corners of the top surface 501 of the container 5.

[0092] The identification unit 11 generates container position data 10d indicating the range occupied by the container 5 in the space of the robot base coordinate system based on the identified four points and the container shape data 10c, and stores the generated container position data 10d in the storage unit 10. In this way, the identification unit 11 identifies the position and posture of the container 5.

[0093] When the container shape data 10c indicates the width W and depth D of the container 5, the identification unit 11 simply identifies the position and orientation of the container 5 based on at least two pixels in the 2D image 80 that capture the corners of the outer contour of the container 5.

[0094] For example, the position and posture of the container 5 is identified as follows based on pixels 85 and 86. The identification unit 11 calculates the distance between two points on the virtual surface 90 that are reflected in pixels 85 and 86, respectively, based on the coordinates of the two points (robot base coordinate system).

[0095] If the calculated distance is within W±the measurement error range, the identification unit 11 determines that the line connecting the two points on the virtual surface 90 that are captured by pixels 85 and 86 is parallel to the width direction of the container 5. The identification unit 11 identifies a line on the virtual surface 90 that passes through one of the two points and is perpendicular to the line connecting the two points. The identification unit 11 determines a point on the virtual surface 90 that is on the identified line and is a distance to the left of the one point equal to the depth D, as one corner of the top surface 501 of the container 5. Furthermore, the identification unit 11 identifies a line on the virtual surface 90 that passes through the other of the two points and is perpendicular to the line connecting the two points. The identification unit 11 determines a point on the virtual surface 90 that is on the identified line and is a distance to the left of the other point equal to the width W, as one corner of the top surface 501 of the container 5.

[0096] In this way, the identification unit 11 can identify the position and orientation of the container 5 based on at least two pixels in the 2D image 80 that show corners of the outer contour of the container 5. Therefore, by using a 2D image that shows at least two of the corners 504 to 507 (see FIG. 6) of the top surface 501 of the container 5, the identification unit 11 can successfully identify the position and orientation of the container 5.

[0097] 8 is a diagram illustrating another example of a method for identifying the position and orientation of a container. The identification unit 11 identifies a pixel group 89a in which a mark 502 (characteristic portion) formed on the container 5 appears in the 2D image 80. The mark 502 is formed on the top surface 501 of the container 5 and is therefore located on a virtual plane 90. Therefore, the identification unit 11 can identify the center coordinates (robot base coordinate system) of the mark 502 appearing in the pixel group 89a from the positional relationship of the pixel group 89a with respect to the upper right pixel 81, the lower right pixel 82, the lower left pixel 83, and the upper left pixel 84 in the 2D image 80.

[0098] Furthermore, the identification unit 11 identifies a pixel group 89b in the 2D image 80 that is near the pixel group 89a and that captures an edge line (characteristic portion) that forms the boundary between the container 5 and the background. Referring to FIG. 6, an edge 508 near the mark 502 is captured in the pixel group 89b. The edge 508 is a line at the edge of the top surface 501 of the container 5, and is therefore located on the virtual surface 90. Therefore, the identification unit 11 can identify the equation (an equation expressed in the robot base coordinate system) of the straight line that includes the edge 508 captured in the pixel group 89a, from the positional relationship of the pixel group 89b with respect to the upper right pixel 81, the lower right pixel 82, the lower left pixel 83, and the upper left pixel 84 in the 2D image 80.

[0099] The identification unit 11 refers to the container shape data 10c and reads out the correlation between the positions of the corners 504 to 507 of the top surface 501 and the position of the mark 502 and the equation of a line including an edge 508 near the mark 502. The identification unit 11 applies the equation of a line including the identified position of the mark 502 and the edge 508 to the read-out correlation, thereby identifying the coordinates of the corners 504 to 507 of the container 5. The coordinates are expressed, for example, in a robot base coordinate system.

[0100] The identification unit 11 generates container position data 10d indicating the range occupied by the container 5 in the space of the robot base coordinate system based on the coordinates of the identified four corners 504 to 507 and the container shape data 10c, and stores the generated container position data 10d in the storage unit 10. In this way, the identification unit 11 identifies the position and posture of the container 5.

[0101] In this way, the position and orientation of the container 5 is identified based on the mark 502 of the container 5 and the edges 508 near the mark 502, which appear in the 2D image 80. Therefore, by using the 2D image in which the mark 502 of the container 5 and the edges 508 near it appear, the identification unit 11 can successfully identify the position and orientation of the container 5.

[0102] (Exploration Department) The search unit 12 searches for the position and orientation of the camera 200 when the identification unit 11 has succeeded in identifying the camera 200.

[0103] FIG. 9 is a diagram illustrating an example of a search process for the camera position. The search unit 12 starts a search in response to a failure to identify the position and orientation of the container 5 based on the 2D image obtained from the camera 200 when the flange plate 303 of the robot 300 is located at the home position. The home position is indicated by the home position data 10f. The home position is determined in advance so that the optical axis of the imaging unit 22 of the camera 200 is vertical. Hereinafter, the position of the camera 200 when the flange plate 303 is located at the home position will be referred to as the reference position P0.

[0104] As shown in the lower left of FIG. 9 , the search unit 12 moves the camera 200 upward from the reference position P0 to search for the position and orientation of the camera 200 when identification by the identification unit 11 is successful. Specifically, after moving the camera 200 upward by a predetermined distance, the search unit 12 outputs an image capture instruction in the second mode to the camera 200 and executes a first control process to check whether or not identification of the position and orientation of the container 5 has been successful based on the 2D image received from the camera 200. If identification of the position and orientation of the container 5 is successful, the search unit 12 ends the search. If identification of the position and orientation of the container 5 has failed, the search unit 12 repeats the first control process until identification of the position and orientation of the container 5 is successful.

[0105] 9, the search unit 12 may search for the position and orientation of the camera 200 when the identification by the identification unit 11 is successful by moving the camera 200 in the horizontal direction from the reference position P0. Specifically, the search unit 12 moves the camera 200 horizontally a predetermined distance, and then outputs an image capture instruction in the second mode to the camera 200, and executes a second control process to check whether the position and orientation of the container 5 have been successfully identified based on the 2D image received from the camera 200. If the position and orientation of the container 5 have been successfully identified, the search unit 12 ends the search. If the position and orientation of the container 5 have not been successfully identified, the search unit 12 repeats the second control process until the position and orientation of the container 5 have been successfully identified.

[0106] If the position and orientation of the container 5 are not successfully identified even when the position of the camera 200 reaches its limit, the search unit 12 issues an error notification.

[0107] (Measurement section) The measurement unit 16 measures the position and orientation of the article 2 by performing a three-dimensional measurement process that measures the three-dimensional shape of the subject based on the image acquired from the camera 200 and a detection process that detects the shape of the article 2 from the measured three-dimensional shape. As shown in Fig. 5, the measurement unit 16 includes a point cloud data generation unit 18 that performs the three-dimensional measurement process and a work detection unit 19 that performs the detection process.

[0108] Point cloud data generation unit 18 performs three-dimensional measurement of the field of view of camera 200 based on the image, and generates three-dimensional point cloud data. The three-dimensional point cloud data indicates the three-dimensional coordinates of each point on the object surface (measurement surface) present in the field of view of camera 200. In this embodiment, point cloud data generation unit 18 uses, for example, a structured illumination method as the three-dimensional measurement process.

[0109] The workpiece detection unit 19 detects the position and orientation of the item 2 by a three-dimensional search. Specifically, the workpiece detection unit 19 compares multiple template data 10b stored in the memory unit 10 with the three-dimensional point cloud data, and searches for data similar to the templates in the three-dimensional point cloud data. The searched data corresponds to data of the portion where the item 2 is located. The workpiece detection unit 19 detects the position and orientation of the item 2 based on the data similar to the templates searched for in the three-dimensional point cloud data. The position and orientation of the item 2 detected by the workpiece detection unit 19 is indicated in the camera coordinate system.

[0110] The workpiece detection unit 19 may use a known detection algorithm to detect the position and orientation of the item 2. Specifically, the workpiece detection unit 19 calculates a correlation value between the three-dimensional point cloud data and the template data 10b, and determines that the item 2 is present if the calculated correlation value is equal to or greater than a predetermined threshold. Then, the workpiece detection unit 19 detects the position and orientation of the item 2 according to the template data 10b with the highest correlation value.

[0111] (Decision section) The determination unit 13 determines one of the one or more articles 2 whose position and orientation have been identified by the measurement unit 16 as the target object, and determines the movement path of the end effector 305 for picking up the target object.

[0112] The determination unit 13 selects one item 2 from one or more items 2 whose positions and orientations have been identified by the measurement unit 16. The determination unit 13 calculates the position and orientation of the end effector 305 when gripping the item 2, based on the position and orientation of the selected item 2 and the gripping point data 10e. The determination unit 13 determines whether the end effector 305 with the calculated position and orientation and the articulated arm 302 supporting the end effector 305 will interfere with the container 5, based on the container position data 10d. At this time, the determination unit 13 may also determine whether the end effector 305 and the articulated arm 302 will interfere with another item 2. If interference occurs, the determination unit 13 selects another item 2 from the one or more items 2 whose positions and orientations have been identified by the measurement unit 16. If no interference occurs, the determination unit 13 determines the selected item 2 as the object to be picked.

[0113] The determination unit 13 determines a movement path from the current position and posture of the end effector 305 to the position and posture of the end effector 305 when gripping the target object. At this time, the determination unit 13 determines the movement path based on the container position data 10d so as to avoid interference between the container 5 and the end effector 305.

[0114] The determination unit 13 calculates the position and posture of the flange plate 303 for each control cycle for moving the end effector 305 along the movement path, and outputs a movement command to the calculated position and posture to the robot control device 400. As a result, the robot 300 moves the end effector 305 to the target object. As described above, the movement path is determined so as to avoid interference between the container 5 and the end effector 305, and therefore, the end effector 305 is prevented from colliding with the container 5 during movement.

[0115] <Processing flow of image processing device> 10 is a flowchart showing an example of the processing flow of the image processing device. First, the CPU 101 of the image processing device 100 outputs a command to move to the home position to the robot control device 400, and then outputs an instruction to capture images in the second mode to the camera 200 (step S1). As a result, the camera 200 captures images in the second mode at the reference position P0, and outputs a 2D image obtained by the capture to the image processing device 100.

[0116] The CPU 101 determines whether the container 5 is shown in the 2D image to the extent that the position and orientation of the container 5 can be identified (step S2). For example, the CPU 101 determines YES in step S2 if it can identify pixels in the 2D image that show at least two of the corners 504 to 507 of the top surface 501 of the container 5. Alternatively, the CPU 101 determines YES in step S2 if it can identify the mark 502 and its nearby edge line from the 2D image.

[0117] If the result of step S2 is NO, the CPU 101 determines whether the camera 200 has been moved to its upper limit (step S3). If the result of step S3 is NO, the CPU 101 executes a first movement control to generate a movement command for moving the camera 200 upward by a predetermined distance and output the generated movement command to the robot control device 400 (step S4).

[0118] If the answer is YES in step S3, the CPU 101 determines whether the camera 200 has been moved to its lateral limit (step S5). If the answer is NO in step S5, the CPU 101 generates a movement command to move the camera 200 a predetermined distance in the lateral direction (any direction parallel to the horizontal plane) and executes second movement control to output the generated movement command to the robot control device 400 (step S6). That is, the CPU 101 executes the second movement control in response to failure to successfully identify the position and posture of the container 5 in the first movement control. Note that in the first iteration of step S6, the CPU 101 first moves the camera 200 to the reference position P0 and then executes the second movement control.

[0119] After step S4 or step S6, CPU 101 outputs an instruction to capture images in the second mode to camera 200 (step S7). As a result, camera 200 captures images in the second mode and outputs a 2D image obtained by the capture to image processing device 100. After step S7, CPU 101 returns the process to step S2.

[0120] If the answer is YES in step S5, CPU 101 issues an error notification (step S8). After step S8, CPU 101 ends the process.

[0121] If the answer is YES in step S2, the CPU 101 identifies the position and orientation of the container 5 based on the 2D image and the container shape data 10c (step S9).

[0122] Next, the CPU 101 determines the internal space of the container 5 as the search range for the items to be picked (step S10).

[0123] Next, the CPU 101 outputs a command to move to the home position to the robot control device 400, and then outputs an image capturing command in the first mode to the camera 200. Based on the image received from the camera 200, the CPU 101 performs 3D measurement of the search range for the picking target, and detects the position and orientation of one or more items 2 present in the container 5 (step S11).

[0124] Next, based on the 3D measurement result, the CPU 101 determines a movement path of the end effector 305 for picking up a target item from among the one or more items 2 in the container 5 (step S12). In step S12, the CPU 101 determines a movement path based on the position and orientation of the container 5 identified in step S9 so that the container 5 and the end effector 305 do not interfere with each other.

[0125] Next, the CPU 101 outputs a picking instruction along the determined movement path to the robot control device 400 (step S13).

[0126] Next, the CPU 101 determines whether or not there are any remaining articles 2 in the container 5 (step S14). For example, if the positions and orientations of two or more articles 2 were detected in the previous step S11, the CPU 101 determines YES in step S14.

[0127] If the result of step S14 is YES, CPU 101 returns the process to step S11. If the result of step S14 is NO, CPU 101 ends the process.

[0128] <Modification> (Variation 1) Fig. 11 is a block diagram showing the functional configuration of an image processing device according to Modification 1. As shown in Fig. 11, image processing device 100A according to Modification 1 differs from image processing device 100 shown in Fig. 5 in that it includes adjustment unit 17. Adjustment unit 17 is realized by CPU 101 shown in Fig. 3 executing a program stored in hard disk 103.

[0129] The adjustment unit 17 adjusts the home position in accordance with the position and posture of the container 5 identified by the identification unit 11. For example, the adjustment unit 17 determines a point at a predetermined height above the center of the container 5 in the identified position and posture as the updated home position, and updates the home position data 10f stored in the storage unit 10. As a result, the home position is appropriately adjusted in accordance with the position of the container 5, and the reference position P0 is also adjusted.

[0130] (Variation 2) Fig. 12 is a block diagram showing the functional configuration of an image processing device according to Modification 2. As shown in Fig. 12, image processing device 100B according to Modification 2 differs from image processing device 100 shown in Fig. 5 in that it includes a misalignment detection unit 20. The misalignment detection unit 20 is realized by CPU 101 shown in Fig. 3 executing a program stored in hard disk 103.

[0131] The positional deviation detection unit 20 detects a positional deviation of the container 5 based on the 3D measurement results obtained by the measurement unit 16. The position and orientation of the container 5 may change due to some cause after being identified by the identification unit 11. For example, the position and orientation of the container 5 may change due to an unintended external force. The positional deviation detection unit 20 detects such a change in the position and orientation of the container 5.

[0132] Fig. 13 is a diagram illustrating the processing of the positional deviation detection unit, which shows a first 3D measurement result 60 measured when picking up object A and a second 3D measurement result 61 measured when picking up the next object B.

[0133] If there is no change in the position and orientation of the container 5, the first 3D measurement result 60 and the second 3D measurement result 61 differ only in that the first 3D measurement result 60 contains the object A, while the second 3D measurement result 60 does not contain the object A. On the other hand, if there is a change in the position and orientation of the container 5 after the first 3D measurement result 60 is obtained, a difference will occur between the first 3D measurement result 60 and the second 3D measurement result 61 depending on the change in the position and orientation of the container 5. Therefore, the positional deviation detection unit 20 calculates the difference between the first 3D measurement result 60 and the second 3D measurement result 61 and compares this difference with a predetermined threshold value. The threshold value is a value greater than the difference between 3D measurement results corresponding to the presence or absence of one item 2. In this way, the positional deviation detection unit 20 can detect a change in the position and orientation of the container 5 when the difference between the first 3D measurement result 60 and the second 3D measurement result 61 exceeds the threshold value.

[0134] The process of detecting a change in the position and orientation of the container 5 is executed in step S11 in the flowchart shown in Fig. 10. That is, the CPU 101 determines whether or not there is a change in the position and orientation of the container 5 by comparing the difference between the 3D measurement result obtained in the previous step S11 and the 3D measurement result obtained in the current step S11 with a threshold. If a change in the position and orientation of the container 5 is detected, the CPU 101 returns the process to step S1 and identifies the position and orientation of the container 5 again in accordance with steps S1 to S9. That is, the identification unit 11 and the search unit 12 start operating in response to the detection of a change in the position and orientation of the container by the position deviation detection unit 20.

[0135] (Variation 3) In the flowchart shown in Fig. 10, steps S5 and S6 may be omitted, or steps S3 and S4 may be omitted.

[0136] (Variation 4) In the above description, the camera 200 is mounted on the robot 300 so that its relative position with respect to the end effector 305 is constant. However, the camera 200 may be mounted on the robot 300 so that it is movable. For example, the robot 300 may be a dual-arm robot, with the end effector 305 attached to one arm and the camera 200 mounted on the other arm.

[0137] (Variation 5) Camera 200 is not limited to the configuration shown in FIG. 2 . For example, camera 200 may be a twin-lens stereo camera. In this case, when camera 200 receives an instruction to capture an image in the first mode, it captures an image according to a stereo system. Camera 200 may also be a twin-lens stereo camera with an attached projector, as disclosed in Japanese Patent Application Laid-Open No. 2019-138822, for example. In this way, camera 200 only needs to have a first mode in which it operates as a 3D camera and a second mode in which it operates as a 2D camera, and various systems may be adopted as the system for operating as a 3D camera.

[0138] §3 Supplementary Note As described above, the present embodiment includes the following disclosure.

[0139] (Configuration 1) An image processing device (100, 100A, 100B) that processes an image obtained by a camera (200) having a first mode in which it operates as a 3D camera and a second mode in which it operates as a 2D camera, wherein the camera (200) is movably mounted on a robot (300) having an end effector (305) that picks up an object (2) in a container (5), and the image processing device (100, 100A, 100B) an identification unit (11, 101) that identifies the position and orientation of the container (5) based on a 2D image obtained by imaging with the camera (200) operating in the second mode; a search unit (12, 101) for searching the position and orientation of the camera (200) when the identification by the identification unit (11, 101) is successful; a determination unit (13, 101) that determines a movement path of the end effector (305) for picking up the object (2) based on the position and orientation of the object (2) detected using a 3D measurement result obtained by imaging the object (2) by the camera (200) operating in the first mode, The determination unit (13, 101) determines the movement path so as to avoid interference between the container (5) and the end effector (305) based on the position and posture of the container (5) identified by the identification unit (11, 101).

[0140] (Configuration 2) The image processing device (100, 100A, 100B) according to configuration 1, wherein the search unit (12, 101) starts the search in response to a failure to identify the position and orientation of the container (5) based on the 2D image when the camera (200) is located at a reference position.

[0141] (Configuration 3) 3. The image processing device (100, 100A, 100B) according to configuration 2, wherein the search unit (12, 101) moves the camera (200) upward from the reference position.

[0142] (Configuration 4) 3. The image processing device (100, 100A, 100B) according to configuration 2, wherein the search unit (12, 101) moves the camera (200) in a horizontal direction from the reference position.

[0143] (Configuration 5) The search unit (12, 101) executing a first movement control to move the camera upward from the reference position; The image processing device (100, 100A, 100B) according to configuration 2 executes a second movement control to move the camera (200) horizontally from the reference position in response to failure to successfully identify the position and orientation of the container in the first movement control.

[0144] (Configuration 6) An image processing device (100, 100A, 100B) according to any one of configurations 2 to 5, wherein the reference position is the position of the camera (200) when the object is included in the depth of field of the camera (200) operating in the first mode.

[0145] (Configuration 7) The search unit (12, 101) moves the camera (200) until a characteristic part of the container (5) appears in the 2D image, The identification unit (11, 101) Identifying the location of the feature based on the 2D image; The image processing device (100, 100A, 100B) according to any one of configurations 1 to 6, which identifies the position and orientation of the container (5) using shape data indicating the position of the characteristic part and the shape of the container (5).

[0146] (Configuration 8) The container (5) contains a plurality of articles (2) including the target object, the 3D measurement results indicate the positions and orientations of the plurality of items (2); The image processing device (100B) further The image processing device (100B) according to any one of configurations 1 to 7, further comprising a detection unit (20, 101) that detects a change in the position and orientation of the container (5) when a difference between a first 3D measurement result (60) and a second 3D measurement result (61) obtained after the first 3D measurement result (60) exceeds a threshold.

[0147] (Configuration 9) 9. The image processing device according to configuration 8, wherein the identification unit (11, 101) and the search unit (12, 101) start operating in response to detection of a change in the position and orientation of the container (5) by the detection unit (20, 101).

[0148] (Configuration 10) The image processing device (100A) according to any one of configurations 1 to 9, further comprising an adjustment unit (17) that adjusts the home position of the end effector in accordance with the position and posture of the container (5) identified by the identification unit (11, 101).

[0149] (Configuration 11) A robot control system (1), a robot (300) having an end effector (305) for picking up an object in a container (5); a camera (200) movably mounted on the robot (300) and having a first mode in which it operates as a 3D camera and a second mode in which it operates as a 2D camera; an identification unit (11, 101) that identifies the position and orientation of the container (5) based on a 2D image obtained by imaging with the camera (200) operating in the second mode; a search unit (12, 101) for searching the position and orientation of the camera (200) when the identification by the identification unit (11, 101) is successful; a determination unit (13, 101) that determines a movement path of the end effector (305) for picking up the object based on the position and orientation of the object detected using a 3D measurement result obtained by imaging the object with the camera (200) operating in the first mode, The determination unit (13, 101) determines the movement path based on the position and posture of the container (5) identified by the identification unit (11, 101) so as to avoid interference between the container (5) and the end effector (305), in a robot control system (1).

[0150] (Configuration 12) A control method for an image processing device (100, 100A, 100B) that processes an image obtained by a camera (200) having a first mode in which it operates as a 3D camera and a second mode in which it operates as a 2D camera, wherein the camera (200) is movably mounted on a robot (300) having an end effector (305) that picks up an object in a container (5), the control method comprising: determining the position and orientation of the container (5) based on a 2D image obtained by capturing an image with the camera (200) operating in the second mode; a step of searching for the position and orientation of the camera (200) when the position and orientation of the container (5) is successfully identified; determining a movement path of the end effector (305) for picking up the object based on the position and orientation of the object detected using 3D measurement results obtained by imaging the object with the camera (200) operating in the first mode; The determining step includes a step of determining the movement path based on the identified position and orientation of the container (5) so as to avoid interference between the container (5) and the end effector (305).

[0151] Although the embodiments of the present invention have been described, the embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0152] 1 Robot control system, 2 Item, 5 Container, 10 Memory unit, 10a Calibration data, 10b Template data, 10c Container shape data, 10d Container position data, 10e Grasping point data, 10f Home position data, 11 Identification unit, 12 Search unit, 13 Determination unit, 14 Shape data acquisition unit, 15 Image data acquisition unit, 16 Measurement unit, 17 Adjustment unit, 18 Point cloud data generation unit, 19 Work detection unit, 20 Position deviation detection unit, 21 Projection unit, 22 Imaging unit, 60, 61 3D measurement result, 80 2D image, 81 Upper right pixel, 82 Lower right pixel, 83 Lower left pixel, 84 Upper left pixel, 85 to 88 Pixel, 89a, 89b Pixel group, 90 Virtual surface, 100, 100A, 100B Image processing device, 101 CPU, 102 Main memory, 103 hard disk, 104 camera interface, 104a image buffer, 105 input interface, 106 display controller, 107 communication interface, 108 data reader / writer, 109 bus, 150 input device, 160 display, 200 camera, 210 light source, 211 filter, 212, 221 optical system, 220 image sensor, 300 robot, 301 base, 302 articulated arm, 303 flange plate, 304 support member, 305 end effector, 400 robot control device, 501 top surface, 502 mark, 504 to 507 corners, 508 edge, 700 memory card, P0 reference position.

Claims

1. An image processing device that processes an image obtained by a camera having a first mode in which it operates as a 3D camera and a second mode in which it operates as a 2D camera, the camera being movably mounted on a robot having an end effector that picks up an object in a container, the image processing device comprising: an identification unit that identifies the position and orientation of the container based on a 2D image obtained by imaging with the camera operating in the second mode; a search unit that searches for the position and orientation of the camera when the identification by the identification unit is successful; a determination unit that determines a movement path of the end effector for picking up the object based on a position and orientation of the object detected using a 3D measurement result obtained by imaging the object with the camera operating in the first mode, the determination unit determines the movement path based on the position and posture of the container identified by the identification unit so as to avoid interference between the container and the end effector; The search unit starts the search in response to failure to identify the position and orientation of the container based on the 2D image when the camera is located at a reference position.

2. The image processing device according to claim 1 , wherein the search unit moves the camera upward from the reference position.

3. The image processing device according to claim 1 , wherein the search unit moves the camera in a horizontal direction from the reference position.

4. The search unit executing a first movement control to move the camera upward from the reference position; The image processing device according to claim 1 , further comprising: a second movement control for moving the camera in a horizontal direction from the reference position when the first movement control fails to identify the position and orientation of the container.

5. The image processing device according to claim 1 , wherein the reference position is a position of the camera when the object is included in the depth of field of the camera operating in the first mode.

6. An image processing device that processes images obtained by a camera having a first mode that operates as a 3D camera and a second mode that operates as a 2D camera, the camera being mounted on a robot that has a movable end effector that picks up an object in a container, the image processing device comprising: an identification unit that identifies the position and orientation of the container based on a 2D image obtained by imaging with the camera operating in the second mode; a search unit that searches for the position and orientation of the camera when the identification by the identification unit is successful; a determination unit that determines a movement path of the end effector for picking up the object based on a position and orientation of the object detected using a 3D measurement result obtained by imaging the object with the camera operating in the first mode, the determination unit determines the movement path based on the position and posture of the container identified by the identification unit so as to avoid interference between the container and the end effector; The container contains a plurality of articles including the target object, the 3D measurement results indicate the positions and orientations of the plurality of articles; The image processing device further comprises: an image processing device comprising a detection unit that detects a change in the position and orientation of the container when a difference between a first 3D measurement result and a second 3D measurement result obtained after the first 3D measurement result exceeds a threshold;

7. An image processing device that processes images obtained by a camera having a first mode that operates as a 3D camera and a second mode that operates as a 2D camera, the camera being mounted on a robot that has a movable end effector that picks up an object in a container, the image processing device comprising: an identification unit that identifies the position and orientation of the container based on a 2D image obtained by imaging with the camera operating in the second mode; a search unit that searches for the position and orientation of the camera when the identification by the identification unit is successful; a determination unit that determines a movement path of the end effector for picking up the object based on a position and orientation of the object detected using a 3D measurement result obtained by imaging the object with the camera operating in the first mode, the determination unit determines the movement path based on the position and posture of the container identified by the identification unit so as to avoid interference between the container and the end effector; The image processing device further includes an adjustment unit that adjusts a home position of the end effector in accordance with the position and orientation of the container identified by the identification unit.

8. The search unit moves the camera until a characteristic part of the container is captured in the 2D image, The identification unit Identifying the location of the feature based on the 2D image; The image processing device according to claim 1 , wherein the position and orientation of the container are identified using the position of the characteristic portion and shape data indicating the shape of the container.

9. The image processing device according to claim 6 , wherein the identifying unit and the searching unit start their operations in response to the detection of a change in the position and orientation of the container by the detecting unit.

10. 1. A robot control system, comprising: a robot having an end effector for picking up an object in a container; a camera movably mounted on the robot, the camera having a first mode in which it operates as a 3D camera and a second mode in which it operates as a 2D camera; A robot control system comprising: the image processing device according to any one of claims 1 to 9.

11. A control method for an image processing device that processes an image obtained by a camera having a first mode in which it operates as a 3D camera and a second mode in which it operates as a 2D camera, the camera being movably mounted on a robot having an end effector that picks up an object in a container, the control method comprising: determining the position and orientation of the container based on a 2D image captured by the camera operating in the second mode; a step of searching for the position and orientation of the camera when the position and orientation of the container is successfully identified; determining a movement path of the end effector for picking up the object based on the position and orientation of the object detected using 3D measurement results obtained by imaging with the camera operating in the first mode; the determining step includes a step of determining the movement path based on the identified position and orientation of the container so as to avoid interference between the container and the end effector; A control method, wherein the step of performing the search includes a step of starting the search in response to failure to identify the position and orientation of the container based on the 2D image when the camera is located at a reference position.

12. A control method for an image processing device that processes an image obtained by a camera having a first mode that operates as a 3D camera and a second mode that operates as a 2D camera, wherein the camera is movably mounted on a robot having an end effector that picks up an object in a container, the control method comprising: determining the position and orientation of the container based on a 2D image captured by the camera operating in the second mode; a step of searching for the position and orientation of the camera when the position and orientation of the container is successfully identified; determining a movement path of the end effector for picking up the object based on the position and orientation of the object detected using 3D measurement results obtained by imaging with the camera operating in the first mode; the determining step includes a step of determining the movement path based on the identified position and orientation of the container so as to avoid interference between the container and the end effector; The container contains a plurality of articles including the target object, the 3D measurement results indicate the positions and orientations of the plurality of articles; The control method further comprises: A control method comprising a step of detecting a change in the position and orientation of the container when a difference between a first 3D measurement result and a second 3D measurement result obtained after the first 3D measurement result exceeds a threshold.

13. A control method for an image processing device that processes an image obtained by a camera having a first mode that operates as a 3D camera and a second mode that operates as a 2D camera, wherein the camera is movably mounted on a robot having an end effector that picks up an object in a container, the control method comprising: determining the position and orientation of the container based on a 2D image captured by the camera operating in the second mode; a step of searching for the position and orientation of the camera when the position and orientation of the container is successfully identified; determining a movement path of the end effector for picking up the object based on the position and orientation of the object detected using 3D measurement results obtained by imaging with the camera operating in the first mode; the determining step includes a step of determining the movement path based on the identified position and orientation of the container so as to avoid interference between the container and the end effector; The control method further comprises: A control method comprising a step of adjusting a home position of the end effector in accordance with the position and orientation of the container identified in the identifying step.

Images