Image processing device, imaging device, robot, and robot system

Abstract

The image processing apparatus (100) includes: a storage unit (115) that stores the shape of the subject (A); an extraction unit (111a) that extracts a second image of an object region from a first image obtained by the camera (41) photographing the subject; a distance detection unit (111b) that detects the distances from at least three parts appearing in the object region to the camera by processing the second image; a plane inference unit (111c) that infers a plane appearing in the object region using the distances at the at least three parts; an angle detection unit (111d) that detects an angle of the plane with respect to the optical axis of the camera; a contour inference unit (111e) that infers the contour of the subject appearing in the first image using the shape of the subject and the angle of the plane; and a determination unit (111g) that determines the subject on the first image using the contour.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus, a photographing apparatus, a robot, and a robot system. Background Art

[0002] Conventionally, there has been a robot that controls its actions using an image of a workpiece taken. For example, Patent Document 1 discloses an apparatus that uses an image of a workpiece taken by a camera to measure the three-dimensional position and orientation of the workpiece, and uses the measured information to control the tip of a robot arm and a robot hand. The camera of Patent Document 1 includes left and right lens mechanisms and a photographing element. The workpiece is photographed by the left and right lens mechanisms simultaneously and recorded on the photographing element. Then, by performing image processing using a stereo matching method between two images of the workpiece in the left and right lens mechanisms, the three-dimensional position and orientation of the workpiece are calculated.

[0003] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2009-241247

[0004] However, Patent Document 1 does not disclose a method for identifying a workpiece in two images. The shape, size, color, pattern, etc. of the workpiece reflected in the image vary corresponding to the positional relationship between the camera and the workpiece and the orientation of the workpiece relative to the camera. Therefore, there are cases where a lot of processing is required to identify the workpiece in the image. Summary of the Invention

[0005] Accordingly, an object of the present disclosure is to provide an image processing apparatus, a photographing apparatus, a robot, and a robot system that reduce the amount of processing for identifying a subject such as a workpiece in an image.

[0006] An image processing apparatus according to one aspect of the present disclosure is an image processing apparatus that processes an image, and includes: a storage unit that stores the shape of a subject; an extraction unit that extracts a second image of a partial region, i.e., an object region, from a first image obtained by photographing the subject by a camera; a distance detection unit that detects distances from at least three portions reflected in the object region to the camera by processing the second image; a plane inference unit that infers a plane reflected in the object region using the distances at the at least three portions; an angle detection unit that detects an angle of the plane with respect to the optical axis of the camera; a contour inference unit that infers a contour of the subject reflected in the first image, i.e., a subject contour, using the shape of the subject stored in the storage unit and the angle of the plane; and a determination unit that identifies the subject in the first image using the subject contour.

[0007] In addition, an image processing apparatus according to an aspect of the present disclosure is an image processing apparatus that processes an image, and includes: a storage unit that stores the shape of a subject; an extraction unit that extracts a second image of a partial region, that is, an object region, from a first image obtained by the camera photographing the subject; a distance detection unit that detects distances from at least three parts reflected in the object region to the camera by processing the second image; a plane inference unit that infers a plane reflected in the object region using the distances at the at least three parts; an angle detection unit that detects an angle of the plane with respect to the optical axis of the camera; a direction determination unit that orients the camera such that the direction of the optical axis of the camera coincides with a direction perpendicular to the plane, that is, a first direction; a contour inference unit that infers a contour of the subject, that is, a subject contour, reflected in a third image using the shape of the subject stored in the storage unit, where the third image is an image obtained by the camera whose optical axis is oriented in the first direction photographing the subject; and a determination unit that determines the subject on the third image using the subject contour.

[0008] In addition, a photographing apparatus according to an aspect of the present disclosure includes: a camera; and the image processing apparatus according to an aspect of the present disclosure that processes an image photographed by the camera.

[0009] In addition, a robot according to an aspect of the present disclosure includes: the photographing apparatus according to an aspect of the present disclosure; a robot arm having an end effector that performs a processing operation on an article; and a control device that controls the operations of the end effector and the robot arm based on an image of the article that is the subject determined by the determination unit.

[0010] In addition, a robot system according to an aspect of the present disclosure includes: the robot according to an aspect of the present disclosure; and an operation device that operates the robot.

[0011] According to the technology of the present disclosure, it is possible to reduce the processing amount for determining a subject in an image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a diagram showing an example of the configuration of a robot system according to an embodiment.

[0013] Figure 2 is Figure 1 a side view of the robot enlarged.

[0014] Figure 3 is a cross-sectional side view showing an example of the configuration of a camera according to an embodiment.

[0015] Figure 4 It is a block diagram showing an example of the hardware structure of the robot system related to the embodiment.

[0016] Figure 5 It is a block diagram showing an example of the functional structure of the control device related to the embodiment.

[0017] Figure 6 It is a block diagram showing an example of the functional structure of the image processing unit related to the embodiment.

[0018] Figure 7 It is a diagram showing an example of the images captured by two cameras.

[0019] Figure 8 It is a diagram showing an example of the inferred plane in the camera coordinate system.

[0020] Figure 9 It is a diagram showing an example of the shape of the contour of the object inferred by the image processing unit.

[0021] Figure 10 It is a flowchart showing an example of the operation of the robot system related to the embodiment.

[0022] Figure 11 It is a block diagram showing an example of the functional structure of the image processing unit related to Modification 1.

[0023] Figure 12 It is a flowchart showing an example of the operation of the robot system related to Modification 1.

[0024] Figure 13 It is a block diagram showing an example of the functional structure of the image processing unit related to Modification 2.

[0025] Figure 14 It is a flowchart showing an example of the operation of the robot system related to Modification 2.

[0026] Figure 15 It is a block diagram showing an example of the functional structure of the image processing unit related to Modification 3.

[0027] Figure 16 It is a flowchart showing an example of the operation of the robot system related to Modification 3.

[0028] Figure 17 It is a block diagram showing an example of the functional structure of the image processing unit related to Modification 4.

[0029] Figure 18 It is a flowchart showing an example of the operation of the robot system related to Modification 4.

[0030] Figure 19 This is a block diagram showing an example of the functional structure of the image processing unit related to Modification 5. Detailed implementation manners

[0031] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Moreover, all the embodiments described below represent inclusive or specific examples. In addition, with respect to the constituent elements in the following embodiments that are not described in the independent claims representing the most general concepts, they will be described as optional constituent elements. In addition, each drawing in the accompanying drawings is a schematic diagram and is not necessarily strictly illustrated. And, in each drawing, the same reference numerals are assigned to substantially the same constituent elements, and redundant descriptions may be omitted or simplified. In addition, in this specification and the claims, "device" may refer not only to a single device but also to a system composed of multiple devices.

[0032] <Structure of the robot system>

[0033] The structure of the robot system 1 according to the embodiment will be described. Figure 1 This is a diagram showing an example of the structure of the robot system 1 according to the embodiment. As Figure 1 shown, the robot system 1 includes a robot 10, an input device 20, an output device 30, a camera 41, and a control device 100 as constituent elements. It should be noted that not all of the above constituent elements are necessary.

[0034] In the present embodiment, the robot system 1 is a system for performing an operation of moving an article A using the robot 10, and the following description will be given, but it is not limited thereto. For example, the robot system 1 may be a system in which the robot 10 performs a certain processing action on an article. And, the article A moved by the robot 10 is taken as a cardboard box, and the following description will be given, but it is not limited thereto. The article A preferably includes a flat surface on its outer surface. Such an article A may also be another object having a specified shape, or an object such as a rock that does not have a specified shape. The input device 20 is a device for operating the robot 10 and outputs a signal corresponding to the operator's input to the control device 100. The control device 100 controls the entire robot system 1, for example, outputs a control signal corresponding to the signal from the input device 20 to the robot 10 to control the operation of the robot 10.

[0035] <Structure of the robot>

[0036] The structure of the robot 10 will be described. Figure 2 This is a side view of the robot 10 of Figure 1 enlarged. As Figure 1 andFigure 2 As shown in Figure 2 , the robot 10 includes a housing portion 11, a robot arm 12, a robot hand (also referred to as an "end effector") 13, and a carrier vehicle 14. The housing portion 11, the robot arm 12, and the robot hand 13 are disposed on the carrier vehicle 14. The carrier vehicle 14 can move the robot 10 on the ground or the like, and includes wheels 14a as a traveling unit and a carrier drive device 14b (not shown) for driving the wheels 14a. However, it is not limited thereto, and other traveling units such as crawlers (also referred to as "caterpillar tracks (registered trademark)") may be provided. The carrier drive device 14b uses electricity as a power source and has a servo motor as an electric motor, but any power source may be used. Such a carrier vehicle 14 may be, for example, an AGV (Automated Guided Vehicle). In addition, the robot 10 may be fixed to the ground or the like without including the carrier vehicle 14.

[0037] The housing portion 11 is disposed on the carrier vehicle 14 and houses a control device 100, a power supply device 16 (not shown), a negative pressure generating device 17 (not shown), and the like. The power supply device 16 supplies power to each component of the robot 10 that consumes power. The power supply device 16 may include a secondary battery and supply the power of the secondary battery to each component, or may be configured to be connected to an industrial power supply or an external power supply such as a device outside the robot 10 and supply the power of the external power supply to each component. The power supply device 16 may be configured to include a secondary battery and be connected to an external power supply. The secondary battery is a storage battery capable of charging and discharging electric power, and may be a lead storage battery, a lithium ion secondary battery, a nickel metal hydride storage battery, a nickel cadmium storage battery, or the like. The power supply device 16 may also include a primary battery that can only discharge electric power.

[0038] The base of the robot arm 12 is fixedly mounted on the carrier vehicle 14, and the robot hand 13 is mounted at the front end of the robot arm 12. The robot hand 13 is configured to be able to hold an article A in the form of a cardboard box. The robot arm 12 and the robot hand 13 operate according to the control of the control device 100. For example, the article A held by the robot hand 13 is moved to another place. In the present embodiment, the robot hand 13 holds the article A by suction using negative pressure. For example, it may also be configured to hold the article A by gripping, lifting, hanging, engaging, adhesive force, and magnetic force. In addition, the robot 10 is configured as an industrial robot of a vertical articulated type as described below, but is not limited thereto.

[0039] The robot arm 12 includes: linkages 12a to 12f arranged in sequence from its base to the front end, joints JT1 to JT6 connecting the linkages 12a to 12f in sequence, and arm driving devices M1 to M6 that drive the joints JT1 to JT6 respectively to rotate them. The operations of the arm driving devices M1 to M6 are controlled by the control device 100. The arm driving devices M1 to M6 use electricity as the power source respectively and have servo motors as the electric motors for driving them, but any power source can be used. In addition, the number of joints of the robot arm 12 is not limited to 6, and can also be 7 or more, or more than one and 5 or less.

[0040] The linkage 12a is mounted on the carrier 14 via the joint JT1, and the robot hand 13 is mounted on the front end portion of the linkage 12f. The joint JT1 connects the base end portion of the carrier 14 and the linkage 12a in such a way that it can rotate about an axis perpendicular to the ground supporting the carrier 14 in the vertical direction. The joint JT2 connects the front end portion of the linkage 12a and the base end portion of the linkage 12b in such a way that it can rotate about an axis in the horizontal direction with respect to the ground. The joint JT3 connects the front end portion of the linkage 12b and the base end portion of the linkage 12c in such a way that it can rotate about an axis in the horizontal direction. The joint JT4 connects the front end portion of the linkage 12c and the base end portion of the linkage 12d in such a way that it can rotate about an axis in the longitudinal direction of the linkage 12c. The joint JT5 connects the front end portion of the linkage 12d and the base end portion of the linkage 12e in such a way that it can rotate about an axis orthogonal to the longitudinal direction of the linkage 12d. The joint JT6 connects the front end portion of the linkage 12e and the base end portion of the linkage 12f in such a way that it can rotate with torsion with respect to the linkage 12e.

[0041] The robot hand 13 has a main body portion 13a, one or more suction portions 13b, and a support portion 13c. The main body portion 13a has a plate-like shape, the linkage 12f is connected to one main surface side of the main body portion 13a, and the suction portion 13b and the support portion 13c are arranged on the main surface on the side opposite to the above main surface. The main body portion 13a rotates together with the linkage 12f about the torsion rotation axis S of the linkage 12f. The support portion 13c protrudes from the main body portion 13a having a plate-like shape in the direction of the axis S.

[0042] The adsorption part 13b is arranged in a direction perpendicular to the axis S so as to be away from the support part 13c. The adsorption part 13b has a hollow shape such as a conical cylinder, for example, and is connected to a negative pressure generating device 17 via a pipe (not shown). The adsorption part 13b is open in the direction of the axis S. The structure of the negative pressure generating device 17 is not particularly limited as long as it can generate a negative pressure in the hollow part of the adsorption part 13b, and any existing structure can be used. For example, the negative pressure generating device 17 may have a structure of an ejector that generates a negative pressure or a vacuum by feeding compressed air, or may have a structure of a vacuum pump or a pneumatic cylinder that generates a negative pressure or a vacuum by sucking air.

[0043] The robot hand 13 adsorbs the article A via the adsorption part 13b by bringing the adsorption part 13b into contact with the surface such as the side surface of the article A and generating a negative pressure in the adsorption part 13b. Thereby, the robot hand 13 can lift the article A, hold the article A, and move it. Further, the robot hand 13 can support and hold the article A by bringing the support part 13c into contact with the surface such as the bottom surface of the article A.

[0044] <Structure of the input device>

[0045] The structure of the input device 20 will be described. As Figure 1 shown, the input device 20 receives the input of instructions, information, etc. from an operator or the like, and outputs the instructions, information, etc. to the control device 100. The input device 20 is connected to the control device 100 via wired communication or wireless communication. The forms of wired communication and wireless communication can be any form. For example, the input device 20 is used for remotely operating the robot 10 and is arranged at a predetermined distance from the robot 10.

[0046] The input device 20 may also receive the input of each operation constituting the manual operation of the robot 10 by the operator and output the command value of the operation to the control device 100. The input device 20 may also receive the input of the operation content of the automatic operation of the robot 10 and output the command value of the operation content to the control device 100. The input device 20 may also receive the input of information such as the shape and / or size of the article A, which is the moving object of the robot 10, and output the information to the control device 100.

[0047] Such an input device 20 may also include devices such as a handle, a lever, a pedal, a button, a touch panel, a microphone, and a camera. It may also be that the input device 20 receives the displacement, direction, speed, and operating force of the handle or the lever, the pressing of the button, the contact with the screen of the touch panel, the contact trajectory and contact pressure, the sound signal collected by the microphone, the analysis result of images such as the gestures of the operator captured by the camera, etc., as the input instructions and information, etc.

[0048] <Structure of the output device>

[0049] The structure of the output device 30 will be described. As Figure 1 shown, the output device 30 is disposed near the input device 20. The output device 30 outputs the signal of the image acquired from the camera 41 as an image and displays it to the operator. Examples of the output device 30 are a liquid crystal display (Liquid Crystal Display) and an organic or inorganic EL display (Electro-Luminescence Display), but are not limited to these. The output device 30 may also include a speaker that generates sound. Alternatively, the output device 30 outputs an image and / or sound for operations output by the control device 100.

[0050] <Structure of the camera>

[0051] The structure of the camera 41 will be described. The camera 41 is a camera that captures digital images and is a camera that can detect the three-dimensional position such as the distance to the object in the captured image. As Figure 2 shown, the camera 41 is arranged and oriented such that the front of the robotic hand 13 is the field of view. Specifically, the camera 41 is disposed on the link 12d and is oriented along the length direction of the link 12d toward the front end of the link 12d. Such a camera 41 can capture the article A that is the moving object of the robotic hand 13.

[0052] Figure 3 is a cross-sectional side view showing an example of the structure of the camera 41 according to the embodiment. As Figure 3 shown, in the present embodiment, the camera 41 is a compound eye camera including two or more cameras. The camera 41 includes two cameras 41a and 41b disposed at different positions. The cameras 41a and 41b have the same structure and performance, but may also have different structures and performances. The cameras 41a and 41b may be integrated as Figure 3 shown, or may be separately arranged. The cameras 41a and 41b are arranged such that their fields of view overlap each other. The cameras 41a and 41b each include a lens 42 and an imaging element 43. In the camera 41a, the imaging element 43 is disposed on the optical axis center LAa of the lens 42. In the camera 41b, the imaging element 43 is disposed on the optical axis center LAb of the lens 42. The optical axis centers LAa and LAb are parallel to each other, but may not be parallel. The optical axis centers LAa and LAb are an example of the optical axis of the camera 41.

[0053] The imaging element 43 receives the light incident through the lens 42 and forms an image based on the received light. Examples of the imaging element 43 include a CMOS (Complementary Metal-Oxide Semiconductor) image sensor and a CCD (Charge Coupled Device) image sensor. The imaging element 43 includes a plurality of light-receiving elements arranged in a grid pattern in a plane. Each light-receiving element outputs a pixel value representing the intensity of the received light to the control device 100. An example of the pixel value is a luminance value. In Figure 3 two imaging elements 43 are arranged, but they may also be integrated. In other words, one imaging element may be arranged over the area of the two imaging elements 43.

[0054] A camera coordinate system is set in the camera 41. The camera coordinate system is a local coordinate system with the cameras 41a and 41b as references, and is defined by the origin O, which is the midpoint of the centers of the lenses 42 of the cameras 41a and 41b, and the mutually orthogonal X-axis, Y-axis, and Z-axis. The Z-axis is parallel to the optical axis centers LAa and LAb. The X-axis is parallel to the plane including the optical axis centers LAa and LAb. The Y-axis is perpendicular to the plane including the optical axis centers LAa and LAb. The light-receiving surface of the imaging element 43, that is, the light-receiving face, is parallel to the XY plane. The images captured by the cameras 41a and 41b are images of the subject projected onto a plane parallel to the XY plane.

[0055] <Hardware Structure of the Robot System>

[0056] The hardware structure of the robot system 1 will be described. Figure 4 is a block diagram showing an example of the hardware structure of the robot system 1 according to the embodiment. As Figure 4 shown, the robot system 1 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a memory 104, an input I / F (Interface) 21, an imaging element I / F 44, a graphics card 31, a motor drive circuit 52, a negative pressure drive circuit 62, an input device 20, cameras 41a and 41b, an output device 30, a servo motor 51, and a negative pressure generating device 17 as components. The above components are connected via a bus, wired communication, or wireless communication. It should be noted that not all of the above components are necessary.

[0057] Although not limited thereto, in the present embodiment, the CPU 101, ROM 102, RAM 103, memory 104, input I / F 21, imaging element I / F 44, and graphics card 31 constitute the control device 100. The CPU 101 is a processor and controls the overall processing and operation of the robot system 1. The ROM 102 is composed of a non-volatile semiconductor memory or the like and stores programs, data, etc. for causing the CPU 101 to control processing and operation. The RAM 103 is composed of a volatile semiconductor memory or the like and temporarily stores programs executed by the CPU 101, data during or after processing, etc. The memory 104 is composed of semiconductor memories such as volatile memories and non-volatile memories, storage devices such as hard disk drives (HDDs) and solid state drives (SSDs).

[0058] A program for causing the CPU 101 to operate is pre-held in the ROM 102 or the memory 104. The CPU 101 reads and expands the program from the ROM 102 or the memory 104 to the RAM 103. The CPU 101 executes each encoded command in the program expanded in the RAM 103.

[0059] The input I / F 21 is connected to the input device 20 and receives the input of information, instructions, etc. from the input device 20. The input I / F 21 may also include a circuit for converting input signals.

[0060] The imaging element I / F 44 controls the driving of the imaging elements 43 of the cameras 41a and 41b corresponding to the execution of the program, and acquires the images captured by the cameras 41a and 41b into the RAM 103 or the memory 104. The imaging element I / F 44 may also include a circuit for driving the imaging element 43.

[0061] The graphics card 31 processes the images captured by the cameras 41a and 41b and / or the images generated from the images corresponding to the execution of the program, and outputs them to the output device 30 for display.

[0062] The motor drive circuit 52 supplies the power of the power supply device 16 to each servo motor 51 according to the instruction of the CPU 101 and controls the driving of the servo motor 51. The servo motor 51 is the servo motor of the arm drive devices M1 to M6 of the robot arm 12 and the conveyance drive device 14b of the conveyance vehicle 14. The motor drive circuit 52 controls the driving of all the servo motors 51.

[0063] In addition, each servo motor 51 includes: an electric motor, an encoder that detects the rotation angle of the rotor of the electric motor, and a current detector that detects the current value applied to the electric motor. Each servo motor 51 operates the electric motor according to an instruction output from the CPU 101 and outputs the detection value of the encoder and the current value of the electric motor to the CPU 101. The CPU 101 detects the rotation amount, rotation speed, rotational acceleration, rotational torque, etc. of the rotor of the servo motor 51 based on the detection value of the encoder and the current value fed back from each servo motor 51, and uses the detection results to control the rotation start, rotation stop, rotation speed, rotational acceleration, rotational torque, etc. of the servo motor 51. Thus, the CPU 101 can stop each servo motor 51 at an arbitrary rotation position, can rotate at an arbitrary rotation speed, and can operate with an arbitrary rotational torque. Therefore, the CPU 101 can cause the robot arm 12 and the carrier 14 to operate diversely and precisely.

[0064] The negative pressure drive circuit 62 supplies the power of the power supply device 16 to the negative pressure generating device 17 according to an instruction from the CPU 101 and controls the drive of the negative pressure generating device 17.

[0065] Each function of the control device 100 as described above can also be implemented by a computer system composed of the CPU 101, ROM 102, RAM 103, etc., can also be implemented by a dedicated hardware circuit such as an electronic circuit or an integrated circuit, or can be implemented by a combination of the above computer system and hardware circuit. The control device 100 can also execute each process by centralized control based on a single control device, or can execute each process by distributed control based on the cooperation of multiple control devices. It can also be that the program is provided as an application program through communication via a communication network such as the Internet, communication based on a mobile communication standard, other wireless networks, a wired network, or a broadcast.

[0066] For example, each function of the control device 100 can also be implemented by circuits such as LSI (Large Scale Integration), system LSI, etc. The functions of multiple components can also be individually integrated into a single chip, or integrated into a single chip in a manner that includes a part or all of them. Additionally, each circuit can be a general-purpose circuit or a dedicated circuit. As for LSI, an FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, a reconfigurable processor that can reconfigure the connections and / or settings of circuit units inside the LSI, or an ASIC (Application Specific Integrated Circuit) that integrates circuits with multiple functions suitable for specific purposes into one can be used, etc.

[0067] <Function Structure of Control Device>

[0068] The function structure of the control device 100 will be described. Figure 5 It is a block diagram showing an example of the function structure of the control device 100 according to the embodiment. As Figure 5 shown, the control device 100 includes an input processing unit 201, an output processing unit 301, a shooting control unit 401, an image processing unit 111, an arm control unit 112, a hand control unit 113, a handling control unit 114, and a storage unit 115 as functional structure elements. It should be noted that not all of the above functional structure elements are necessary. The control device 100 is an example of an image processing device.

[0069] The functions of the functional structure elements other than the storage unit 115 are implemented by the CPU 101, etc. The function of the storage unit 115 is implemented by the memory 104, ROM 102, RAM 103, etc. The shooting control unit 401, the image processing unit 111, and the cameras 41a and 41b constitute a shooting device 40.

[0070] The storage unit 115 can store various information and read the stored information. The storage unit 115 stores the camera parameters of the cameras 41a and 41b, and the shape and size of the items of the objects that the robot 10 can move, etc. Additionally, the storage unit 115 stores the information of the area processed by the image processing unit 111 in the images captured by the cameras 41a and 41b. The storage unit 115 can also store programs.

[0071] The camera parameters include external parameters and internal parameters. Examples of the external parameters are parameters indicating the positions and orientations of cameras 41a and 41b, etc. Examples of the internal parameters are parameters indicating the deformation of the lenses of cameras 41a and 41b, the focal length, the size of one pixel of the imaging element 43, and the pixel coordinates of the optical axis center, etc. The pixel coordinates are coordinates in units of pixels and are two-dimensional coordinates on the image.

[0072] The shape of the article is the three-dimensional shape and / or two-dimensional shape of the article. The shape of the article may also be stored in the storage unit 115 as information indicating the three-dimensional shape and / or two-dimensional shape of the article. The three-dimensional shape and two-dimensional shape of the article may also be stored in the storage unit 115 as a three-dimensional model and a two-dimensional model of the article, respectively. For example, the two-dimensional shape of the article may also be the contour shape of the article on the projection plane when the article is projected from various directions.

[0073] The size of the article represents the size of the actual article, i.e., the size of the physical object. The size of the article may also represent the sizes of the respective parts of the three-dimensional shape and two-dimensional shape of the article, or may represent the sizes of the characteristic parts of the three-dimensional shape and two-dimensional shape. For example, the size of the characteristic part may also be representative sizes of the outer shape of the article such as the width, height, and depth of the outer shape, and sizes of representative parts of the article such as the size of the edge of the article and the cross-sectional size.

[0074] The input processing unit 201 outputs instructions, information, etc. obtained from the input device 20 to each functional structural element of the control device 100.

[0075] The output processing unit 301 outputs output information such as the images captured by cameras 41a and 41b, the images processed by the image processing unit 111, the action results and detection results of each functional structural element of the control device 100, etc. to the output device 30. The output processing unit 301 outputs a screen for operating the robot 10 to the output device 30.

[0076] The shooting control unit 401 controls the operations of cameras 41a and 41b. The shooting control unit 401 may also cause cameras 41a and 41b to perform shooting according to an instruction to execute shooting input to the input device 20. The shooting control unit 401 causes cameras 41a and 41b to perform shooting synchronously with each other. The shooting control unit 401 makes the images captured by cameras 41a and 41b at the same time correspond to each other and outputs them to the storage unit 115, etc.

[0077] The image processing unit 111 uses the images captured by the cameras 41a and 41b to identify the articles depicted in the images and detect the three-dimensional positions and postures of the articles. The image processing unit 111 outputs the information on the three-dimensional positions and postures of the articles to the arm control unit 112, the hand control unit 113, and the transfer control unit 114. The image processing unit 111 may also output this information to the storage unit 115 for storage. Details of the image processing unit 111 will be described later.

[0078] Based on the instructions received from the input processing unit 201 and so on, the arm control unit 112 uses the information received from the image processing unit 111 to control the operations of the arm driving devices M1 to M6, thereby performing operations corresponding to the robot arm 12. The arm control unit 112 detects position and posture information including the positions, postures, moving directions, and moving speeds of the respective linkages 12a to 12f of the robot arm 12 and the robot hand 13 based on the operation amounts such as the rotation amounts of the servo motors 51 of the arm driving devices M1 to M6. The arm control unit 112 controls the operations of the arm driving devices M1 to M6 based on the position and posture information. In addition, the arm control unit 112 outputs the position and posture information to the hand control unit 113 and the transfer control unit 114.

[0079] Based on the instructions received from the input processing unit 201 and so on, the hand control unit 113 controls the operation of the negative pressure generating device 17. For example, the hand control unit 113 may detect the contact or proximity of the suction portion 13b to the article based on the information received from the image processing unit 111 and the position and posture information of the robot hand 13 received from the arm control unit 112, and after this detection, start the negative pressure generating device 17 to cause the suction portion 13b to adsorb the article.

[0080] Based on the instructions received from the input processing unit 201 and so on, the transfer control unit 114 controls the operation of the transfer driving device 14b, thereby performing operations corresponding to the transfer cart 14. For example, the transfer control unit 114 may use the information received from the image processing unit 111 and the position and posture information of the robot arm 12 and the robot hand 13 received from the arm control unit 112 to detect whether the article to be moved is within the reach of the robot arm 12. Alternatively, when the article is outside the reach, the transfer control unit 114 may operate in such a way as to bring the transfer cart 14 closer to the article.

[0081] Alternatively, it may be that the conveyance control unit 114 detects the position and orientation of the carrier vehicle 14 based on the amount of movement such as the rotation amount of the servo motor 51 of the conveyance drive device 14b, and controls the conveyance drive device 14b based on the position and orientation. Alternatively, it may be that the carrier vehicle 14 is equipped with a position measurement device such as a GPS (Global Positioning System) receiver and an IMU (Inertial Measurement Unit), and the position and orientation of the carrier vehicle 14 are detected using the measurement results of the position measurement device. Alternatively, it may be that the conveyance control unit 114 detects a weak induced current from a wire buried in the ground, for example, and detects the position and orientation of the carrier vehicle 14 based on the detected value.

[0082] <Functional Structure of Image Processing Unit>

[0083] The functional structure of the image processing unit 111 will be described. Figure 6 It is a block diagram showing an example of the functional structure of the image processing unit 111 according to the embodiment. As Figure 6 shown, the image processing unit 111 includes an extraction unit 111a, a distance detection unit 111b, a plane inference unit 111c, an angle detection unit 111d, a contour inference unit 111e, an edge extraction unit 111f, a determination unit 111g, and an object detection unit 111h as functional structure elements.

[0084] Figure 7 It is a diagram showing an example of the images captured by the cameras 41a and 41b. As Figure 6 and Figure 7 shown, the extraction unit 111a extracts images (hereinafter, also referred to as "second images") IBa and IBb of a partial region, that is, an object region TA, from the images (hereinafter, also referred to as "first images") IAa and IAb captured by the cameras 41a and 41b respectively, and outputs them to the distance detection unit 111b. The first images IAa and IAb are images captured at the same time. The positions, shapes, and sizes of the object regions TA of the first images IAa and IAb are the same and are preset. Information on the position, shape, and size of the object region TA is stored in the storage unit 115. The position, shape, and size of the object region TA can also be set using pixel coordinates, for example.

[0085] The object area TA has a rectangular shape with a size smaller than those of the first images IAa and IAb, and is disposed at the centers of the first images IAa and IAb. It should be noted that the size, shape, and position of the object area TA are not particularly limited, and it is only necessary to configure the object area TA to be smaller than the first images IAa and IAb and to be included within the first images IAa and IAb. Preferably, the object area TA has a shape and size such that the plane inference unit 111c can infer the plane area of the article within the second images IBa and IBb as described later. For example, the object area TA may also be smaller than the outer shape of one article shown in the first image. The smaller the size of the object area TA, the more the processing amounts of the distance detection unit 111b and the plane inference unit 111c can be reduced.

[0086] In addition, the position, shape, and size of the object area TA may also be changeable. For example, this change may also be performed via the input device 20. If information on the position, shape, and size of the changed object area TA is input to the input device 20, the extraction unit 111a replaces, that is, updates, the position, shape, and size of the object area TA stored in the storage unit 115 with the changed position, shape, and size of the object area TA. In this way, the position, shape, and size of the object area TA may also be arbitrarily set.

[0087] The distance detection unit 111b processes the second images IBa and IBb to detect the actual distance from the subject shown in the object area TA to the camera 41, and outputs it to the plane inference unit 111c. The position of the reference point for the distance of the camera 41 may also be any position. For example, the above reference point may also be a reference point set in the camera 41a, a reference point set in the camera 41b, or the midpoint О between the cameras 41a and 41b (refer to Figure 3 ), etc. In the present embodiment, the distance detection unit 111b calculates the actual distance, that is, the three-dimensional distance (hereinafter, simply referred to as "distance"), between the subject shown in the pixel pair of the second images IBa and IBb and the camera 41 for all pixel pairs of the second images IBa and IBb. The pixel pairs of the second images IBa and IBb are corresponding pixel pairs that show the same subject. In addition, it may also be that the distance detection unit 111b calculates the above distance for at least three pixel pairs with different pixel positions in the second images IBa and IBb.

[0088] For example, the distance detection unit 111b detects the distance from the subject imaged from at least three pixel pairs to the camera 41 through stereoscopic observation using the second images IBa and IBb. At this time, the distance detection unit 111b uses the second images IBa and IBb and the camera parameters of the cameras 41a and 41b stored in the storage unit 115. Specifically, for each pixel on the second image IBa, the distance detection unit 111b detects the pixel coordinates of this pixel and the pixel coordinates of the corresponding point on the second image IBb that represents the same subject as the subject imaged at this pixel. The detection method can be any known method.

[0089] The distance detection unit 111b calculates the parallax generated between the second images IBa and IBb according to the distance between the subject imaged at the pixel and the camera 41. "Parallax" refers to the difference in the direction or position of the object point when observing the same object point from cameras 41a and 41b at different positions. The parallax corresponds to the deviation of the positions of the pixels of the points imaged the same in the second images IBa and IBb. In other words, the difference in the pixel positions of the pixel coordinates of the pixel pair is the parallax. The distance detection unit 111b calculates the distance between the subject imaged at this pixel pair and the camera 41 through stereoscopic observation using the parallax based on the two pixel coordinates of the pixel pair and the camera parameters. The distance detection unit 111b performs the above processing for all the pixels on the second image IBa. Thus, the distance detection unit 111b calculates the distance value of the position of each pixel. Alternatively, the distance detection unit 111b generates a distance image with the distance value of the position of each pixel as the pixel value of this pixel.

[0090] Moreover, alternatively, the distance detection unit 111b uses the distance value of each pixel of at least one of the second images IBa and IBb and the direction of the line of sight from the cameras 41a and 41b towards this pixel to calculate the actual three-dimensional position of the subject imaged at this pixel, and outputs it to the plane inference unit 111c. Hereinafter, there are cases where the "three-dimensional position of the subject imaged at the pixel" is expressed as the "three-dimensional position corresponding to the pixel". The three-dimensional position can also be represented by three-dimensional coordinates such as a world coordinate system or a camera coordinate system representing a position on the earth.

[0091] The plane inference unit 111c infers a plane mapped within the object region TA using the distances of at least three parts in at least one of the second images IBa and IBb. In the present embodiment, a case where one part is one pixel is described, but it is not limited thereto. For example, one part may also be a pixel group composed of one point, one pixel, and multiple pixels. The plane inference unit 111c obtains from the distance detection unit 111b or calculates by itself the three-dimensional positions corresponding to at least three pixels of the second image IBa or IBb. The plane inference unit 111c uses the three-dimensional positions corresponding to at least three pixels to infer the above-mentioned plane. The plane inference unit 111c may calculate the plane passing through the three-dimensional positions corresponding to at least three pixels or a plane near them as the plane to be inferred. The plane inference unit 111c outputs information indicating the calculated plane, that is, the inferred plane, to the angle detection unit 111d. This information is information indicating the inferred plane in the three-dimensional space. The inferred plane may also be represented by coordinates such as in the world coordinate system or the camera coordinate system.

[0092] Alternatively, the plane inference unit 111c may smooth by removing noise components by averaging the distance values of multiple pixels of the second image IBa or IBb. The averaging method may be any known method. The plane inference unit 111c may average for all pixels of the second image IBa or IBb, or may average for a part of the pixels. In the latter case, it may be that the plane inference unit 111c extracts pixels whose difference in distance values between adjacent pixels is within a first threshold from the second image IBa or IBb, and averages for the pixel group of the extracted pixels. It may also be that the first threshold is arbitrarily set according to the smoothness and roughness of the inferred plane, etc., and stored in the storage unit 115. It may also be that, in the case of extracting multiple pixel groups, the plane inference unit 111c extracts pixel groups whose distance between pixel groups is within a second threshold, and regards the extracted pixel groups as pixel groups representing one plane. It may also be that the second threshold is arbitrarily set according to the shape of the article, etc., and stored in the storage unit 115.

[0093] The plane inference unit 111c uses the distance value of each pixel after averaging and the direction of the line of sight from the camera 41a or 41b toward the pixel to calculate the three-dimensional position corresponding to the pixel. Moreover, the plane inference unit 111c calculates the plane passing through the three-dimensional positions corresponding to each pixel or a plane near them as the plane to be inferred.

[0094] The angle detection unit 111d detects the angles of the inferred plane with respect to the optical axis centers LAa and LAb of the cameras 41a and 41b, and outputs them to the contour inference unit 111e. For example, the angle detection unit 111d calculates the above angles using the coordinates or functions representing the inferred plane and the coordinates or functions representing the optical axis centers LAa and LAb. For example, as Figure 8 shown, the angle detection unit 111d may also use the camera coordinate system to calculate the above angles. Figure 8 FIG. is an example of the inferred plane in the camera coordinate system.

[0095] The angle of the inferred plane is the angle between the inferred plane PE and the Z-axis. In the present embodiment, the angle detection unit 111d calculates the angle θa at which the intersection line La of the inferred plane PE and the YZ plane intersects the Z-axis. The angle θa is positive when representing the positive direction of the Y-axis angle and negative when representing the negative direction of the Y-axis angle. Figure 8 In, the angle θa is positive. Further, the angle detection unit 111d calculates the angle θb at which the intersection line Lb of the inferred plane PE and the XZ plane intersects the Z-axis. The angle θb is positive when representing the positive direction of the X-axis angle and negative when representing the negative direction of the X-axis angle. Figure 8 In, the angle θb is positive. The angle detection unit 111d calculates the angles θa and θb as the angles of the inferred plane. Thereby, the tilt direction and tilt angle of the inferred plane with respect to the Z-axis can be determined.

[0096] The contour inference unit 111e uses the shape of the article A stored in the storage unit 115 and the angle of the inferred plane to infer the contour of the article A reflected in the first images Iaa and Iab. The contour inference unit 111e outputs the information of the inferred contour to the determination unit 111g. Specifically, the contour inference unit 111e regards the angle between the inferred plane and the Z-axis as the angle representing the posture of the article A with respect to the optical axis centers LAa and LAb. At this time, the contour inference unit 111e extracts the planes on the outer surface of the article A from the shape of the article A. Further, the contour inference unit 111e calculates the posture of the article A when the angle between the extracted plane and the optical axis centers LAa and LAb, that is, the angle between the extracted plane and the Z-axis, coincides with the angle between the inferred plane and the Z-axis, and regards this posture as the posture of the article A.

[0097] Furthermore, based on the shape of article A, the contour inference unit 111e calculates the shape of the contour of the image obtained by projecting article A in this posture onto the XY plane (hereinafter also referred to as "contour shape"). This contour shape corresponds to the contour shape of article A reflected in the first images Iaa and Iab. This contour shape may be deformed with respect to the contour shape of article A when observed in the direction perpendicular to the extraction plane, and represents the contour shape of the appearance of article A. In this way, the contour inference unit 111e infers as the contour shape of article A reflected in the first images Iaa and Iab the shape obtained by deforming the shape of article A corresponding to the angles of the inference plane. In addition, the contour inference unit 111e may use the three-dimensional model of article A as the shape of article A. Thereby, the calculation accuracy of the contour shape of article A can be improved. In addition, when multiple postures of article A are calculated by means of multiple extraction planes or the like, the contour inference unit 111e calculates the contour shape of article A for each posture.

[0098] As Figure 7 shown, in the present embodiment, article A is a cardboard box in the shape of a rectangular parallelepiped, and multiple articles A of the same shape are stacked vertically, horizontally, left and right. Therefore, among the angles θa and θb of the inference plane, the angle θa is approximately 90°, and the angle θb is greater than 0° and less than or equal to 90°. Moreover, as Figure 9 shown, the shape of the contour CA of article A inferred by the contour inference unit 111e is trapezoidal or rectangular. Figure 9 FIG. is an example showing the shape of the contour CA of article A inferred by the contour inference unit 111e of the image processing unit 111.

[0099] The edge extraction unit 111f extracts edges from at least one of the first images Iaa and Iab. The edge extraction unit 111f generates an edge image including only the extracted edges and outputs it to the determination unit 111g. The edge image is an image corresponding to the first image Iaa or Iab that reflects only the edges.

[0100] The identification unit 111g identifies the object A in at least one of the first images IAa and IAb using the contour shape of the object A estimated by the contour estimation unit 111e. Specifically, the identification unit 111g compares the shape of the edge included in the edge image of the first image IAa or IAb with the contour shape of the object A and performs pattern matching to thereby identify the edge representing the object A. Alternatively, the identification unit 111g may identify the position of the object A in the first image IAa or IAb based on the position of the edge image of the identified edge. When the contour shapes of a plurality of objects A are estimated, the identification unit 111g performs pattern matching between the contour shapes of the plurality of objects A and the shape of the edge. Alternatively, when the first images IAa and IAb do not reflect the entire object A, the identification unit 111g may identify the edge representing the object A by performing pattern matching between a portion of the contour shape of the object A and a portion of the shape of the edge. Alternatively, the pattern matching described above may be geometric pattern matching using any known method.

[0101] exist Figure 9 In the example of , the edge of the outline CA of the object A shown in the image includes a corner protruding toward the upper left or upper right relative to the image. Therefore, the determination unit 111g searches for the edge representing the corner protruding toward the upper left or upper right, and performs pattern matching based on the corner. Thus, the processing amount of the determination unit 111g can be reduced.

[0102] The subject detection unit 111h detects the three-dimensional position and posture of the object A using the image of the object A on the first image IAa or IAb determined by the determination unit 111g and the angle of the estimated plane. Specifically, the subject detection unit 111h obtains the distance value or three-dimensional position corresponding to the pixel representing the object A on the first image IAa or IAb. At this time, the subject detection unit 111h uses the distance value or three-dimensional position of the pixel of the second image IBa or IBb. The subject detection unit 111h detects the three-dimensional position of the object A using the acquired distance value or three-dimensional position. Furthermore, the subject detection unit 111h detects the three-dimensional posture of the object A using the angle of the estimated plane and the shape of the object A. The subject detection unit 111h outputs the detection result to the arm control unit 112, the hand control unit 113, and the transport control unit 114.

[0103] <Robot system operation>

[0104] The operation of the robot system 1 according to the embodiment will be described. Figure 10 FIG. 1 is a flowchart showing an example of the operation of the robot system 1 according to the embodiment. Figure 10As shown, the imaging control unit 401 of the control device 100 of the robot system 1 causes the cameras 41a and 41b to image and acquire images of the object A (hereinafter, also referred to as "object A1") of the moving object of the robot 10, that is, the first images Iaa and Iab (step S101). For example, at the moment when the object A1 enters the fields of view of the cameras 41a and 41b that move along with the movement of the robot 10, the imaging control unit 401 performs imaging. In this case, it may also be that an operator or the like who visually recognizes the images of the cameras 41a and / or 41b displayed on the output device 30 determines the above moment and inputs an instruction to execute imaging to the input device 20. Alternatively, it may also be that a distance sensor (not shown) is provided on the robot hand 13, and if the distance value detected by the distance sensor indicates that the robot hand 13 approaches the object A1, the imaging control unit 401 performs imaging.

[0105] In addition, it may also be that before step S101 or during step S101, an operator or the like inputs information on the object A1 via the input device 20. The information on the object A1 at least includes information indicating the shape of the object A1. For example, an operator or the like may input the name or symbol of the shape attached to the object A1 as the above information, or may also cause the output device 30 to display a list of the shapes of the objects, and select the shape of the object A1 from the list. This list may also be stored in advance in the storage unit 115 of the control device 100.

[0106] Next, the image processing unit 111 of the control device 100 determines the object regions TA in the first images Iaa and Iab respectively, and extracts the second images Iba and Ibb included in the object regions TA (step S102).

[0107] Next, the image processing unit 111 detects the actual distance from the camera 41 to the object imaged in the second images Iba and Ibb (step S103). In this example, the image processing unit 111 detects the above distance for all pixel pairs corresponding to each other between the second images Iba and Ibb, but it may also detect the above distance for at least three pixel pairs. And the image processing unit 111 may also generate a distance image using the second image Iba or Ibb.

[0108] Next, the image processing unit 111 infers the planes imaged in the second images Iba and Ibb (step S104). In this example, the image processing unit 111 averages the distance values of a plurality of pixels in the second image Iba or Ibb, and based on the averaged distance values of each pixel, infers the pixel group of the imaged plane.

[0109] Next, the image processing unit 111 detects the angle of the inference plane with respect to the optical axis of the camera 41 (step S105). Specifically, the image processing unit 111 uses the distance value of each pixel showing the inference plane and the direction of the line of sight from the camera 41a or 41b toward the pixel to calculate the three-dimensional position corresponding to the pixel, that is, the three-dimensional position of the inference plane. Further, the image processing unit 111 detects the angle of the inference plane with respect to the optical axis centers LAa and LAb based on the three-dimensional position of the inference plane.

[0110] Next, the image processing unit 111 infers the contour of the article A1 imaged in the first images Iaa and Iab using the angle of the inference plane and the shape of the article A1 stored in the storage unit 115 (step S106). Specifically, the image processing unit 111 calculates the posture of the article A1 when the angles of the planes on the article A1 with respect to the optical axis centers LAa and LAb coincide with the angle of the inference plane, and infers the shape of the contour of the image of the article A1 in this posture projected onto the first images Iaa and Iab.

[0111] Next, the image processing unit 111 extracts edges from the first image Iaa or Iab and generates an edge image including only the extracted edges (step S107). Further, the image processing unit 111 determines the article A1 in the first image Iaa or Iab using the edge image and the contour shape of the article A1 (step S108). Specifically, the image processing unit 111 determines the edge indicating the contour shape of the article A1 in the edge image, and determines the article A1 in the first image Iaa or Iab based on the position of the edge in the edge image.

[0112] Next, the image processing unit 111 detects the three-dimensional position and posture of the article A1 using the image of the article A1 in the first image Iaa or Iab and the angle of the inference plane (step S109). Specifically, the image processing unit 111 detects the three-dimensional position of the article A1 using the distance value or three-dimensional position corresponding to the pixel including the article A1 and imaged in the second image Iba or Ibb. The image processing unit 111 detects the three-dimensional posture of the article A1 using the angle of the inference plane and the shape of the article A1.

[0113] Next, the image processing unit 111 outputs the detection result in step S109 to the arm control unit 112, the hand control unit 113, the conveyance control unit 114, etc. The arm control unit 112, the hand control unit 113, the conveyance control unit 114, etc. control the operations of the robot arm 12, the robot hand 13, and the conveyance vehicle 14 based on the above detection result, and adsorb and move the article A1 using the robot hand 13 (step S110).

[0114] Through steps S101 to S110, the control device 100 uses the first image of the article A1 captured by the cameras 41a and 41b to infer the angle with respect to the optical axis of the plane reflected in the first image, that is, the pose. Based on the inferred pose of the plane, the control device 100 infers the pose of the article A1 reflected in the first image, and based on this pose, infers the contour shape of the article A1 reflected in the first image. Further, the control device 100 uses the inferred contour shape of the article A1 to identify the article A1 in the first image.

[0115] <Effect, etc.>

[0116] The control device 100 of the image processing device according to the embodiment includes: a storage unit 115 that stores the shape of the article A as a subject; an extraction unit 111a that extracts a second image of an object region as a partial region from the first image obtained by the camera 41 photographing the article A; a distance detection unit 111b that detects the distances from at least three parts reflected in the object region to the camera 41 by processing the second image; a plane inference unit 111c that infers the plane reflected in the object region using the distances of at least three parts; an angle detection unit 111d that detects the angle of the inferred plane with respect to the optical axis of the camera 41; a contour inference unit 111e that infers the contour of the article A reflected in the first image using the shape of the article A stored in the storage unit 115 and the angle of the inferred plane; and a determination unit 111g that identifies the article A in the first image using the contour of the article A.

[0117] According to the above structure, the angle of the inferred plane reflected in the object region can represent the pose of the article A with respect to the optical axis of the camera 41. By using such an angle of the inferred plane and the shape of the article A, the contour of the article A in the first image can be inferred. Further, by using such a contour, the article A can be accurately identified in the first image, and the processing amount for identifying the article A can also be reduced. In addition, since the inference of the inferred plane uses the image of a partial region of the first image, the processing amount can be reduced. Therefore, the image processing accuracy and speed are improved.

[0118] Alternatively, the contour inference unit 111e may infer the contour of the article A in the first image as the shape obtained by deforming the shape of the article A stored in the storage unit 115 corresponding to the angle of the inferred plane. According to the above structure, the inferred contour of the article A represents the contour of the appearance of the article A in the first image. For example, when the angle of the inferred plane with respect to the optical axis is not 90°, the inferred plane is displayed in a deformed manner in the first image. The contour of the appearance of the article A in the first image can represent a high-precision contour reflecting the deformation of such an inferred plane.

[0119] Alternatively, it may be that the contour inference unit 111e uses the angle of the inferred plane as the angle representing the pose of the article A with respect to the optical axis of the camera 41 to infer the contour of the article A in the first image. According to the above structure, the processing amount for inferring the contour of the article A in the first image can be reduced.

[0120] Alternatively, it may be that the control device 100 includes an edge extraction unit 111f that extracts edges from an image, and the determination unit 111g determines the article A by comparing the shape of the edge with the contour of the article A. According to the above structure, the processing amount for determining the article A can be reduced.

[0121] Alternatively, it may be that the control device 100 includes a subject detection unit 111h, and the subject detection unit 111h uses the image of the article A determined by the determination unit 111g and the angle of the inferred plane to detect the three-dimensional position and pose of the article A. According to the above structure, the article A in the three-dimensional space can be detected.

[0122] In addition, the imaging device 40 according to the embodiment includes: a camera 41; and a control device 100 that processes the image captured by the camera 41. According to the above structure, the same effect as that of the control device 100 according to the embodiment is obtained.

[0123] Furthermore, it may be that the camera 41 is composed of at least two cameras 41a and 41b, the extraction unit 111a extracts at least two second images from the first images captured by the at least two cameras 41a and 41b, and the distance detection unit 111b detects the distances from at least three parts reflected in the object area to the camera 41 through stereo observation using the at least two second images. According to the above structure, the detection accuracy of the distances to the at least three parts is improved.

[0124] In addition, the robot 10 according to the embodiment includes: an imaging device 40; a robot arm 12 having a robot hand 13 as an end effector for performing a processing operation on an article; and a control device 100 that controls the operations of the robot hand 13 and the robot arm 12 based on the image of the article A determined by the determination unit 111g. According to the above structure, the same effect as that of the imaging device 40 according to the embodiment is obtained.

[0125] Alternatively, it is also possible that the imaging device 40 uses the image of the article A determined by the determination unit 111g and the angle of the inference plane to detect the three-dimensional position and posture of the article A, and the control device 100 controls the operations of the robotic hand 13 and the robotic arm 12 based on the three-dimensional position and posture of the article A. According to the above structure, the robot 10 can perform operations relative to the article A using the robotic hand 13 and the robotic arm 12 based on the three-dimensional position and posture of the article A detected using the image of the article A.

[0126] In addition, the robot system 1 according to the embodiment includes: a robot 10; and an input device 20 as an operating device for operating the robot 10. According to the above structure, the same effects as those of the robot 10 according to the embodiment are obtained.

[0127] (Modification 1)

[0128] A modification 1 of the embodiment will be described. The image processing unit 111A of the control device 100 according to the modification 1 is different from that of the embodiment in that it includes a size inference unit 111i. Hereinafter, for the modification 1, the description will focus on the differences from the embodiment, and the description of the same points as the embodiment will be appropriately omitted.

[0129] Figure 11 FIG. is a block diagram showing an example of the functional structure of the image processing unit 111A according to the modification 1. As Figure 11 shown, the image processing unit 111A further includes a size inference unit 111i as a functional structure element with respect to the image processing unit 111 according to the embodiment. In this modification, the storage unit 115 stores information on the shape and size of the articles that the robot 10 can move. The size of the article is the actual size of the article.

[0130] The size inference unit 111i uses the size of the article stored in the storage unit 115, that is, the first size, to infer the size of the article A1 on the first images IAa and IAb, that is, the second size. Specifically, the distance detection unit 111b detects the three-dimensional positions corresponding to at least two pixels on the second image IBa or IBb. The size inference unit 111i calculates the ratio of the size between the real image and the image on the image by comparing the three-dimensional positions corresponding to the at least two pixels with the two-dimensional positions of the at least two pixels on the second image IBa or IBb. The size inference unit 111i uses the above ratio and the first size of the article A1 to infer the second size of the article A1.

[0131] In addition, the determination unit 111g determines the article A1 on the first image IAa or IAb by using the contour shape of the article A1 inferred by the contour inference unit 111e and the second dimension of the article A1. Specifically, the determination unit 111g determines the size of the contour shape of the article A1 corresponding to the second dimension, and uses the determined size and the contour shape of the size in the vicinity thereof to determine the edge representing the article A1 on the edge image of the first image IAa or IAb. The determination unit 111g determines the position of the article A1 on the first image IAa or IAb based on the position of the edge on the edge image.

[0132] The operation of the robot system 1 according to the first modification will be described. Figure 12 FIG. is a flowchart showing an example of the operation of the robot system 1 according to the first modification. As Figure 12 shown, the processes of steps S201 to S206 are the same as the processes of steps S101 to S106 in the embodiment. Alternatively, before step S201 or during step S201, an operator or the like may input information of the article A1 including at least information indicating the shape and size of the article A1 via the input device 20. Further, in step S203, the image processing unit 111A of the control device 100 detects the three-dimensional position corresponding to all pixel pairs corresponding to each other between the second images IBa and IBb.

[0133] In step S207, the image processing unit 111A infers the second dimension of the article A1 on the first images IAa and IAb by using the three-dimensional positions corresponding to the respective pixels of the second images IBa and IBb, the two-dimensional positions of the respective pixels on the images, and the first dimension of the article A1. Further, the processes of steps S208, S210, and S211 are the same as the processes of steps S107, S109, and S110 in the embodiment.

[0134] In step S209, the image processing unit 111A determines the article A1 on the first image IAa or IAb by using the contour shape and the second dimension of the article A1.

[0135] According to Modification Example 1 as described above, the same effects as those of the embodiment are obtained. Further, it may be that the image processing unit 111A according to Modification Example 1 includes a size inference unit 111i. The size inference unit 111i infers the size of the article A on the image captured by the camera 41, that is, the second size. The storage unit 115 stores the actual size of the article A, that is, the first size. The distance detection unit 111b uses the distance at the part reflected in the object area to detect the three-dimensional position of the part. Further, it may be that the size inference unit 111i infers the second size using the three-dimensional position of the part, the two-dimensional position of the part on the image captured by the camera 41, and the first size. The determination unit 111g determines the article A using the outline of the article A and the second size. According to the above structure, the article A is determined on the image using the outline of the article A and the size of the article A on the image. Therefore, the determination accuracy of the article A is improved. Also, the processing amount for determining the article A can be reduced.

[0136] (Modification Example 2)

[0137] A modification example 2 of the embodiment will be described. The image processing unit 111B of the control device 100 according to Modification Example 2 is different from that of the embodiment in that it includes a direction determination unit 111j. Hereinafter, for Modification Example 2, the description will be centered on the differences from the embodiment and Modification Example 1, and the description of the same points as those of the embodiment and Modification Example 1 will be appropriately omitted.

[0138] Figure 13 FIG. is a block diagram showing an example of the functional configuration of the image processing unit 111B according to Modification Example 2. As Figure 13 shown, the image processing unit 111B further includes a direction determination unit 111j as a functional configuration element with respect to the image processing unit 111 according to the embodiment.

[0139] The direction determination unit 111j orients the camera 41 so that the optical axis direction of the camera 41 coincides with the direction perpendicular to the inferred plane inferred by the plane inference unit 111c, that is, the first direction. Specifically, the direction determination unit 111j determines the directions of the optical axis centers LAa and LAb of the cameras 41a and 41b to be the first direction perpendicular to the inferred plane. Further, the direction determination unit 111j outputs an instruction to orient the cameras 41a and 41b so that the directions of the optical axis centers LAa and LAb coincide with the first direction and an instruction to cause the cameras 41a and 41b to capture images after orientation to the shooting control unit 401. After orientation, the cameras 41a and 41b face the article A1, and the optical axis centers LAa and LAb are perpendicular to the inferred plane.

[0140] Alternatively, the shooting control unit 401 outputs an instruction to the arm control unit 112 to move the robot arm 12, thereby orienting the cameras 41a and 41b in a desired direction. Additionally, the cameras 41 may be arranged on the robot arm 12 via a cymbal-shaped member (not shown) capable of arbitrarily changing the orientation of the cameras 41, and the shooting control unit 401 controls the movement of the cymbal-shaped member to orient the cameras 41a and 41b in a desired direction. Alternatively, after the orientation, the shooting control unit 401 causes the cameras 41a and 41b to capture an image of the article A1, i.e., the third image, and outputs the third image to the contour inference unit 111e. Alternatively, the shooting control unit 401 stores the third image in the storage unit 115.

[0141] The contour inference unit 111e uses the shape of the article A1 stored in the storage unit 115 to infer the contour of the article A1 reflected in the third image. The contour inference unit 111e calculates the posture of the article A1 when the plane on the article A1 is perpendicular to the optical axis centers LAa and LAb. Then, based on the shape of the article A1, the contour inference unit 111e calculates the contour shape of the image obtained by projecting the article A1 in the calculated posture onto the XY plane of the camera coordinate system. The number of contour shapes that can be calculated when the plane on the article A1 is perpendicular to the optical axis centers LAa and LAb is smaller than when it is not perpendicular. Therefore, the processing amount of the determination unit 111g can be reduced.

[0142] The operation of the robot system 1 according to the second modification example will be described. Figure 14 It is a flowchart showing an example of the operation of the robot system 1 according to the second modification example. As Figure 14 shown, the processes in steps S301 to S305 are the same as those in steps S101 to S105 of the embodiment, respectively.

[0143] In step S306, the image processing unit 111B of the control device 100 orients the camera 41 so that the angle thereof is consistent with the inferred plane inferred in step S305. Specifically, the image processing unit 111B outputs an instruction to the shooting control unit 401 to orient the cameras 41a and 41b such that the directions of the optical axis centers LAa and LAb of the cameras 41a and 41b are perpendicular to the inferred plane. In this example, the shooting control unit 401 outputs an instruction to the arm control unit 112 to move the robot arm 12, thereby orienting the cameras 41a and 41b in the above direction. Then, the shooting control unit 401 causes the cameras 41a and 41b after the orientation to capture the article A1 and obtains a third image including the article A1 (step S307).

[0144] Next, the image processing unit 111B infers the outline of the article A1 on the third image (step S308). Specifically, the image processing unit 111B infers the outline shape of the image obtained by projecting the article A1 onto the third image when the plane on the article A1 is perpendicular to the optical axis centers LAa and LAb. In this example, the outline shape of the article A1 is rectangular. Next, the image processing unit 111B generates an edge image by extracting edges on the third image (step S309). And, the image processing unit 111B determines the article A1 on the third image using the edge image and the outline shape of the article A1 (step S310).

[0145] Next, the image processing unit 111B uses the image of the article A1 on the third image to detect the three-dimensional position and orientation of the article A1 (step S311). Specifically, the image processing unit 111B, in the same manner as in the case of the second image, detects the three-dimensional position and orientation of the article A1 through stereo observation of two third images using the cameras 41a and 41b. In addition, the processing of step S312 is the same as that of step S110 in the embodiment.

[0146] According to the modification example 2 as described above, the same effects as those of the embodiment are obtained. And, it may be that the image processing unit 111B according to the modification example 2 includes an orientation determination unit 111j, and the orientation determination unit 111j orients the camera 41 so that the direction of the optical axis of the camera 41 coincides with the direction perpendicular to the inferred plane inferred by the plane inference unit 111c, that is, the first direction, and the outline inference unit 111e infers the outline of the article A that appears in the third image obtained by photographing the article A with the camera 41 whose optical axis is oriented in the first direction using the shape of the article A stored in the storage unit 115. According to the above structure, the camera 41 is oriented so that the direction of the optical axis becomes the first direction perpendicular to the inferred plane and the plane on the article A. The outline of the article A on the third image photographed by such a camera 41 represents the outline when observing the plane on the article A from the front. Therefore, it is possible to improve the inference accuracy of the outline of the article A and reduce its processing amount.

[0147] (Modification Example 3)

[0148] A modification example 3 of the embodiment will be described. The image processing unit 111C of the control device 100 according to the modification example 3 is different from that of the modification example 2 in that it includes a size inference unit 111i. Hereinafter, for the modification example 3, the description will be centered on the differences from the embodiment and the modification examples 1 to 2, and the description of the same points as those of the embodiment and the modification examples 1 to 2 will be appropriately omitted.

[0149] Figure 15 is a block diagram showing an example of the functional structure of the image processing unit 111C according to the modification example 3. AsFigure 15 As shown, the image processing unit 111C further includes a size inference unit 111i as a functional structural element with respect to the image processing unit 111B according to Modification 2. The structure of the size inference unit 111i is the same as that of Modification 1.

[0150] The operation of the robot system 1 according to Modification 3 will be described. Figure 16 is a flowchart showing an example of the operation of the robot system 1 according to Modification 3. As Figure 16 shown, the processes of steps S401 to S405 are the same as those of steps S301 to S305 of Modification 2, respectively. The process of step S406 is the same as that of step S207 of Modification 1. The processes of steps S407 to S410 are the same as those of steps S306 to S309 of Modification 2, respectively.

[0151] In step S411, the image processing unit 111C of the control device 100 determines the article A1 in the third image using the contour shape of the article A1 on the third image inferred in step S409 and the second size of the article A1 on the first image inferred in step S406. Specifically, the image processing unit 111C determines the size of the contour shape of the article A1 in a manner corresponding to the second size, and uses the determined size and the contour shape of the article A1 with the sizes in the vicinity thereof to determine the edge representing the article A1 on the edge image of the third image. The image processing unit 111C determines the position of the article A1 in the third image based on the position of the edge on the edge image. In addition, the processes of steps S412 and S413 are the same as those of steps S311 and S312 of Modification 2, respectively. Moreover, according to Modification 3 as described above, the same effects as those of Modifications 1 and 2 are obtained.

[0152] (Modification 4)

[0153] Modification 4 of the embodiment will be described. The image processing unit 111D of the control device 100 according to Modification 4 is different from the embodiment in that it includes a surface area extraction unit 111k. Hereinafter, for Modification 4, the description will focus on the differences from the embodiment and Modifications 1 to 3, and the description of the same points as the embodiment and Modifications 1 to 3 will be appropriately omitted.

[0154] Figure 17 is a block diagram showing an example of the functional structure of the image processing unit 111D according to Modification 4. As Figure 17As shown, the image processing unit 111D further includes a surface area extraction unit 111k as a functional structural element with respect to the image processing unit 111 according to the embodiment. In this modified example, the storage unit 115 stores information on the surface state of the object item that the robot 10 can move. The information on the surface state of the item is information on surface elements indicating the surface state such as the color and texture of the surface of the item.

[0155] The surface area extraction unit 111k extracts, on the image, the area that reflects the image corresponding to the surface state of the item A1 as the object item, that is, the surface area. Specifically, the surface area extraction unit 111k extracts the surface area that reflects the image in the same state as the surface state of the item A1.

[0156] For example, the surface area extraction unit 111k extracts the surface area that reflects the same color as the surface of the item A1. In this case, the surface area extraction unit 111k sequentially scans the pixels in the image and extracts the pixels representing the luminance values of the color that is the same as or similar to the color of the surface of the item A1 as the extraction pixels. And the surface area extraction unit 111k extracts the area formed by a group of adjacent extraction pixels as the surface area. For example, the surface area extraction unit 111k may also use the area including a specified number or more of extraction pixels as the surface area. The specified number may be determined according to the structure of the item A1 and the distance from the camera 41 to the item A1, etc.

[0157] For example, the surface area extraction unit 111k extracts the surface area that reflects the same texture as the surface of the item A1. In this case, the surface area extraction unit 111k extracts the area of the texture representing the feature quantity that is the same as or similar to the feature quantity of the surface texture of the item A1. Specifically, the surface area extraction unit 111k moves the scanning frame to scan the entire image, and at each detection position during the scanning, detects the feature quantity of the texture formed by the pixels within the scanning frame. The detection positions may be configured such that the entire scanning frame at all detection positions covers the entire image. For example, the detection positions may be configured such that the scanning frames of adjacent detection positions are adjacent to each other or partially overlap.

[0158] The surface area extraction unit 111k compares the feature quantity of the surface texture of the item A1 stored in the storage unit 115 with the texture feature quantity within the scanning frame at each detection position, and determines the sameness, similarity, and non - similarity of the textures. The surface area extraction unit 111k uses the area within the scanning frame at the detection position where the textures are the same or similar as the surface area. The size of the scanning frame may also be such that it includes a predetermined number or more of pixels that can be regarded as the surface area.

[0159] The surface area extraction unit 111k may also determine, as the surface area, a region in which at least one of surface elements such as color and texture is the same as or similar to the surface state of the object article A1.

[0160] In addition, the determination unit 111g uses the contour shape of the article A1 inferred by the contour inference unit 111e and the surface area extracted by the surface area extraction unit 111k to determine the article A1 on the first image I Aa or I Ab. Specifically, the determination unit 111g determines the surface area on the edge image of the first image I Aa or I Ab based on the pixel coordinates of the pixels constituting the surface area. Further, the determination unit 111g extracts the edges near the surface area on the edge image. The edges near the surface area may include edges overlapping the surface area, edges adjacent to the surface area, edges within a predetermined distance from the surface area, and edges surrounding at least a part of the surface area. The determination unit 111g determines the edges representing the article A1 by comparing the shape of the extracted edges with the contour shape of the article A1 and performing pattern matching. The determination unit 111g determines the position of the article A1 on the first image I Aa or I Ab based on the position of the edges on the edge image.

[0161] The operation of the robot system 1 according to the fourth modification will be described. Figure 18 is a flowchart showing an example of the operation of the robot system 1 according to the fourth modification. As Figure 18 shown, the processes of steps S501 to S506 are the same as the processes of steps S101 to S106 in the embodiment, respectively.

[0162] In step S507, the image processing unit 111D of the control device 100 extracts, on the first image I Aa or I Ab, the surface area where an image corresponding to the surface state of the article A1 is reflected. Next, in step S508, the image processing unit 111D generates an edge image of the first image I Aa or I Ab. Next, in step S509, the image processing unit 111D uses the edge image, the contour shape of the article A1, and the information on the surface area corresponding to the surface state of the article A1 to determine the article A1 on the first image I Aa or I Ab. In addition, the processes of steps S510 and S511 are the same as the processes of steps S109 and S110 in the embodiment, respectively.

[0163] According to Modification Example 4 as described above, the same effects as those of the embodiment are obtained. Further, it may be that the image processing unit 111D according to Modification Example 4 includes a surface area extraction unit 111k. The surface area extraction unit 111k extracts a surface area corresponding to the surface state of the article A in the image captured by the camera 41. The surface area extraction unit 111k extracts the surface area using the information on the surface state stored in the storage unit 115. The determination unit 111g determines the article A using the contour and the surface area of the article A. According to the above structure, the article A is determined in the image using the contour of the article A and the surface area corresponding to the article A. Therefore, the determination accuracy of the article A is improved. Further, the process using the surface area extraction unit 111k of the image processing unit 111D can also be applied to Modification Examples 1 to 3.

[0164] (Modification Example 5)

[0165] A fifth modification example of the embodiment will be described. The image processing unit 111E of the control device 100 according to the fifth modification example is different from that of the embodiment in that it is configured to search for other articles in the image. Hereinafter, the fifth modification example will be mainly described with respect to the differences from the embodiment and the first to fourth modification examples, and the description of the same points as the embodiment and the first to fourth modification examples will be appropriately omitted.

[0166] Figure 19 FIG. is a block diagram showing an example of the functional configuration of the image processing unit 111E according to the fifth modification example. As Figure 19 shown, the image processing unit 111E further includes a peripheral search unit 111p as a functional configuration element with respect to the image processing unit 111 according to the embodiment.

[0167] The peripheral search unit 111p searches around the image of the article A1 determined by the determination unit 111g in at least one of the first images Iaa and Iab, thereby detecting an image of another article A that is the same as the article A1. Specifically, the peripheral search unit 111p compares the shape of the edge around the image of the article A1 with the contour shape of the article A1 inferred by the contour inference unit 111e on the edge image of the first image Iaa or Iab and performs pattern matching, thereby determining the edge representing the other article A. Although not limited thereto, in the present embodiment, the peripheral search unit 111p determines the edge of the other article A by regarding the posture of the plane on the article A1 as the same as the posture of the plane on the other article A.

[0168] For example, as Figure 7As shown in the first images I Aa and I Ab, around the article A1 imaged in the object area TA, there may be an article A imaged as a whole and an article A imaged as a part. Therefore, the peripheral search unit 111p performs pattern matching for the overall shape and the partial shape.

[0169] Based on the position of the edge of the other article A determined, the peripheral search unit 111p determines the position of the other article A on the first images I Aa and I Ab. Further, the peripheral search unit 111p outputs the position information of the other article A on the first images I Aa and I Ab to the photographing control unit 401. The position information of the article A may be information indicating the position of a determined part such as the center, corner, or side of the article A, and is represented by pixel coordinates, for example. When the peripheral search unit 111p detects a plurality of articles A, it may output the position information of one article A, or may output the position information of two or more articles A. For example, when the peripheral search unit 111p outputs the position information of one article A, it may determine the article A closest to the article A1 as the output target, or may determine the article A that is the output target according to a preset movement order of the article A such as the up-down direction or the left-right direction.

[0170] After the robot 10 moves the article A1, the photographing control unit 401 directs the cameras 41a and 41b so that the article A that is the target of the position information received from the peripheral search unit 111p becomes the center of the image, and photographs the article A. In the image photographed by the cameras 41a and 41b in this way, the article A is imaged in the object area TA and can be subjected to processing by the image processing unit 111E.

[0171] According to the modification 5 as described above, the same effect as the embodiment is obtained. Further, the image processing unit 111E according to the modification 5 includes a peripheral search unit 111p that searches around the article A1 determined by the determination unit 111g on the first image. Further, the peripheral search unit 111p uses the contour of the article A1 inferred by the contour inference unit 111e to search for other articles A similar to the article A1, and detects the position of the other article A on the first image. According to the above configuration, the image processing unit 111E can detect the image and position of the article A similar to the article A1 on the first image. The photographing control unit 401 can photograph the article A based on the position of the article A as the next movement target of the article A1. Further, since the image processing unit 111E uses the contour of the article A1 inferred by the contour inference unit 111e for the search, the processing amount thereof can be suppressed to be low. In addition, the processing of the peripheral search unit 111p of the image processing unit 111E can also be used for Modifications 1 to 4.

[0172] <Other Embodiments>

[0173] As described above, examples of the embodiments of the present disclosure have been described, but the present disclosure is not limited to the above-described embodiments and modified examples. That is, various modifications and improvements can be made within the scope of the present disclosure. For example, the manner in which various modifications are applied to the embodiments and modified examples and the forms constructed by combining the constituent elements of different embodiments and modified examples are also included within the scope of the present disclosure.

[0174] For example, in the embodiments and modified examples, the camera 41 has a structure including a compound eye camera, specifically a structure including a stereo camera, but is not limited thereto. For example, the camera 41 may also have a structure including a monocular camera, a TOF camera (Time-of-Flight-Camera), a pattern light projection camera such as fringe projection, or a camera using the light cutting method. For example, when the camera 41 has a structure including a monocular camera, the camera 41 further includes a distance measuring device such as a laser or a lidar. In this case, it may also be that the image processing unit detects the distance between the position of the light of the distance measuring device in the camera image and the measurement result of the distance measuring device, and the three-dimensional position of the light irradiation destination based on the position of the light of the distance measuring device in the camera image and the measurement result of the distance measuring device. Alternatively, it may also be that when the camera 41 does not include a distance measuring device, the image processing unit detects the distance between the camera 41 and the subject and the three-dimensional position of the subject based on the changes in the shape and size of the subject reflected in the camera image accompanying the change in the shooting position of the monocular camera.

[0175] In addition, in the robot system 1 according to the embodiments and modified examples, the number and structure of the input device 20 and the robot 10 can be any number and structure. For example, the robot system 1 may also include one or more input devices 20 and one or more robots 10. In addition, the robot 10 may also include one or more robot arms 12. In addition, the robot 10 is not limited to an industrial robot of the vertical multi-joint type. For example, it may also be configured as an industrial robot of the horizontal multi-joint type, polar coordinate type, cylindrical coordinate type, rectangular coordinate type, vertical multi-joint type, or other forms. And the robot 10 may also be a robot other than an industrial robot, and may also be a service robot, a construction machine, a crane, a cargo handling vehicle, and a humanoid robot, etc. A service robot is a robot used in various service industries such as nursing, medical care, cleaning, security, guidance, rescue, cooking, and commodity provision.

[0176] In addition, all the numbers such as ordinal numbers and quantities used above are merely examples for specifically illustrating the technology of the present disclosure, and the present disclosure is not limited to the illustrated numbers. In addition, the connection relationships between components and the step sequences are examples for specifically illustrating the technology of the present disclosure. The connection relationships between components and the step sequences for implementing the functions of the present disclosure are not limited to the embodiments and modification examples, and can be changed or replaced.

[0177] In addition, the division of the functional blocks in the functional block diagram is an example, and multiple functional blocks can also be implemented as one functional block, one functional block can be divided into multiple, and / or a part of the function can be transferred to other functional blocks. In addition, the functions of multiple functional blocks with similar functions can also be processed in parallel by a single piece of hardware or software or in a time-division manner.

[0178] Description of Reference Numerals

[0179] 1... Robot system; 10... Robot; 12... Robot arm; 13... Robot hand (end effector); 20... Input device (operating device); 40... Imaging device; 41, 41a, 41b... Cameras; 100... Control device (image processing device); 111, 111A, 111B, 111C, 111D, 111E... Image processing units; 111a... Extraction unit; 111b... Distance detection unit; 111c... Plane inference unit; 111d... Angle detection unit; 111e... Contour inference unit; 111f... Edge extraction unit; 111g... Determination unit; 111h... Object detection unit; 111i... Size inference unit; 111j... Direction determination unit; 111k... Surface area extraction unit; 111p... Peripheral search unit.

Claims

1. An image processing apparatus for processing an image, characterized in that, Comprising: a storage unit that stores a subject model representing the shape of a subject; an extraction unit that extracts a second image of a partial region, i.e., an object region, from a first image obtained by the camera photographing the subject; a distance detection unit that detects the distances from at least three parts imaged in the object region to the camera by processing the second image; a plane inference unit that infers a plane imaged in the object region using the distances at the at least three parts; an angle detection unit that detects the angle of the plane with respect to the optical axis of the camera; a contour inference unit that infers the contour of the subject imaged in the first image, i.e., the subject contour, using the subject model stored in the storage unit and the angle of the plane; and a determination unit that determines the subject on the first image using the subject contour, wherein the contour inference unit detects the posture of the subject with respect to the optical axis of the camera based on the subject model and the angle of the plane, and infers the contour of the image of the subject when the subject in the posture is projected onto the first image as the subject contour based on the subject model.

2. The image processing apparatus according to claim 1, wherein the contour inference unit infers the shape obtained by deforming the shape of the subject stored in the storage unit corresponding to the angle of the plane as the subject contour.

3. The image processing apparatus according to claim 2, wherein the contour inference unit uses the angle of the plane as the angle representing the posture of the subject with respect to the optical axis of the camera to infer the subject contour.

4. The image processing apparatus according to any one of claims 1 to 3, wherein it further comprises an edge extraction unit that extracts edges from an image, and the determination unit determines the subject by comparing the shape of the edges with the subject contour.

5. The image processing apparatus according to any one of claims 1 to 3, wherein it further comprises a size inference unit that infers the size of the subject, i.e., a second size, on the image photographed by the camera, the storage unit stores the actual size of the subject, i.e., a first size, the distance detection unit uses the distances at the parts to detect the three-dimensional positions at the parts, the size inference unit uses the three-dimensional positions at the parts, the two-dimensional positions of the parts on the image photographed by the camera, and the first size to infer the second size, and the determination unit uses the subject contour and the second size to determine the subject.

6. The image processing apparatus according to any one of claims 1 to 3, wherein it further comprises a subject detection unit that uses the image of the subject determined by the determination unit and the angle of the plane to detect the three-dimensional position and posture of the subject.

7. The image processing apparatus according to any one of claims 1 to 3, characterized in that: It further includes a surface area extraction unit that extracts a surface area corresponding to the surface state of the subject on the first image, The storage unit stores information on the surface state of the subject, The surface area extraction unit extracts the surface area using the information on the surface state stored in the storage unit, The determination unit determines the subject on the first image using the subject contour and the surface area.

8. The image processing apparatus according to any one of claims 1 to 3, characterized in that: It further includes a peripheral search unit that searches the periphery of the subject determined by the determination unit on the first image, The peripheral search unit uses the subject contour to detect other subjects similar to the subject.

9. An image processing apparatus for processing an image, characterized in that, Comprises: A storage unit that stores a subject model representing the shape of the subject; An extraction unit that extracts a second image of a partial area, i.e., an object area, from a first image obtained by the camera photographing the subject; A distance detection unit that detects the distances from at least three parts imaged in the object area to the camera by processing the second image; A plane inference unit that infers the plane imaged in the object area using the distances at the at least three parts; An angle detection unit that detects the angle of the plane with respect to the optical axis of the camera; A direction determination unit that orients the camera so that the direction of the optical axis of the camera coincides with the direction perpendicular to the plane, i.e., the first direction; A contour inference unit that infers the contour of the subject, i.e., the subject contour, imaged in a third image using the subject model stored in the storage unit, where the third image is an image obtained by the camera with the optical axis oriented in the first direction photographing the subject; And A determination unit that determines the subject on the third image using the subject contour, The contour inference unit calculates the pose of the subject when the optical axis is oriented in the first direction, and infers the contour of the image of the subject when the subject in the pose is projected onto the third image based on the subject model as the subject contour.

10. The image processing apparatus according to claim 9, characterized in that: It further includes an edge extraction unit that extracts edges from the image, The determination unit determines the subject by comparing the shape of the edge with the subject contour.

11. The image processing apparatus according to claim 9 or 10, characterized in that: It further includes a size inference unit that infers the size of the subject, i.e., the second size, on the image photographed by the camera, The storage unit stores the actual size of the subject, i.e., the first size, The distance detection unit uses the distances at the parts to detect the three-dimensional positions at the parts, The dimension inference unit uses the three-dimensional position at the part, the two-dimensional position of the part on the image captured by the camera, and the first dimension to infer the second dimension. The determination unit uses the captured object contour and the second dimension to determine the captured object.

12. The image processing apparatus according to claim 9 or 10, wherein: It further includes a captured object detection unit that uses the image of the captured object determined by the determination unit and the angle of the plane to detect the three-dimensional position and posture of the captured object.

13. The image processing apparatus according to claim 9 or 10, wherein: It further includes a surface area extraction unit that extracts a surface area corresponding to the surface state of the captured object on the first image. The storage unit stores information on the surface state of the captured object. The surface area extraction unit uses the information on the surface state stored in the storage unit to extract the surface area. The determination unit uses the captured object contour and the surface area to determine the captured object on the first image.

14. The image processing apparatus according to claim 9 or 10, wherein: It further includes a surrounding search unit that searches the periphery of the captured object determined by the determination unit on the first image. The surrounding search unit uses the captured object contour to detect other captured objects similar to the captured object.

15. A photographing device, characterized in that, Comprising: A camera; and The image processing apparatus according to any one of claims 1 to 14 that processes the image captured by the camera.

16. A robot, characterized in that, Comprising: The imaging device according to claim 15; A robot arm having an end effector for performing a processing operation on an article; and A control device that controls the actions of the end effector and the robot arm based on an image of the article that is the captured object determined by the determination unit.

17. The robot according to claim 16, wherein: The imaging device uses the image of the article determined by the determination unit and the angle of the plane to detect the three-dimensional position and posture of the article. The control device controls the actions of the end effector and the robot arm based on the three-dimensional position and posture of the article.

18. A robot system, characterized in that, Comprising: The robot according to claim 16 or 17; and An operating device for operating the robot.

Images