Information processing apparatus, information processing method, robot system, control method of robot system, method for manufacturing articles, program, and recording medium

Abstract

To generate a point cloud with high accuracy.SOLUTION: An information processing apparatus is equipped with a processing unit. The processing unit, respectively, acquires a plurality of image data of an object captured under different relative positions of the object and an imaging unit and different imaging conditions of the imaging unit which set the brightness and darkness of the image to be acquired (S21-S23); calculates, from the plurality of image data, an area in which a point cloud can be generated in each image data (S31-S32); and sets the image capture conditions for capturing the object with the image capture unit based on the calculated area (S41).SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, a robot system, a control method for a robot system, a manufacturing method for an article, a program, and a recording medium. [Background technology]

[0002] For example, there is a system that uses an imaging unit such as a stereo camera to capture an image of an object (workpiece), measures the three-dimensional shape of the object (height information of the workpiece), and calculates the position and orientation of the object (see Patent Document 1). When capturing an image of an object in this way, if the exposure time of the image sensor is not appropriate, the brightness value in the captured image data may saturate (i.e., white out) or the contrast may be low. Then, when generating (extracting) a point cloud representing the object from the image data, the point cloud cannot be generated with high accuracy, that is, the position and orientation of the object cannot be measured with high accuracy. For this reason, Patent Document 1 proposes capturing an image of the object while changing the exposure time in stages, determining the exposure time at which the number of effective pixels satisfies a predetermined condition, and setting the determined exposure time to a set value. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-27947 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the posture (angle) of the object changes with respect to the imaging direction of the imaging unit, the amount of light reflected by the surface of the object changes. Therefore, even if the exposure time is set as in Patent Document 1, the exposure time may not be appropriate for an object whose posture is different from when it was set, which means that the point cloud may not be extracted accurately.

[0005] Therefore, an object of the present invention is to provide an information processing device, an information processing method, a robot system, a control method for a robot system, a manufacturing method for an article, a program, and a recording medium that are capable of generating a point cloud with high accuracy. [Means for solving the problem]

[0006] One aspect of the present invention is an information processing device that includes a processing unit, wherein the processing unit acquires multiple image data of an object captured under different relative positional relationships between the object and an imaging unit and different imaging conditions of the imaging unit that set the brightness of the image to be acquired, acquires from the multiple image data an area from which a point cloud can be generated in each image data, and sets the imaging conditions for when the object is captured by the imaging unit based on the acquired areas.

[0007] One aspect of the present invention is an information processing device that includes a processing unit, wherein the processing unit acquires multiple image data of an object captured under different relative positional relationships between the object and an imaging unit and different imaging conditions of the imaging unit that set the brightness of the image to be acquired, generates a point cloud corresponding to each of the multiple image data from each of the multiple image data, calculates, for each of the point clouds corresponding to each of the generated image data, a ratio of the number of points to a point cloud that would be captured and generated if the object were captured by the imaging unit in the said positional relationship, and sets the imaging conditions for when the object is captured by the imaging unit based on the calculated ratio.

[0008] One aspect of the present invention is an information processing method for processing information by a processing unit, comprising: an image acquisition step in which the processing unit acquires multiple image data of an object captured under different relative positional relationships between the object and an imaging unit and different imaging conditions of the imaging unit that set the brightness of the image to be acquired; an area calculation step in which the processing unit calculates, from the multiple image data, an area in each image data from which a point cloud can be generated; and a setting step in which the processing unit sets the imaging conditions when capturing an image of the object with the imaging unit based on the calculated area.

[0009] One aspect of the present invention is an information processing method for processing information by a processing unit, comprising: an image acquisition step in which the processing unit acquires multiple image data of an object captured under different relative positional relationships between the object and an imaging unit and different imaging conditions of the imaging unit that set the brightness of the image to be acquired; a proportion calculation step in which the processing unit generates a point cloud corresponding to each of the multiple image data from each of the multiple image data, and calculates, for each of the point clouds corresponding to each of the generated image data, the proportion of the number of points to a point cloud that would be captured and generated if the object were captured by the imaging unit in the said positional relationship; and a setting step in which the processing unit sets the imaging conditions for when the object is captured by the imaging unit based on the calculated proportion. Effect of the Invention

[0010] According to the present invention, a point cloud can be generated with high accuracy regardless of the posture of an object. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic diagram of a robot system according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram of a robot system according to a first embodiment. [Diagram 3] FIG. 2 is a block diagram of an image processing controller according to the first embodiment. [Figure 4]4 is a flowchart of an operation process in an operation mode according to the first embodiment. [Diagram 5] 5A to 5C are schematic diagrams illustrating the operation of the robot system in a setting mode according to the first embodiment. [Figure 6] FIG. 4 is an explanatory diagram showing an example of a start screen of a setting mode according to the first embodiment; [Figure 7] 5 is a flowchart showing exposure time setting control in a setting mode according to the first embodiment. [Figure 8] 6 is a diagram showing the relationship between the number of pixels at each angle measured in effective pixel number measurement according to the first embodiment and exposure time. FIG. [Figure 9] 13 is an explanatory diagram showing an example of a display image of an exposure time adjustment result screen in the setting mode according to the first embodiment; FIG. [Figure 10] 10 is a flowchart showing exposure time setting control in a setting mode according to the second embodiment. [Figure 11] 13A is a diagram showing a case where a single exposure time is selected in relation to the exposure time and the point cloud acquisition rate of each angle measured in the point cloud acquisition rate measurement according to the second embodiment, and FIG. 13B is a diagram showing a case where a plurality of exposure times are selected in relation to the exposure time and the point cloud acquisition rate of each angle measured in the point cloud acquisition rate measurement according to the second embodiment. [Figure 12] 13 is an explanatory diagram showing the relationship between a virtual camera, a virtual workpiece model, and a virtual point cloud in a virtual space according to the second embodiment. FIG. [Figure 13] FIG. 11 is an explanatory diagram showing an example of a display image of an exposure time adjustment result screen in a setting mode according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] First Embodiment A first embodiment for carrying out the present invention will be described below with reference to Figs. 1 to 9. First, the configuration of a robot system 1 according to the first embodiment will be described with reference to Figs. 1 to 3. Fig. 1 is a schematic diagram of the robot system according to the first embodiment. Fig. 2 is a block diagram of the robot system according to the first embodiment. Fig. 3 is a block diagram of an image processing controller according to the first embodiment.

[0013] (Outline of the robot system configuration) The robot system 1 includes a robot 30, a camera unit 41 which is an example of an imaging section, a control panel 21 which is an example of a display section, a main controller 20, an image processing controller 40, and a robot controller 50.

[0014] The robot 30 is a manipulator. The robot 30 of this embodiment is an industrial robot, and can be used to hold (grasp) an object. The robot 30 is provided in a production line and used to manufacture articles. A camera unit 41 is attached to the robot 30. A pick-up box 10 and a placement box 12 are arranged around the robot 30.

[0015] The robot system 1 of this embodiment is a picking device that picks out a plurality of parts 11 contained in a picking box 10 one by one and transfers them to a storage box 12. That is, in this embodiment, the robot 30 holds the parts 11 in the picking box 10, and operates the robot 30 to perform a transport operation of transporting the parts 11 to the storage box 12. The parts 11 are an example of an object. Note that a plurality of picking boxes 10 may be prepared for each type of part 11. In this case, the plurality of picking boxes 10 may be sequentially replaced by a conveyor, an AGV, or the like. The storage box 12 has a partition plate provided therein, and is configured so that the position where the parts 11 are placed can be changed for each type of part 11. As a result, the storage box 12 can be used to collect parts required for assembling a product.

[0016] The work of manufacturing an article is not limited to a conveying work, and may be, for example, an assembly work of assembling a first workpiece held by the robot 30 to a second workpiece, a processing work, or a coating work. The processing work may be, for example, a cutting work, a grinding work, a polishing work, or a sealing work.

[0017] The robot 30 has a robot arm 31 and a holding unit 32 which is an example of an end effector and serves as a holding section. An end effector according to the work is attached to the robot arm 31, and in this embodiment, the holding unit 32 is attached.

[0018] In this embodiment, the robot arm 31 is a vertically articulated robot arm. The base end (fixed end) of the robot arm 31 is installed on a pedestal (not shown). A holding unit 32 and a camera unit 41 are attached to a tip (free end) which is a predetermined location of the robot arm 31. The robot arm 31 is configured to represent the holding unit 32, an object held by the holding unit 32, and the camera unit 41 in a world coordinate system Σ W In this case, the position (including the posture) can be adjusted to any position.

[0019] In this way, the holding unit 32, the object held by the holding unit 32, and the camera unit 41 are in the world coordinate system Σ W In other words, holding unit 32, the object held by holding unit 32, and camera unit 41 are capable of movement with three linear degrees of freedom along the X-axis, Y-axis, and Z-axis in the world coordinate system, and movement with three rotational degrees of freedom along the A-axis around the X-axis, the B-axis around the Y-axis, and the C-axis around the Z-axis. The X-axis, Y-axis, and Z-axis are mutually perpendicular axes.

[0020] The camera unit 41 is, for example, a stereo camera unit in which two digital cameras are arranged at a known distance, and each digital camera has a lens (not shown) and an image sensor (not shown). The image sensor is, for example, a CMOS image sensor or a CCD image sensor. The camera unit 41 also has a lighting device, and the lighting device is preferably capable of projecting pattern light so that a three-dimensional point cloud can be easily generated even for a workpiece with few irregularities on the surface or a workpiece without texture. Examples of the pattern light include a random dot pattern and a spatial code pattern.

[0021] The camera unit 41 is configured to capture an image of an object present in the field of view, i.e., the imaging area, according to setting information in which imaging conditions are set, and transmit image data obtained by this imaging, i.e., captured image data, to the image processing controller 40. The imaging conditions referred to here are conditions for setting the brightness and darkness of the captured image data. In this embodiment, the brightness of the illumination and the aperture value are constant, and the exposure time is used as a variable parameter of the imaging conditions, that is, only the exposure time is set as a variable parameter. However, as long as the brightness and darkness in the image data can be changed, the brightness of the illumination, the shutter speed (for example, the speed of the image sensor shutter), and the aperture value if the lens has an aperture function may be set as a variable parameter of the imaging conditions.

[0022] The main controller 20 is a device that comprehensively controls the entire robot system 1. The image processing controller 40, the robot controller 50, and the operation panel 21 are connected to the main controller 20.

[0023] The operation panel 21 has a touch panel display 250 and functions as a display unit capable of displaying images and an input unit capable of inputting various information by a user through operation. In this embodiment, the operation panel 21 displays a user interface (UI) image UI1 as a graphical user interface (GUI) on the touch panel display 250 in accordance with an instruction from the main controller 20. That is, the main controller 20 accepts input operations from the user through the UI image UI1 displayed on the operation panel 21. Note that the operation panel 21 has a touch panel display 250 in which a display unit and an input unit are integrated, but this is not limited thereto. For example, the display unit and the input unit may be configured separately. In this embodiment, the touch panel display 250 of the operation panel 21 is described as an example, but may be an external computer connected via a network or the like. In this case, a monitor connected to the external computer is configured as the display unit, and a mouse, a keyboard, and the like are configured as the input unit.

[0024] The holding unit 32 is a holding mechanism capable of holding the component 11, and for example, a suction type hand, a vacuum suction type hand, a clamping type hand having fingers, or the like can be used.

[0025] The robot controller 50 can control the operation of the robot 30, i.e., the operation of the robot arm 31 and the holding unit 32, according to commands from the main controller 20. This allows the position and attitude of the holding unit 32 to be freely controlled, and allows the parts 11 to be freely held (picked up). In addition, by controlling the operation of the robot arm 31, the position and attitude of the camera unit 41 can be freely controlled. In other words, the attitude of the camera unit 41 can be controlled, and the imaging direction (imaging angle AG) with respect to the workpiece can be controlled. In short, the imaging angle AG as the relative positional relationship between the camera unit 41 and the workpiece can be controlled.

[0026] The image processing controller 40 is capable of controlling the camera unit 41 according to commands from the main controller 20, and of performing image processing on image data obtained from the camera unit 41. The image processing includes processing of two-dimensional image data, generation of a three-dimensional point cloud, recognition of the position and orientation of the part 11 by matching processing, and judgment of interference between the part 11 and the holding unit 32 based on the acquired point cloud.

[0027] The matching process is a process for detecting the position and orientation of the part 11 by matching image data (captured image data) as the first image data I1 obtained by capturing an image of the part 11 with the reference image data. Specifically, the reference image data is image data of a three-dimensional model created based on three-dimensional CAD data, for example. The position and orientation of the part 11 can be calculated by performing a matching process between a point cloud generated from the image data of the model and a point cloud generated from the captured image data of the part 11.

[0028] (Outline of each controller) Next, the schematic configuration of each controller will be described. As shown in Fig. 2, the main controller 20 is configured as a computer. The main controller 20 has a CPU (Central Processing Unit) 201 as a processor.

[0029] The main controller 20 also has a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, and a HDD (Hard Disk Drive) 204, which are examples of storage units. The main controller 20 also has a recording disk drive 205, and an input / output interface (I / O) 206. The ROM 202, the RAM 203, the HDD 204, the recording disk drive 205, and the I / O 206 are connected to the CPU 201 via a bus 210.

[0030] Basic programs such as BIOS are stored in the ROM 202. The RAM 203 is a storage device that temporarily stores various data such as the results of calculations performed by the CPU 201.

[0031] The HDD 204 is a storage device that stores the results of arithmetic processing by the CPU 201 and various data acquired from the outside. A program 230 for causing the CPU 201 to execute arithmetic processing is recorded in this HDD 204. The program 230 is application software. The CPU 201 executes each process of a part of an information processing method described later, that is, a part of an image processing method, based on the program 230 recorded (stored) in the HDD 204. The recording disk drive 205 can read out various data, programs, etc. recorded in the recording disk 220.

[0032] The I / O 206 is connected to the robot controller 50, the image processing controller 40, and the operation panel 21. The CPU 201 sends commands to each of the robot controller 50, the image processing controller 40, and the operation panel 21 via the I / O 206. The CPU 201 also obtains information from each of the robot controller 50, the image processing controller 40, and the operation panel 21 via the I / O 206. Note that a storage device such as a rewritable nonvolatile memory or an external HDD may be connectable to the I / O 206. Also, a network may be connectable to the I / O 206.

[0033] The HDD 204 is also a non-transient recording medium readable by a computer. In this embodiment, the program 230 is stored in the HDD 204, but the present invention is not limited to this. The program 230 may be recorded in any recording medium as long as it is a non-transient recording medium readable by a computer. For example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a magnetic tape, or a non-volatile memory may be used as a recording medium for supplying the program 230. The optical disk may be a disk medium such as a Blu-ray disk, a DVD, or a CD. The non-volatile memory may be a storage device such as a USB memory, a memory card, a ROM, or an SSD. The program 230 may also be downloaded from a network.

[0034] 3, the image processing controller 40 is configured as a computer. The image processing controller 40 has a CPU 401 as a processor.

[0035] The image processing controller 40 also has a ROM 402, a RAM 403, and a HDD 404, which are examples of storage units. The image processing controller 40 also has a recording disk drive 405, and an input / output interface (I / O) 406. The ROM 402, the RAM 403, the HDD 404, the recording disk drive 405, and the I / O 406 are connected to the CPU 401 via a bus 410.

[0036] Basic programs such as BIOS are stored in the ROM 402. The RAM 403 is a storage device that temporarily stores various data such as the results of calculations performed by the CPU 401.

[0037] The HDD 404 is a storage device that stores the results of arithmetic processing by the CPU 401 and various data acquired from the outside. A program 430 for causing the CPU 401 to execute arithmetic processing is recorded in this HDD 404. The program 430 is application software. The CPU 401 executes each process of a part of an information processing method described later, that is, a part of an image processing method, based on the program 430 recorded (stored) in the HDD 404. The recording disk drive 405 can read out various data, programs, etc. recorded in the recording disk 420.

[0038] The main controller 20 and the camera unit 41 are connected to the I / O 406. The CPU 401 receives commands from the main controller 20 via the I / O 406. The CPU 401 controls the camera unit 41 according to the commands. The CPU 401 also acquires captured image data from the camera unit 41 via the I / O 406. The CPU 401 then executes image processing and sends information indicating the results of the image processing to the main controller 20. Note that a storage device such as a rewritable non-volatile memory or an external HDD may be connectable to the I / O 406. A network may also be connectable to the I / O 406.

[0039] The HDD 404 is also a non-transient recording medium readable by a computer. In this embodiment, the program 430 is stored in the HDD 404, but is not limited to this. The program 430 may be recorded in any recording medium as long as it is a non-transient recording medium readable by a computer. For example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a magnetic tape, or a non-volatile memory may be used as a recording medium for supplying the program 430. The optical disk is, for example, a disk medium such as a Blu-ray disk, a DVD, or a CD. The non-volatile memory is, for example, a storage device such as a USB memory, a memory card, a ROM, or an SSD. The program 430 may also be downloaded from a network.

[0040] The configurations of the main controller 20 and the image processing controller 40 have been described in detail above, but the robot controller 50 is also configured as a similar computer, so a detailed description will be omitted. That is, the robot controller 50 also has a processor such as a CPU and a storage device such as a HDD. The processor of the robot controller 50 operates based on a program to control the robot 30.

[0041] In this embodiment, the robot system 1 includes a control system 100 that functions as an information processing device (i.e., an image processing device). The control system 100 includes a main controller 20, an image processing controller 40, and a robot controller 50. That is, the control system 100 is configured with a plurality of computers. Therefore, in this embodiment, the CPU 201 executes the program 230, and the CPU 401 executes the program 430, whereby the CPU 201 and the CPU 401 function as processing units.

[0042] Although the main controller 20 and the image processing controller 40 are configured as separate computers, this is not limiting, and the main controller 20 and the image processing controller 40 may be configured as one computer. Also, the main controller 20, the image processing controller 40, and the robot controller 50 may be configured as one computer.

[0043] That is, the control system 100 may be configured with one or more computers. In the example of this embodiment, the control system 100 is configured with three computers. Furthermore, one, two, or all of the three computers may be external computers connected via a network or the like. That is, the control system as an information processing device may be configured with an external computer.

[0044] 1, a UI image UI1 is displayed on the touch panel display 250 in response to a command from the main controller 20. The CPU 201 of the main controller 20 is configured to be able to selectively execute a first mode and a second mode. The first mode is a setting mode in which an exposure time is determined and set by the image processing controller 40. The second mode is an operation mode in which an image of the part 11 is captured by the camera unit 41 at a set exposure time, the image processing controller 40 is caused to execute image processing, and the robot 30 is caused to perform a picking operation of picking the part 11 based on the result of the image processing. The setting mode is executed when the robot system 1 is installed or started up.

[0045] 1, the UI image UI1 includes a button B1 for causing the main controller 20 to execute a setting mode, and a button B2 for causing the main controller 20 to execute an operation mode. When the button B1 is operated by the user, the CPU 201 of the main controller 20 executes the setting mode, and when the button B2 is operated by the user, the CPU 201 of the main controller 20 executes the operation mode. This allows the user to select which mode to cause the control system 100 (main controller 20) to execute by operating the UI image UI1.

[0046] (Operation processing in operation mode)

[0047] Next, a case where an operation mode is selected by a user will be described. For example, when the button B2 in Fig. 1 is operated, the CPU 201 and the CPU 401 execute a process corresponding to the operation mode. Fig. 4 is a flowchart of the operation process in the operation mode according to the first embodiment.

[0048] In step S11, the CPU 201 of the main controller 20 sends a command to the robot controller 50 to operate the robot 30, and moves the camera unit 41 attached to the robot 30 to above the pick-up box .

[0049] It is preferable that the CPU 201 of the main controller 20 inputs information on the posture of the robot 30 before the robot 30 operates, and sets waypoints according to the situation to operate the robot 30 so as not to interfere with the placement box 12 or the removal box 10.

[0050] Next, in step S12, the CPU 401 of the image processing controller 40 causes the camera unit 41 to capture an image of the parts 11 in the box 10 (imaging step). The second image data I2 generated in the camera unit 41 is transferred to the image processing controller 40 via wiring, as shown in Fig. 1. The captured image data acquired in this step S12 is referred to as second image data I2, to be distinguished from the captured image data (first image data I1) acquired in step S23 described below.

[0051] The camera unit 41 turns on the illumination device before capturing an image and turns off the illumination device after capturing an image. The exposure time when the camera unit 41 captures an image is the exposure time T2 set by the exposure time setting control described later in detail. In short, the CPU 401 of the image processing controller 40 controls the camera unit 41 based on the exposure time T2 to capture an image.

[0052] Next, in step S13, the CPU 401 of the image processing controller 40 performs a matching process with the model image for the second image data I2 obtained by imaging in step S12. This detects information on the position (including the attitude) of the part 11, and outputs information on the detected detection position PA to the main controller 20.

[0053] When there are multiple components 11 in the pick-up box 10, the matching process obtains multiple pieces of detection position information and the degree of match corresponding to each piece of detection position information from the captured image data. Therefore, it is preferable that the CPU 401 of the image processing controller 40 sorts the information on the detection positions of the components 11 in descending order of degree of match and outputs the information on the detection positions of the components 11.

[0054] Next, in step S14, the CPU 201 of the main controller 20 calculates the posture of the robot 30 for orienting the holding unit 32 directly toward the component 11 to be removed, based on the information on the detection position PA of the component 11 acquired from the CPU 401. The positional relationship between the camera unit 41 and the holding unit 32 in the flange coordinate system is determined in advance by calibration or the like. When there are multiple components 11 in the removal box 10, it is preferable that the CPU 201 selects the component 11 to be removed by prioritizing the components 11 so as not to interfere with the removal box 10 or surrounding components 11, in order to acquire information on the multiple detection positions.

[0055] Next, in step S15, the CPU 201 of the main controller 20 sends a command to the robot controller 50 to operate the robot 30 to the posture calculated in step S14, and to move the holding unit 32 to a position where the holding unit 32 can hold the component 11. At that time, it is preferable to operate the robot 30 by appropriately setting waypoints so as not to interfere with the pick-up box 10 or the surrounding components 11.

[0056] Next, in step S16, the CPU 201 of the main controller 20 sends a command to the robot controller 50 to cause the holding unit 32 to hold (pick) the part 11.

[0057] Next, in step S17, the CPU 201 of the main controller 20 sends a command to the robot controller 50 to operate the robot 30 and move the part 11 to the placement box 12. If the placement box 12 has a partition, the position at which the part 11 is placed may be changed depending on the type of part 11 or the number of times it has been taken out. When moving the part 11, it is preferable to appropriately set waypoints so that the robot 30 and the part 11 do not interfere with the pick-up box 10 or the placement box 12.

[0058] Next, in step S18, the CPU 201 of the main controller 20 sends a command to the robot controller 50 to have the holding unit 32 release the part 11 and place the part 11 in the placement box 12. This completes the operation of picking one part 11 from the pick-up box 10 and placing it in the placement box 12. If work needs to be done on the next part 11, this operation mode is repeated the number of times equal to the number of parts 11 required.

[0059] (Outline of setting mode) Next, a setting mode for setting the exposure time T2 when imaging is performed by the camera unit 41 in step S12 of the above-mentioned operation mode will be described.

[0060] For example, when the part 11 is imaged and a matching process is performed, a point cloud is generated from the image data of the image and matching is performed. The accuracy of the point cloud generated at this time is affected by the brightness (contrast) of the image data of the image. That is, if the image of the part 11 in the image data is too bright, the brightness value is saturated and the point cloud cannot be acquired correctly, resulting in missing points. Also, if the image of the part 11 in the image data is too dark, the contrast is low and the point cloud cannot be acquired correctly, resulting in missing points. In particular, as described above, when the pattern light is irradiated from the illumination of the camera unit 41 onto the surface of the part 11, it is assumed that the camera is positioned in a direction significantly different from the regular reflection direction of the incident light from the illumination. In this case, if the exposure time is short, the image is captured with a low contrast of the pattern light from the illumination, and there is a problem that the point cloud cannot be acquired correctly and missing points are not acquired.

[0061] As described above, when the robot system 1 performs a picking operation of the part 11, the position and orientation of the part 11 are detected by a matching process of the part 11 from image data as the second image data I2 captured by the camera unit 41. The exposure time when capturing an image by the camera unit 41 at this time has conventionally been set as follows: That is, images of samples in which the parts 11 have been manually piled up in bulk are captured by the camera unit 41 while changing the exposure time, and an exposure time that allows a point cloud to be generated with high accuracy (matching process with high accuracy) is selected and set.

[0062] However, if the sample does not include any components 11 whose surfaces (specular reflection direction) face a large angle relative to the camera unit 41, the reflected light will be brighter and the exposure time will be set to a short time. Then, in an actual picking operation, if there are components 11 whose surfaces face a large angle relative to the camera unit 41, the matching process will not go well, and picking may not be possible or interference with the holding unit 32 may occur. Therefore, the variation in the samples created in the bulk picking operation will cause variation in the exposure time settings.

[0063] In order to solve the problems described above, it is desirable to develop a system that can generate a point cloud with high accuracy regardless of the orientation of the part 11 relative to the camera unit 41. This problem is solved by the exposure time setting control according to this embodiment, which will be described in detail later.

[0064] (Operation process in setting mode) Next, a case where the setting mode (execution of exposure time setting control) is selected by the user will be described with reference to Figs. 5 to 9. Fig. 5 is a schematic diagram for explaining the operation of the robot system in the setting mode according to the first embodiment. Fig. 6 is an explanatory diagram showing an example of a start screen of the setting mode according to the first embodiment. Fig. 7 is a flowchart showing the exposure time setting control in the setting mode according to the first embodiment. Fig. 8 is a diagram showing the relationship between the number of pixels at each angle measured by the effective pixel number measurement according to the first embodiment and the exposure time. Fig. 9 is an explanatory diagram showing an example of a display image of the exposure time adjustment result screen in the setting mode according to the first embodiment.

[0065] (Preparing for operation in setting mode) First, before executing the setting mode, one part 11 is placed in advance at the center of the box 10 as shown in Fig. 5. At this time, the relative positional relationship (imaging direction (imaging angle AG)) between the part 11 and the camera unit 41 is set so that the imaging direction (imaging angle AG) of the camera unit 41 faces in the direction of specular reflection of the incident light from the lighting.

[0066] Next, when the button B1 of the UI image UI1 shown in Fig. 5 is operated, the CPU 201 and the CPU 401 execute a process corresponding to the setting mode. First, the CPU 201 switches and displays the UI image UI2 (start screen of the setting mode) shown in Fig. 6 from the UI image UI1 shown in Fig. 5 on the operation panel 21.

[0067] Meanwhile, the CPU 201 causes the CPU 401 of the image processing controller 40 to read the CAD data. The CAD data is data necessary for obtaining the contour of the part 11 when performing matching processing, and is stored in a storage device, such as the HDD 404, inside or outside the image processing controller 40. The user selects the CAD data of the part 11 in a file format in a dialog box or the like, and operates the read button B3 in FIG. 6 to cause the CPU 401 of the image processing controller 40 to read the CAD data. The CAD data read into the CPU 401 is temporarily stored in the RAM 403. After the CPU 201 causes the CPU 401 to read the CAD data by the user's operation, when the user operates the start button B4, the CPU 201 and the CPU 401 execute the following exposure time setting process.

[0068] (Exposure Time Setting Control According to the First Embodiment) Next, the exposure time setting control as the first setting process according to the first embodiment will be described. As shown in FIG. 7, the exposure time setting control according to the first embodiment is roughly divided into an imaging process, an effective pixel number measurement process, and an exposure time setting process, which are executed in this order. The imaging process is an image acquisition process in which a plurality of first image data I1 are acquired by the main controller 20. The effective pixel number measurement process is an area calculation process in which the size of an area in which a point cloud can be generated is calculated in each of the first image data I1 acquired in the image acquisition process. The exposure time setting process is a setting process in which the imaging condition (exposure time T2) when the second image data I2 is captured by the camera unit 41 is set based on the size of the area calculated in the area calculation process.

[0069] In this embodiment, the effective pixel number measurement process is performed after the imaging process is performed for all the first image data I1. However, the present invention is not limited to this, and the effective pixel number measurement process may be performed every time imaging process is performed for one image or a predetermined number of images, and then the imaging process may be returned to for another image.

[0070] First, in step S21, the CPU 201 of the main controller 20 sends a command to the robot controller 50 to operate the robot 30, tilting the camera unit 41 at an arbitrary angle and moving the part 11 so that it is positioned at the center of the imaging area.

[0071] This step S21 is repeatedly processed by a loop LB, and each time, the camera unit 41 is changed so as to be tilted to a different angle (imaging angle AG), that is, the imaging angle AG at which the component 11 is imaged is changed. The direction in which the camera unit 41 is tilted is preferably a combination of two directions shown by the arrows in FIG. 5, that is, the camera unit 41 is moved on a hemispherical surface to comprehensively image the upper half of the component 11. In one loop LB, the angle at which the camera unit 41 is tilted (the moving angle of the imaging angle AG for one time) may be determined by the user, or may be adjusted to be coarse or fine increments according to the shape, material, and color of the component 11. In this embodiment, an example will be described in which the camera unit 41 is moved in increments of 10 degrees (see FIG. 8).

[0072] In this embodiment, the imaging angle AG is changed by moving the camera unit 41 with respect to the component 11, but the present invention is not limited to this, and any configuration may be used as long as the imaging angle AG between the camera unit 41 and the component 11 can be changed relatively to capture an image. For example, the camera unit 41 may be fixed, and the component 11 may be supported by the robot 30 to change the posture of the component 11. Also, for example, a separate actuator may be provided to change the posture of the camera unit 41 or the component 11, thereby changing the imaging angle AG relatively.

[0073] Next, in step S22, the CPU 401 of the image processing controller 40 changes the exposure time T1 used when capturing an image with the camera unit 41. This step S22 is repeatedly processed by a loop LB, and a different exposure time T1 is set each time. The range in which the exposure time T1 is changed preferably includes a range from a short exposure time that causes crushed blacks in the captured image even when the component 11 is white, to a long exposure time that causes blown-out highlights in the captured image even when the component 11 is black. In this embodiment, a change in increments of 1 ms within a range of 1 ms to 15 ms will be described as an example (see FIG. 8).

[0074] Then, in step S23, the CPU 401 of the image processing controller 40 captures an image of the part 11 in the pick-up box 10 with the camera unit 41. That is, image data in which the part 11 is captured is generated as first image data I1 by the camera unit 41, transferred to the image processing controller 40 via wiring, and stored in a storage device of the image processing controller 40, for example, the HDD 404.

[0075] In step S23, imaging is performed hundreds of times by a double loop of a loop LB that changes the imaging angle AG and a loop LA that changes the exposure time. As a result, a plurality of first image data I1 obtained by the camera unit 41 imaging one and the same part 11 at various imaging angles AG and various exposure times T1 are acquired by the CPU 401 of the image processing controller 40 and stored in the HDD 404.

[0076] In this manner, in steps S21 to S23 of the imaging process, the CPU 401 causes the camera unit 41 to capture images of the component 11 with different exposure times T1 and imaging angles AG of the camera unit 41, thereby acquiring a plurality of first image data I1.

[0077] Next, the process proceeds to effective pixel number measurement processing, and in step S31, the CPU 401 of the image processing controller 40 reads out the first image data I1 captured and stored in step S23. Step S31 is repeatedly processed by a loop LC, and each time, images with different imaging angles AG and exposure times T1 are read out in order. This loop LC is performed for all images captured with different imaging angles AG and exposure times T1.

[0078] Next, in step S32, the CPU 401 of the image processing controller 40 counts the number of pixels from which a point cloud can be generated for each of the first image data I1 read out in step S31. For example, when a random dot pattern is used as the illumination, the portion in the first image data I1 where the luminance value is not saturated and there is no blackout is counted as a pixel from which a point cloud can be generated, and the total number counted is calculated as the number of valid pixels. Also, for example, when a spatial code pattern is used as the illumination, the pixel where the contrast of the luminance value of each pixel is equal to or greater than a threshold is counted as a pixel from which a point cloud can be generated, and the total number counted is calculated as the number of valid pixels. Note that the type of pattern illumination to be used may be set by the user, and the counting method may be switched according to the setting. Also, a point cloud may actually be generated from the first image data I1, and the number of valid point clouds may be counted.

[0079] In the present embodiment, the number of pixels is measured, but it is possible that the size of each pixel may differ depending on the performance of the image sensor, or that the size of the part 11 in one image may differ. Therefore, instead of measuring the number of pixels, it is possible to measure the part where a point cloud can be generated as a region (area). That is, the number of effective pixels can be said to be the size of the region where a point cloud can be generated. Similarly, even if a point cloud is actually generated, the part where the point cloud is generated in the image is the size of the region where the point cloud can be generated. In this embodiment, the same camera unit 41 (image sensor) is used to capture images, and the camera unit 41 is moved on a hemispherical surface so that the relative distance between the camera unit 41 and the part 11 is approximately the same, so there is no particular problem with measuring the number of effective pixels.

[0080] Next, the process proceeds to exposure time setting processing, and in step S41, the CPU 401 of the image processing controller 40 selects (selects, determines, and sets) the exposure time T2 when the second image data I2 is captured by the camera unit 41 in the above-mentioned operation mode. That is, first, for each exposure time T1 changed in step S22, the sum of the effective pixel numbers calculated in step S32 is calculated for the multiple first image data I1 with different imaging angles AG changed in step S21. When the sum of the effective pixel numbers is calculated in this way, the exposure time T2 that maximizes the sum is selected and sent to the main controller 20. The CPU 201 of the main controller 20 sets the exposure time T2 as the exposure time when the camera unit 41 captures images (see S12) in the above-mentioned operation mode.

[0081] In detail, as shown in FIG. 8, when the imaging angle AG between the part 11 and the camera unit 41 is an angle where the camera unit 41 is in the direction of specular reflection with respect to the incident light from the lighting (0 degrees), the number of effective pixels from which a point cloud can be generated is large in the range where the exposure time T1 is short. However, when the exposure time T1 is long, the number of pixels that are blown out due to specular reflection from the lighting increases, and the number of effective pixels from which a point cloud can be generated decreases. On the other hand, when the imaging angle AG between the part 11 and the camera unit 41 is an angle where the camera unit 41 is in a direction that is significantly different from the specular reflection direction with respect to the incident light from the lighting (10 degrees to 40 degrees), the number of effective pixels from which a point cloud can be generated is large as the exposure time T1 is long. However, as the exposure time T1 becomes shorter, pixels become blacked out and the contrast of the lighting pattern decreases, and the number of effective pixels from which a point cloud can be generated decreases. Therefore, when the sum of the effective pixels from which a point cloud can be generated at each imaging angle AG is calculated for each exposure time T1, a peak is obtained where the value is maximum. The exposure time T1 when this sum reaches its peak, that is, when the number of effective pixels from which a point cloud can be generated reaches its maximum value, is set as the exposure time T2 in the operation mode (when picking).

[0082] In addition, the graph shown in FIG. 8 shows a graph in which the imaging angle AG is changed in increments of 10 degrees, but the imaging angle AG may be changed more finely depending on the shape of the part. In addition, in the graph shown in FIG. 8, the curves for each imaging angle AG are shown in a form of linear complementation, but as long as each imaging angle AG and the number of effective pixels can be obtained for each exposure time T1, that is, any method of displaying the graph may be used. In addition, in the graph shown in FIG. 8, an example in which the imaging angle AG is changed by rotating around one axis is shown, but it is preferable to select the exposure time T2 from a combination of the imaging angles AG changed by rotating around two axes.

[0083] Then, in step S42, the main controller 20 displays the UI image UI3 showing the graph of FIG. 8 (the number of effective pixels calculated from a plurality of first image data I1 each having a different imaging angle) and the selected exposure time T2 on the operation panel 21, as shown in FIG. 9. Here, when the registration button B5 of the UI image UI3 displayed on the operation panel 21 is pressed, the displayed exposure time T2 is overwritten as the setting value of the exposure time T2 in the operation mode. In addition, the UI image UI3 displays a graph of the number of pixels that can generate a point cloud for each exposure time, so that it is possible to confirm that the exposure time T2 has been adjusted normally. Furthermore, when the number of pixels that can generate a point cloud is extremely small due to the imaging angle AG, the user can manually change the exposure time T2 by sliding the exposure time or directly inputting a value.

[0084] After the exposure time T2 is set by executing the exposure time setting control shown in FIG. 7 as described above, the parts 11 may be placed in a random pile in the pick-up box 10 and imaged to confirm whether the exposure time T2 is appropriate. In this case, for example, pixels for which a point cloud can be generated from the captured image and pixels for which a point cloud cannot be generated are displayed on the operation panel 21. This allows the user to check the missing point cloud state. In this case, the parts 11 for which pixels for which a point cloud can be generated are selected from the parts 11 on the surface of the random pile by the user. This is because the parts 11 under the overlapping parts cannot be obtained from a point cloud due to occlusion, and therefore cannot be distinguished from missing point clouds due to exposure time. If the user determines that there are many missing point clouds, it is possible to execute the exposure time setting control again or to manually adjust the exposure time T2. It is also possible to change the exposure time T1 and the imaging angle AG for the currently randomly piled parts 11 and image them, and reacquire a plurality of first image data I1 obtained by imaging the randomly piled parts 11. That is, the effective pixel number measurement process and the exposure time setting process shown in FIG. 7 may be performed from the first image data I1 to set the exposure time T2.

[0085] (Summary of the first embodiment) As described above, according to the first embodiment, first, the main controller 20 acquires a plurality of first image data I1, each of which has a different imaging angle AG of the camera unit 41 with respect to the part 11 and a different exposure time T1 for setting the brightness of the image to be acquired. Next, the number of effective pixels capable of generating a point cloud in each of the first image data I1 is calculated from the plurality of first image data I1, and the exposure time T2 for capturing the second image data I2 with the camera unit 41 is determined and set based on the calculated number of effective pixels. That is, the exposure time T2 is set based on the plurality of first image data I1 captured while comprehensively changing the imaging angle AG and the exposure time T1 for the part 11. Thereby, when capturing and acquiring the second image data I2, the point cloud can be generated with high accuracy regardless of the posture of the part 11. Therefore, for example, in the operation mode (picking work), the accuracy of the matching process can be improved, and the occurrence of an error such as the holding unit 32 interfering with the part 11 or being unable to hold the part 11 can be reduced.

[0086] Furthermore, even if samples of parts 11 manually stacked in a random order are used as samples for setting the exposure time T2, it is possible to set the exposure time T2 without being affected by the variations. That is, by comprehensively changing the imaging angle AG and the exposure time T1 and capturing images, image data of the imaging angle and exposure time that is scarce or not obtained as samples of the parts 11 in a random order, as in the past, is not generated. Therefore, even when the parts 11 are randomly stacked as samples, there is no effect of variations in the work of manually stacking the parts 11, and the random stacking work can be shortened.

[0087] Furthermore, when setting the exposure time T2, the sum of the effective pixel counts calculated from the first image data I1 with different imaging angles AG is calculated for each exposure time T1, and the exposure time that maximizes the sum is set as the exposure time T2. This makes it possible to set the exposure time T2 that maximizes the effective pixel count regardless of the imaging angle AG when capturing the second image data I2.

[0088] In this embodiment, the exposure time T2 is set based on the total number of effective pixels. However, the present invention is not limited to this, and any exposure time that satisfies a predetermined condition may be set as the exposure time T2. In this case, the predetermined condition is preferably a condition that allows the part 11 to be matched at any angle in the operation mode (picking operation). In particular, the predetermined condition is preferably a condition that allows the part 11 to be held by the holding unit 32 and allows the part 11 to be matched with such accuracy that it does not interfere with other parts 11. In other words, since the recognition accuracy of the required position and orientation changes depending on, for example, the size and shape of the part 11 and the size and shape of the holding unit 32, it is preferable that the predetermined condition is determined so that the accuracy is satisfied.

[0089] In this embodiment, the main controller 20 displays on the operation panel 21 a graph showing the relationship between the number of effective pixels calculated from a plurality of first image data I1 each having a different imaging angle AG and the exposure time T1, and the set exposure time T2. This allows the user to check the exposure time T2 set by the main controller 20. In addition, since the user can manually change the exposure time T2, the exposure time T2 set by the main controller 20 can be corrected to the exposure time T2 set by the user.

[0090] <Second embodiment> Next, a second embodiment in which the first embodiment is partially modified will be described with reference to Figs. 10 to 13. Fig. 10 is a flow chart showing exposure time setting control in a setting mode according to the second embodiment. Fig. 11(a) is a diagram showing a case where a single exposure time is selected in the relationship between the point cloud acquisition rate of each angle measured by the point cloud acquisition rate measurement according to the second embodiment and the exposure time. Fig. 11(b) is a diagram showing a case where a plurality of exposure times are selected in the relationship between the point cloud acquisition rate of each angle measured by the point cloud acquisition rate measurement according to the second embodiment and the exposure time. Fig. 12 is an explanatory diagram showing the relationship between the virtual camera, the virtual work model, and the virtual point cloud in the virtual space according to the second embodiment. Fig. 13 is an explanatory diagram showing an example of a display image of an exposure time adjustment result screen in the setting mode according to the second embodiment. In the description of the second embodiment, the same reference numerals are used for the same parts as those in the first embodiment, and the description thereof will be omitted.

[0091] In the above-described first embodiment, the number of effective pixels from which a point cloud can be generated is measured from a plurality of first image data I1 having different imaging angles AG and exposure times T1, and the exposure time T2 is set based on the measurement. In contrast, in the present second embodiment, a point cloud is generated from a plurality of first image data I1, matched with the point cloud of a virtual model, the acquisition rate of the point cloud is calculated, and the exposure time T2 is set based on the calculation.

[0092] (Exposure Time Setting Control According to the Second Embodiment) Next, the exposure time setting control as the second setting process according to the second embodiment will be described. As shown in FIG. 10, the exposure time setting control according to the second embodiment is roughly divided into an imaging process, a point cloud acquisition rate measurement process, and an exposure time setting process, which are executed in this order. As in the first embodiment, the imaging process is an image acquisition process in which a plurality of first image data I1 are acquired in the main controller 20. The point cloud acquisition rate measurement process is a ratio calculation process in which a point cloud is generated from each of the first image data I1 acquired in the image acquisition process, and the ratio of the number of points in the generated point cloud to the point cloud that should be captured when the target object is captured by the camera unit 41 is calculated. The exposure time setting process is a setting process in which the imaging condition (exposure time T2) when the second image data I2 is captured by the camera unit 41 is set based on the acquisition rate of the point cloud that should be captured calculated in the point cloud acquisition rate measurement process, i.e., the size of the area calculated in the area calculation process.

[0093] In the second embodiment, the point cloud acquisition rate measurement process is performed after the imaging process is performed for all the first image data I1. However, the present invention is not limited to this, and the point cloud acquisition rate measurement process may be performed every time imaging process is performed for one image or a predetermined number of images, and then the imaging process may be returned to for the other images.

[0094] As shown in FIG. 10, the imaging process from step S21 to step S23 is similar to that in the first embodiment, and therefore a description thereof will be omitted.

[0095] When the process proceeds to step S51 of the effective point cloud number measurement process, the CPU 401 of the image processing controller 40 reads out the first image data I1 captured and stored in the above step S23. Step S51 is repeatedly processed by a loop LD, and each time, images with different imaging angles AG and exposure times T1 are read out in order. This loop LD is performed for all images captured with different imaging angles AG and exposure times T1.

[0096] Next, in step S52, the CPU 401 of the image processing controller 40 generates a point cloud for each of the first image data I1 read out in step S51, that is, generates a point cloud image of the first image data I1.

[0097] Next, in step S53, the CPU 401 of the image processing controller 40 matches the point cloud image of the first image data I1 generated in step S52 with a point cloud model or contour model of the part 11 generated in advance. Then, the position and orientation of the part 11 are estimated by this matching. Next, in step S54, the CPU 401 of the image processing controller 40 calculates the ratio of the number of points actually acquired to the point cloud that should be captured and generated by the camera unit 41 according to the position and orientation of the part 11 estimated in step S53. In other words, the point cloud acquisition rate is measured.

[0098] Here, the calculation of the point cloud acquisition rate will be specifically described. For example, when a part (work) having a shape as shown in Fig. 12 is imaged from the camera unit 41 located in the upward direction, only the top surface can be captured, and the side and bottom surfaces cannot be captured. Therefore, among the point cloud model V11 of the virtual part created in advance, the number of virtual point clouds VP of the top surface is set to 100%, and the ratio of the number of point clouds that can be acquired when matching with the point cloud model V11 when estimating the position and orientation of the part 11 is calculated. This makes it possible to know how many point clouds of the part 11 that can be captured from the camera unit 41 have been acquired without missing points.

[0099] In determining whether or not a point cloud included in the point cloud model V11 is a point cloud that can be captured and generated (should be captured and generated) from the camera unit 41 at a certain angle, first, a virtual camera unit V41 and a point cloud model V11 are generated (placed) in a virtual space at an estimated position and orientation. When the camera unit V41 placed in this way is connected to a certain point of the point cloud model V11, it can be determined whether or not the connection intersects with a mesh that constitutes the surface of the point cloud model V11. In other words, if a virtual line obtained by virtually extending a light ray from the virtual camera unit V41 to a certain point of the point cloud model V11 intersects only with a mesh that includes a point of the point cloud model V11, it is determined that the point can be captured and generated (should be captured and generated) from the camera unit V41. On the other hand, if a virtual line obtained by virtually extending a light ray from the virtual camera unit V41 to a certain point of the point cloud model V11 intersects with a mesh other than the mesh that includes the point of the point cloud model V11, it is determined that the point cannot be captured and generated from the camera unit V41. The points determined as being those that should be imaged in the virtual space in this way should be the same in the positional relationship between the actual part 11 and the camera unit 41, and so are regarded as a point cloud that should be imaged and generated in the part 11 by the camera unit 41. Note that the determination as to whether the points in the point cloud of the part 11 are points that can be imaged and generated by the camera unit 41 (points that should be imaged and generated) may be made using other methods.

[0100] When the calculation of the point cloud acquisition rate for each of the first image data I1 is completed in this manner, the process proceeds to step S61 of the exposure time setting process. Then, the CPU 401 of the image processing controller 40 determines whether or not there is an exposure time at which the point cloud acquisition rate falls within the threshold range at all of the imaging angles AG, based on the point cloud acquisition rate calculated in step S54.

[0101] 11(a), for example, when the threshold value of the point cloud acquisition rate is set to 98%, the point cloud acquisition rate exceeds the threshold value at all angles when the exposure time range is 6 ms to 10 ms. Therefore, when it is determined that there is an exposure time at which the point cloud acquisition rate is within the threshold range at all angles (Yes in S61), the process proceeds to step S62.

[0102] On the other hand, for example, in the case of a glossy part, if the part has a high specular reflectance and the imaging direction of the camera unit 41 is in the specular reflection direction of the incident light from the lighting, the brightness value of the captured image is saturated and the point cloud is likely to be missing. Also, for example, in the case of a part that does not easily diffuse reflection, if the imaging direction of the camera unit 41 is at an angle significantly different from the specular reflection direction of the incident light from the lighting, the brightness value of the captured image is crushed and the point cloud is likely to be missing. In such a case, as shown in FIG. 11(b), for example, there is no exposure time at which the point cloud acquisition rate is within the threshold range at all angles. If this is determined (No in S61), proceed to step S63.

[0103] When the process proceeds to step S62, the CPU 401 of the image processing controller 40 determines and selects an exposure time T2 at which the point cloud acquisition rate is within the threshold range at all angles based on the point cloud acquisition rate calculated in step S54 (i.e., sets it as an imaging condition). As shown in Fig. 11(a), when there are multiple candidates for the corresponding exposure time, the exposure time with the largest sum of the point cloud acquisition rate values ​​for each angle may be selected as the exposure time T2, or the exposure time with the median value may be selected as the exposure time T2.

[0104] Then, the process proceeds to step S64, where the main controller 20 displays on the operation panel 21 a UI image showing the graph shown in FIG. 11(a) (point cloud acquisition rate calculated from a plurality of first image data I1 each having a different imaging angle) and the selected exposure time T2. Similarly, when the registration button of the UI image displayed on the operation panel 21 is pressed, the displayed exposure time T2 is overwritten as the setting value of the exposure time T2 in the operation mode. Similarly, by displaying a graph of the point cloud acquisition rate for each exposure time on the UI image, it is possible to confirm that the exposure time T2 has been adjusted normally. Furthermore, when the point cloud acquisition rate is extremely low due to the imaging angle AG, the user can manually change the exposure time T2 by sliding the exposure time or directly inputting a value.

[0105] On the other hand, when proceeding to step S63, the CPU 401 of the image processing controller 40 selects a plurality of exposure times T2-1 and T2-2 based on the point cloud acquisition rate calculated in step S54, as shown in FIG. 11(b). At this time, the exposure times T2-1 and T2-2 are selected so that the point cloud acquisition rate of all angles satisfies the threshold, that is, for angles where the point cloud acquisition rate does not satisfy the threshold at the exposure time T2-1, the exposure time is selected so that the point cloud acquisition rate satisfies the threshold at the exposure time T2-2. In other words, for angles where the point cloud acquisition rate does not satisfy the threshold at the exposure time T2-2, the exposure time is selected so that the point cloud acquisition rate satisfies the threshold at the exposure time T2-1. Specifically, the exposure time T2-1 as the first imaging condition is an exposure time at which the point cloud acquisition rate generated from the first image data I1 where the imaging angle AG is within the first range (for example, from 0 degrees to less than 10 degrees in FIG. 11(b)) is equal to or higher than the threshold (for example, 98% or higher). In addition, the exposure time T2-2 as the second imaging condition is an exposure time at which the point cloud acquisition rate generated from the first image data I1 where the imaging angle AG is within a second range different from the first range (from 10 degrees to 40 degrees in Figure 11 (b)) is greater than or equal to a threshold value (e.g., greater than or equal to 98%).

[0106] In this way, for example, in the example of the point cloud acquisition rate shown in FIG. 11(b), two exposure times are selected, with 3 ms as the exposure time T2-1 and 13 ms as the exposure time T2-2. In the exposure time T2-1, the point cloud acquisition rate of the first image data I1 with the imaging angle AG being 0 degrees falls within the threshold range. In the exposure time T2-2, the point cloud acquisition rate of the first image data I1 with the imaging angle AG being 10 degrees to 40 degrees falls within the threshold range. This makes it within the threshold range for all angles. Note that the method of selecting the exposure times T2-1 and T2-2 is to first obtain the range of exposure times that exceed the threshold at each imaging angle AG, and select one exposure time T2-2 from the range of exposure times that exceed the threshold at the most imaging angles AG. Next, select one exposure time T2-1 from the range of exposure times that exceed the threshold at the remaining imaging angles AG that do not exceed the threshold at the exposure time T2-2.

[0107] In this way, for example, selecting two exposure times T2-1 and T2-2 means that in the imaging of step S12 in the operation mode (picking work) (see FIG. 4), two sets of second image data I2 are acquired with different exposure times. In this case, in step S13, a matching process is performed between the two sets of second image data I2 and the model image, that is, two matching processes are performed. Then, the posture of the part 11 is calculated from these two matching processes.

[0108] In the second embodiment, two exposure times are selected as described above, but the present invention is not limited to this. For example, three or more exposure times may be selected for a workpiece in which the peak of the point cloud acquisition rate differs for each imaging angle AG. In addition, in the operation mode (picking work), if there is a margin in the measurement time (time for imaging, matching processing, and posture calculation) and it is desired to acquire the point cloud more reliably, an even larger number of exposure times may be selected.

[0109] Also, in the operation mode (picking work), the second image data I2 may be captured at a plurality of exposure times set, a point cloud may be generated for each of the second image data I2, and the generated point clouds may be synthesized to be used as a point cloud image of one of the second image data I2. Furthermore, in the operation mode (picking work), HDR synthesis processing may be performed on a plurality of second image data I2 captured at a plurality of exposure times, and a point cloud may be generated for the synthesized second image data I2. In other words, the exposure time T2-1 and the exposure time T2-2 are determined as the exposure times when the second image data I2 is generated by HDR synthesis. This makes it unnecessary to perform multiple matching processes in the matching process of step S13, and makes it possible to shorten the processing time.

[0110] Then, when the selection (determination) of the multiple exposure times T2-1, T2-2 in the above step S63 is completed, the process proceeds to step S64. In this step S64, the main controller 20 displays the UI image UI4 showing the graph shown in FIG. 11(b) (point cloud acquisition rate calculated from multiple first image data I1 with different imaging angles) and the selected exposure times T2-1, T2-2 on the operation panel 21. In addition, when the registration button B6 of the UI image UI4 displayed on the operation panel 21 is pressed, the displayed exposure times T2-1, T2-2 are overwritten as the setting value of the exposure time T2 in the operation mode. Similarly, by displaying a graph of the point cloud acquisition rate for each exposure time on the UI image UI4, it is possible to confirm that the exposure times T2-1, T2-2 have been adjusted normally. Furthermore, when the point cloud acquisition rate is extremely low due to the imaging angle AG, the user can freely change the value and number of the exposure time T2 by manually sliding the exposure time or directly inputting a value. In addition, the user can freely input the threshold value, and by pressing the recalculate button B7 after changing the threshold, the optimal exposure time T2 will be reselected according to the newly set threshold, and a graph and the selected exposure time T2 will be displayed.

[0111] After the exposure time T2 is set by executing the exposure time setting control shown in FIG. 10 as described above, the parts 11 may be placed in a random pile in the pick-up box 10 and imaged to confirm whether the exposure time T2 is appropriate. At this time, the point cloud acquisition rate is calculated for each part 11 from the randomly piled parts 11 from the images captured, and displayed on the operation panel 21. This allows the user to confirm whether the point cloud acquisition rate is within the range of the threshold set by the user. Also in this case, the parts 11 for which the point cloud acquisition rate is calculated are selected from the parts 11 on the surface of the randomly piled parts 11. This is because the parts 11 under the overlapping parts 11 cannot be acquired a point cloud due to occlusion, and therefore cannot be distinguished from a loss of the point cloud due to the exposure time. If the point cloud acquisition rate is not within the range of the threshold set by the user, the exposure time setting control may be executed again, or the user may manually adjust the exposure time T2. It is also possible to change the exposure time T1 and the imaging angle AG to image the parts 11 in the current bulk piled state, and reacquire a plurality of first image data I1 capturing the images of the parts 11 in the bulk piled state. That is, the exposure time T2 may be set by performing the point cloud acquisition rate measurement process and the exposure time setting process shown in FIG. 10 from the first image data I1.

[0112] (Summary of the second embodiment) As described above, according to the second embodiment, first, the main controller 20 acquires a plurality of first image data I1, each of which has a different imaging angle AG of the camera unit 41 with respect to the part 11 and a different exposure time T1 for setting the brightness of the image to be acquired. Next, from the plurality of first image data I1, a point cloud acquisition rate is calculated for each of the first image data I1, and the exposure time T2 for capturing the second image data I2 with the camera unit 41 is set based on the calculated point cloud acquisition rate. That is, the exposure time T2 is set based on a plurality of first image data I1 captured while comprehensively changing the imaging angle AG and the exposure time T1 for the part 11. Thereby, when capturing and acquiring the second image data I2, the point cloud can be generated with high accuracy regardless of the posture of the part 11. For this reason, for example, in the operation mode (picking work), the accuracy of the matching process can be improved, and the occurrence of an error such as the holding unit 32 interfering with the part 11 or being unable to hold the part 11 can be reduced.

[0113] Furthermore, even if samples of parts 11 manually stacked in a random order are used as samples for setting the exposure time T2, it is possible to set the exposure time T2 without being affected by the variations. That is, by comprehensively changing the imaging angle AG and the exposure time T1 and capturing images, image data of the imaging angle and exposure time that is scarce or not obtained as samples of the parts 11 in a random order, as in the past, is not generated. Therefore, even when the parts 11 are randomly stacked as samples, there is no effect of variations in the work of manually stacking the parts 11, and the random stacking work can be shortened.

[0114] Furthermore, in the conventional method of setting the exposure time depending on the amount of point cloud that can be acquired from an image of the parts 11 in a randomly piled state, it is not possible to know if a point cloud is missing due to the attitude of the parts 11. However, by using a method of calculating the point cloud acquisition rate as in this embodiment, it is possible to check whether a sufficient amount of point cloud has been acquired for each relative imaging angle AG between the parts 11 and the camera unit 41.

[0115] In addition, it is determined whether or not the point cloud acquisition rate generated from the multiple first image data I1 is equal to or greater than a threshold, and the exposure time at which all of the point cloud acquisition rates are determined to be equal to or greater than the threshold is set as the exposure time T2 when the second image data I2 is captured by the camera unit 41. This makes it possible to set the optimal exposure time T2.

[0116] Furthermore, if there is no exposure time at which all of the point cloud acquisition rates are determined to be equal to or greater than the threshold, multiple exposure times T2-1 and T2-2 are determined as exposure times when the second image data I2 is captured by the camera unit 41. For example, the exposure time T2-1 is an exposure time at which the point cloud acquisition rate generated from the first image data I1 in which the imaging angle AG is within a first range is equal to or greater than the threshold. Also, for example, the exposure time T2-2 is an exposure time at which the point cloud acquisition rate generated from the first image data I1 in which the imaging angle AG is within a second range different from the first range is equal to or greater than the threshold. That is, when acquiring the second image data I2, conventionally, there were cases where the point cloud could not be acquired sufficiently due to the relative imaging angle AG between the part 11 and the camera unit 41. However, in this way, multiple exposure times T2-1 and T2-2 suitable for each posture (imaging angle AG) of the part 11 are set, and the second image data is acquired by capturing the image at each exposure time, thereby reducing the number of missing point clouds or acquiring all of the point clouds. In addition, by setting multiple exposure times in this manner, it is possible to reduce missing points even if a camera with a small dynamic range is used in the camera unit 41, making it possible to construct the camera unit 41 at low cost.

[0117] In addition, when capturing images at multiple exposure times T2-1 and T2-2, the two second image data captured by the capturing are used to perform HDR synthesis processing to generate synthesized second image data I2. By generating a point cloud for the synthesized second image data I2, it is possible to reduce missing points in the point cloud or to obtain a point cloud without missing points.

[0118] <Possibilities for other embodiments> In the first and second embodiments described above, the exposure time is adjusted (set) as an imaging condition, but the imaging condition is not limited to this, and the shutter speed, aperture, brightness of lighting, etc. may be adjusted.

[0119] In the above-described first and second embodiments, a stereo camera is used as the camera unit 41, but the present invention is not limited to this and any configuration may be used as long as the imaging unit can generate a point cloud from a captured image. For example, the imaging unit may be configured with a two-dimensional camera and a distance sensor capable of measuring the distance to an object, and may acquire the surface shape as a point cloud capable of expressing the height by using an illumination light pattern or the like.

[0120] In addition, in the exposure time setting control as the first setting process in the first embodiment, the exposure time is set by measuring the number of effective pixels at which a point cloud can be generated. In addition, in the exposure time setting control as the second setting process in the second embodiment, the exposure time is set by measuring the point cloud acquisition rate. However, it is not limited to executing only one of these exposure time setting controls, and both exposure time setting controls may be executed selectively or to increase accuracy.

[0121] In the first embodiment, a single exposure time T2 is set based on the total number of effective pixels at each imaging angle AG at each exposure time T1. However, this is not limiting, and as in the second embodiment, it is also possible to set whether or not the number of effective pixels satisfies a threshold, and further, it is also possible to set multiple exposure times by dividing the range of imaging angles at which the number of effective pixels satisfies a threshold. In addition, when multiple exposure times are set in this way, in the operation mode, it is also possible to perform HDR synthesis processing on multiple second image data I2 captured at each of those exposure times, and perform matching processing using the synthesized image.

[0122] In the second embodiment, it is determined whether or not a single exposure time can be selected in step S61, and if a single exposure time cannot be selected, multiple exposure times are selected. However, the present invention is not limited to this, and multiple exposure times may be selected from the beginning.

[0123] In the first and second embodiments, the second image data I2 is described as image data of the part 11 captured during execution of the operation mode (picking operation), but is not limited thereto. In other words, as long as the image data is captured by capturing an image of the part 11 with the camera unit 41 to obtain a point cloud, the image may be captured at any timing and for any purpose. As an example, when measuring the position and orientation of the part 11 conveyed by a belt conveyor or the like through matching processing, it is possible to capture the image with the exposure time T2 set as in the present embodiment, and the image data in such a case can be referred to as the second image data.

[0124] In the above embodiment, the robot 30 is described as a vertically articulated robot arm, but the present invention is not limited to this. The robot 30 may be, for example, a horizontally articulated robot arm, a parallel-link robot arm, an orthogonal robot, or any other type of robot arm. The present invention is also applicable to a machine that can automatically perform movements such as extension, contraction, bending, vertical movement, horizontal movement, or rotation, or a combination of these movements, based on information stored in a storage device provided in a control device.

[0125] The present invention is not limited to the above-described embodiments, and many modifications are possible within the technical concept of the present invention. Two or more of the above-described embodiments may be combined for implementation. The effects described in the embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0126] The present disclosure can also be realized by a process in which a program for realizing one or more functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for realizing one or more functions.

[0127] <Summary of this embodiment> [Configuration 1] An information processing device including a processing unit, The processing unit includes: Acquire a plurality of image data of an object captured under different relative positional relationships between the object and an image capturing unit and different image capturing conditions of the image capturing unit that set brightness of the image to be acquired; Obtaining an area from which a point cloud can be generated in each of the plurality of image data; setting the imaging conditions for imaging the object with the imaging unit based on the acquired area; 23. An information processing apparatus comprising: [Configuration 2] The imaging condition is exposure time. 2. The information processing device according to configuration 1. [Configuration 3] The processing unit determines the imaging condition that satisfies a predetermined condition based on the acquired region, and sets the determined imaging condition as the imaging condition when imaging the object with the imaging unit. 3. The information processing device according to configuration 1 or 2. [Configuration 4] The processing unit acquires a sum of the sizes of the regions acquired from the plurality of image data having different positional relationships for each of the imaging conditions, and sets the imaging condition that maximizes the sum as the imaging condition when the object is imaged by the imaging unit. 4. The information processing device according to any one of configurations 1 to 3. [Configuration 5] The imaging unit is a stereo camera. 5. The information processing device according to any one of configurations 1 to 4. [Configuration 6] The processing unit includes: Acquire image data of an object captured under the set imaging conditions; Obtaining information on the position of the object from the image data; 6. The information processing device according to any one of configurations 1 to 5. [Configuration 7] the processing unit displays, on a display unit, a graph showing a relationship between the size of the region calculated from the plurality of image data having different positional relationships and the imaging conditions, and the imaging conditions that have been set. 7. The information processing device according to any one of configurations 1 to 6. [Configuration 8] The processing unit includes: A process of setting the imaging conditions when an object is imaged by the imaging unit based on the acquired sizes of the plurality of regions can be executed as a first setting process; moreover, generating a point cloud corresponding to each of the plurality of image data from each of the plurality of image data; For each of the point clouds corresponding to each of the generated image data, a ratio of the number of points to a point cloud that would be generated when the object is imaged by the imaging unit in the positional relationship is calculated; A process of setting the imaging conditions when an object is imaged by the imaging unit based on the calculated ratio can be executed as a second setting process. 8. The information processing device according to any one of configurations 1 to 7. [Configuration 9] An information processing device including a processing unit, The processing unit includes: Acquire a plurality of image data of an object captured under different relative positional relationships between the object and an image capturing unit and different image capturing conditions of the image capturing unit that set brightness of the image to be acquired; generating a point cloud corresponding to each of the plurality of image data from each of the plurality of image data; For each of the point clouds corresponding to each of the generated image data, a ratio of the number of points to a point cloud that would be generated when the object is imaged by the imaging unit in the positional relationship is calculated; setting the imaging conditions for imaging the object with the imaging unit based on the calculated ratio; 23. An information processing apparatus comprising: [Configuration 10] A point cloud model of the object is generated in virtual space, A point cloud that can be captured and generated in the point cloud model when the point cloud model is captured by a virtual imaging unit arranged in the positional relationship is defined as the point cloud that is to be captured and generated. 10. The information processing device according to configuration 9. [Configuration 11] The imaging condition is exposure time. 11. The information processing device according to configuration 9 or 10. [Configuration 12] The processing unit includes: determining whether or not the ratio of the point clouds generated from the plurality of image data, each of which has a different positional relationship, is equal to or greater than a threshold value, for each of the imaging conditions; The imaging condition under which all of the ratios are determined to be equal to or greater than the threshold value is set as the imaging condition under which the object is imaged by the imaging unit. 12. The information processing device according to any one of configurations 9 to 11. [Configuration 13] The processing unit includes: If there is no imaging condition in which all of the ratios are determined to be equal to or greater than the threshold value, a first imaging condition under which the ratio of a point cloud generated from image data in which the positional relationship is within a first range is equal to or greater than the threshold, and a second imaging condition under which the ratio of a point cloud generated from image data in which the positional relationship is within a second range different from the first range is equal to or greater than the threshold, are set as the imaging conditions when imaging an object with the imaging unit; 13. The information processing device according to configuration 12. [Configuration 14] The processing unit includes: determining whether or not the ratio of the point clouds generated from the plurality of image data, each of which has a different positional relationship, is equal to or greater than a threshold value, for each of the imaging conditions; a first imaging condition under which the ratio of a point cloud generated from image data in which the positional relationship is within a first range is equal to or greater than the threshold, and a second imaging condition under which the ratio of a point cloud generated from image data in which the positional relationship is within a second range different from the first range is equal to or greater than the threshold, are set as the imaging conditions when imaging an object with the imaging unit; 12. The information processing device according to any one of configurations 9 to 11. [Configuration 15] The first imaging condition and the second imaging condition are set as imaging conditions for generating image data obtained by imaging an object with the imaging unit through HDR synthesis. 15. The information processing device according to configuration 13 or 14. [Configuration 16] The imaging unit is a stereo camera. 16. The information processing device according to any one of configurations 9 to 15. [Configuration 17] The processing unit includes: Acquire image data of an object captured under the set imaging conditions; Obtaining information on the position of the object from the image data; 17. The information processing device according to any one of configurations 9 to 16. [Configuration 18] the processing unit displays, on a display unit, a relationship between the ratio of the point cloud generated from the plurality of image data each having a different positional relationship and the imaging condition, and the imaging condition that has been set. 18. The information processing device according to any one of configurations 9 to 17. [Method 19] An information processing method for processing information by a processing unit, an image acquisition step in which the processing unit acquires a plurality of image data obtained by capturing an image of the object under different relative positional relationships between the object and the image capturing unit and different image capturing conditions of the image capturing unit that set brightness and darkness of the image to be acquired; a region calculation step in which the processing unit calculates a region in each of a plurality of image data from which a point cloud can be generated; A setting step in which the processing unit sets the imaging conditions when the object is imaged by the imaging unit based on the calculated area. 23. An information processing method comprising: [Method 20] An information processing method for processing information by a processing unit, an image acquisition step in which the processing unit acquires a plurality of image data obtained by capturing an image of the object under different relative positional relationships between the object and the imaging unit and different imaging conditions of the imaging unit that set brightness and darkness of the image to be acquired; a ratio calculation step in which the processing unit generates a point cloud corresponding to each of the plurality of image data from each of the image data, and calculates, for each of the generated point clouds corresponding to each of the image data, a ratio of the number of points to a point cloud that would be captured and generated when the target object is captured by the imaging unit in the positional relationship; A setting step in which the processing unit sets the imaging conditions when the object is imaged by the imaging unit based on the calculated ratio. 23. An information processing method comprising: [Configuration 21] Robots and The imaging unit; An information processing device according to any one of configurations 1 to 18, The information processing device controls the imaging unit based on the imaging conditions. A robot system comprising: [Configuration 22] The imaging unit is attached to the robot, the information processing device changes the imaging conditions by the robot when the plurality of image data are captured by the imaging unit; 22. The robot system according to configuration 21. [Configuration 23] a holding unit attached to the robot and capable of holding an object; The information processing device includes: a first mode for executing a process for setting the imaging conditions; a second mode in which an image of an object is captured by the imaging unit under the imaging conditions set in the first mode to obtain image data, and a process of holding the object is executed by controlling the robot and the holding unit based on information regarding the position of the object obtained from the image data. 23. The robot system according to configuration 21 or 22. [Method 24] A method for controlling a robot system including a robot, the imaging unit, and the information processing device according to any one of configurations 1 to 18, an imaging step in which the processing unit controls the imaging unit to perform imaging based on the imaging conditions; A method for controlling a robot system comprising: [Method 25] A method for manufacturing an article, comprising the steps of: manufacturing an article using the robot system according to aspect 23. [Configuration 26] A program for causing a computer to execute the information processing method according to Method 19 or 20. [Configuration 27] A computer-readable recording medium having the program according to configuration 26 recorded thereon. [Explanation of symbols]

[0128] 1...Robot system / 11...Part (object) / 21...Operation panel (display unit) / 30...Robot / 32...Holding unit (holding unit) / 41...Camera unit (imaging unit) / 100...Control system (information processing device) / 201...CPU (processing unit) / 204...HDD (recording medium) / 230...Program / 401...CPU (processing unit) / AG...Imaging angle (positional relationship) / I1...First image data (image data) / I2...Second image data (image data) / T1...Exposure time (imaging condition) / T2...Exposure time (imaging condition) / V11...Point cloud model

Claims

1. An information processing device including a processing unit, The processing unit includes: Acquiring a plurality of image data of an object under different relative positional relationships between the object and an imaging unit and different imaging conditions of the imaging unit that set brightness of the image to be acquired; Obtaining an area from which a point cloud can be generated in each of the plurality of image data; setting the imaging conditions for imaging the object with the imaging unit based on the acquired area; 23. An information processing apparatus comprising:

2. The imaging condition is exposure time.

2. The information processing apparatus according to claim 1,

3. The processing unit determines the imaging condition that satisfies a predetermined condition based on the acquired region, and sets the determined imaging condition as the imaging condition when imaging the object with the imaging unit.

2. The information processing apparatus according to claim 1,

4. The processing unit acquires a sum of the sizes of the regions acquired from the plurality of image data having different positional relationships for each of the imaging conditions, and sets the imaging condition that maximizes the sum as the imaging condition when the object is imaged by the imaging unit.

2. The information processing apparatus according to claim 1,

5. The imaging unit is a stereo camera.

2. The information processing apparatus according to claim 1,

6. The processing unit includes: Acquire image data of an object captured under the set imaging conditions; Obtaining information on the position of the object from the image data; 2. The information processing apparatus according to claim 1,

7. the processing unit displays, on a display unit, a graph showing a relationship between the size of the region calculated from the plurality of image data having different positional relationships and the imaging conditions, and the imaging conditions that have been set.

2. The information processing apparatus according to claim 1,

8. The processing unit includes: A process of setting the imaging conditions when an object is imaged by the imaging unit based on the acquired sizes of the plurality of regions can be executed as a first setting process, moreover, generating a point cloud corresponding to each of the plurality of image data from each of the plurality of image data; For each of the point clouds corresponding to each of the generated image data, a ratio of the number of points to a point cloud that would be generated when the object is imaged by the imaging unit in the positional relationship is calculated; A process of setting the imaging conditions when an object is imaged by the imaging unit based on the calculated ratio can be executed as a second setting process.

2. The information processing apparatus according to claim 1,

9. An information processing device including a processing unit, The processing unit includes: Acquire a plurality of image data of an object captured under different relative positional relationships between the object and an image capturing unit and different image capturing conditions of the image capturing unit that set brightness of the image to be acquired; generating a point cloud corresponding to each of the plurality of image data from each of the plurality of image data; For each of the point clouds corresponding to each of the generated image data, a ratio of the number of points to a point cloud that would be generated when the object is imaged by the imaging unit in the positional relationship is calculated; setting the imaging conditions for imaging the object with the imaging unit based on the calculated ratio; 23. An information processing apparatus comprising:

10. A point cloud model of the object is generated in virtual space, A point cloud that can be captured and generated in the point cloud model when the point cloud model is captured by a virtual imaging unit arranged in the positional relationship is defined as the point cloud that is to be captured and generated.

10. The information processing apparatus according to claim 9,

11. The imaging condition is exposure time.

10. The information processing apparatus according to claim 9,

12. The processing unit includes: determining whether or not the ratio of the point clouds generated from the plurality of image data, each of which has a different positional relationship, is equal to or greater than a threshold value, for each of the imaging conditions; The imaging condition under which all of the ratios are determined to be equal to or greater than the threshold value is set as the imaging condition under which the object is imaged by the imaging unit.

10. The information processing apparatus according to claim 9,

13. The processing unit includes: If there is no imaging condition in which all of the ratios are determined to be equal to or greater than the threshold value, a first imaging condition under which the ratio of a point cloud generated from image data in which the positional relationship is within a first range is equal to or greater than the threshold, and a second imaging condition under which the ratio of a point cloud generated from image data in which the positional relationship is within a second range different from the first range is equal to or greater than the threshold, are set as the imaging conditions when an object is imaged by the imaging unit; 13. The information processing apparatus according to claim 12.

14. The processing unit includes: determining whether or not the ratio of the point clouds generated from the plurality of image data, each of which has a different positional relationship, is equal to or greater than a threshold value, for each of the imaging conditions; a first imaging condition under which the ratio of a point cloud generated from image data in which the positional relationship is within a first range is equal to or greater than the threshold, and a second imaging condition under which the ratio of a point cloud generated from image data in which the positional relationship is within a second range different from the first range is equal to or greater than the threshold, are set as the imaging conditions when an object is imaged by the imaging unit; 10. The information processing apparatus according to claim 9,

15. The first imaging condition and the second imaging condition are set as imaging conditions when image data obtained by imaging an object with the imaging unit is generated by HDR synthesis.

14. The information processing apparatus according to claim 13,

16. The imaging unit is a stereo camera.

10. The information processing apparatus according to claim 9,

17. The processing unit includes: Acquire image data of an object captured under the set imaging conditions; Obtaining information on the position of the object from the image data; 10. The information processing apparatus according to claim 9,

18. the processing unit displays, on a display unit, a relationship between the ratio of the point cloud generated from the plurality of image data each having a different positional relationship and the imaging condition, and the imaging condition that has been set.

10. The information processing apparatus according to claim 9,

19. An information processing method for processing information by a processing unit, an image acquisition step in which the processing unit acquires a plurality of image data obtained by capturing an image of the object under different relative positional relationships between the object and the image capturing unit and different image capturing conditions of the image capturing unit that set brightness and darkness of the image to be acquired; a region calculation step in which the processing unit calculates a region in each of a plurality of image data from which a point cloud can be generated; A setting step in which the processing unit sets the imaging conditions when the object is imaged by the imaging unit based on the calculated area.

23. An information processing method comprising:

20. An information processing method for processing information by a processing unit, an image acquisition step in which the processing unit acquires a plurality of image data obtained by capturing an image of the object under different relative positional relationships between the object and the image capturing unit and different image capturing conditions of the image capturing unit that set brightness and darkness of the image to be acquired; a ratio calculation step in which the processing unit generates a point cloud corresponding to each of the plurality of image data from each of the image data, and calculates, for each of the generated point clouds corresponding to each of the image data, a ratio of the number of points to a point cloud that would be captured and generated when the target object is captured by the imaging unit in the positional relationship; A setting step in which the processing unit sets the imaging conditions when the object is imaged by the imaging unit based on the calculated ratio.

23. An information processing method comprising:

21. Robots and The imaging unit; The information processing device according to claim 1 or 9, The information processing device controls the imaging unit based on the imaging conditions. A robot system comprising:

22. The imaging unit is attached to the robot, the information processing device changes the imaging conditions by the robot when the plurality of image data are captured by the imaging unit; 22. The robotic system of claim 21.

23. a holding unit attached to the robot and capable of holding an object; The information processing device includes: a first mode for executing a process for setting the imaging conditions; a second mode in which an image of an object is captured by the imaging unit under the imaging conditions set in the first mode to obtain image data, and a process of holding the object is executed by controlling the robot and the holding unit based on information regarding the position of the object obtained from the image data.

22. The robotic system of claim 21.

24. A method for controlling a robot system including a robot, the imaging unit, and the information processing device according to claim 1 , comprising: an imaging step in which the processing unit controls the imaging unit to perform imaging based on the imaging conditions; A method for controlling a robot system comprising:

25. A method for manufacturing an article, comprising the steps of: manufacturing an article by using the robot system according to claim 23;

26. 21. A program for causing a computer to execute the information processing method according to claim 19 or 20.

27. A computer-readable recording medium having the program according to claim 26 recorded thereon.

Images