Processing device, robot control device, robot system, and program

Abstract

This processing device is provided with a detection unit. The detection unit detects a partial surface included in a surface of an object on the basis of: surface information for the surface of the object acquired on the basis of information on the distance to the object; and color information indicated by a color image or a gray scale image in which the object and the periphery of the object appear.

Description

Processing device, robot control device, robot system and program

[0001] The present disclosure relates to estimating surfaces of an object.

[0002] Patent Document 1 describes a technique for detecting a plane.

[0003] JP 2014-85940 A

[0004] A processing device, a robot control device, a robot system, and a program are disclosed. In one embodiment, the processing device includes a detection unit. The detection unit detects partial surfaces included in the surface of the object based on surface information related to the surface of the object acquired based on distance information to the object and color information indicated by a color image or grayscale image showing the object and its surroundings.

[0005] In one embodiment, the robot control device includes a control unit that controls the robot based on the detection result of the partial surface by the detection unit included in the processing device.

[0006] In one embodiment, a robot system includes the robot control device described above and a robot controlled by a control unit included in the robot control device.

[0007] In one embodiment, the program is a program for causing a computer device to function as the processing device.

[0008] In one embodiment, the program is a program for causing a computer device to function as the robot control device.

[0009] 1 is a schematic diagram showing an example of the configuration of a processing device. FIG. 1 is a schematic diagram showing an example of the configuration of a processing system. FIG. 1 is a schematic diagram showing an example of a captured image. FIG. 1 is a schematic diagram showing an example of a depth image. FIG. 2 is a flowchart showing an example of the operation of a processing device. FIG. 2 is a schematic diagram showing an example of an object mask image. FIG. 3 is a schematic diagram showing an example of a target surface mask image. FIG. 4 is a flowchart showing an example of the operation of a correction unit. FIG. 4 is a schematic diagram showing an example of a state in which a captured image is divided into a plurality of small regions. FIG. 5 is a schematic diagram for explaining an example of the operation of the correction unit. FIG. 6 is a schematic diagram for explaining an example of the operation of the correction unit. FIG. 7 is a schematic diagram showing an example of a target surface mask image including a shaped target surface region. FIG. 8 is a schematic diagram showing an example of the configuration of a robot system. FIG. 9 is a schematic diagram showing an example of the configuration of a robot control device. FIG. 10 is a schematic diagram showing an example of the configuration of a processing device. FIG. 11 is a schematic diagram showing an example of a distance transformation image. FIG. 12 is a schematic diagram showing an example of the suction position of a suction unit. FIG. 13 is a schematic diagram showing an example of the configuration of a robot control device. FIG. 14 is a schematic diagram showing an example of the configuration of a processing device. FIG. 15 is a schematic diagram showing an example of a captured image. FIG. 16 is a schematic diagram showing an example of a target surface mask image. FIG. 17 is a schematic diagram showing an example of a target surface mask image. FIG. 18 is a schematic diagram for explaining an example of the operation of an estimation unit. FIG. 19 is a schematic diagram for explaining an example of the operation of the estimation unit. 10A and 10B are schematic diagrams for explaining an example of the operation of an estimation unit, a schematic diagram showing an example of a target surface mask image, and a schematic diagram for explaining an example of the operation of an estimation unit.

[0010] Fig. 1 is a schematic diagram showing an example of the configuration of a processing apparatus 1. Fig. 2 is a schematic diagram showing an example of the configuration of a processing system 100 including the processing apparatus 1.

[0011] 2, the processing system 100 includes, for example, a processing device 1 and a sensor device 10. The sensor device 10 is, for example, a three-dimensional camera. The sensor device 10 is capable of capturing an image of a measurement space 50 in which an object 15 exists.

[0012] The sensor device 10 can generate, for example, a depth image 11 that represents a distance within the measurement space 50 and a captured image 12 that captures an object 15. The captured image 12 is, for example, a color image. The depth image 11 is generated, for example, by a stereo camera provided in the sensor device 10. The captured image 12 is generated, for example, by a color camera provided in the sensor device 10. Hereinafter, the captured image 12, which is a color image, may be referred to as the color image 12.

[0013] The depth image 11 is also called a depth image or a distance image. The depth image 11 is, for example, a grayscale image. The multiple pixels constituting the depth image 11 correspond to multiple measurement points included in the measurement space 50. The depth image 11 represents the distance from the sensor device 10 to each measurement point. The pixel value of each pixel in the depth image 11 indicates the distance from the sensor device 10 to the measurement point corresponding to that pixel. The pixel values ​​of the depth image 11 can also be considered distance information. The multiple measurement points corresponding to the multiple pixels in the depth image 11 include multiple measurement points located on the surface of the object 15 in the measurement space 50. The depth image 11 represents the distance to the object 15. Specifically, the depth image 11 represents the distance to the surface of the object 15. The depth image 11 represents the distance to each measurement point on the surface of the object 15. The depth image 11 can also be considered to represent distance information to the object 15. In the depth image 11, for example, the greater the distance to the measurement point corresponding to a pixel, the greater the pixel value of the pixel. Hereinafter, the distance represented by the pixel value of a pixel in the depth image 11 may be referred to as the measurement distance. Furthermore, a pixel corresponding to the measurement distance refers to a pixel having a pixel value representing the measurement distance. Furthermore, a measurement distance corresponding to a pixel refers to the measurement distance represented by the pixel value of the pixel.

[0014] In this example, a stereo system is used to generate the depth image 11, but a system other than the stereo system may also be used. For example, a projector system may be used to generate the depth image 11, a combination of the stereo system and the projector system may be used, or a time-of-flight (ToF) system may be used.

[0015] The multiple pixels constituting the color image 12 correspond to multiple measurement points included in the measurement space 50. For example, the number of pixels in the row and column directions of the color image 12 is the same as the number of pixels in the row and column directions of the depth image 11, respectively. Pixels at the same pixel position between the color image 12 and the depth image 11 correspond to the same measurement point in the measurement space 50. The pixel values ​​of the color image 12 include, for example, an R component (red component), a G component (green component), and a B component (blue component). The pixel values ​​of the color image 12 can also be said to be color information of the pixels, and the color image 12 is also referred to as, for example, an RGB image. The color image 12 represents the color of the object 15. Specifically, the color image 12 represents the color of each measurement point on the surface of the object 15. The pixel position of a pixel in the color image 12, whose pixel value indicates the color of a certain measurement point, is the same as the pixel position of a pixel in the depth image 11, whose pixel value indicates the distance to the certain measurement point.

[0016] The captured image 12 may be a grayscale image that captures the object 15. A captured image 12 that is a grayscale image may be referred to as a grayscale image 12. The following description of the embodiment will focus on the case where the captured image 12 is a color image, but the following description of the case where a color image 12 is used can also be applied to the case where a grayscale image 12 is used.

[0017] The processing device 1 estimates the object surface of the object 15 based on the depth image 11. Then, the processing device 1 corrects the estimation result of the object surface of the object 15 based on a captured image 12 (e.g., a color image 12 or a grayscale image 12) in which the object 15 appears. The processing device 1 acquires, for example, depth image data representing the depth image 11 and color image data representing the color image 12 from the sensor device 10. The depth image data includes pixel values ​​(in other words, distance information) of each pixel constituting the depth image 11. The color image data includes pixel values ​​(in other words, color information of each pixel) of each pixel constituting the color image 12. It can also be said that the processing device 1 estimates the object surface of the object 15 based on the depth image data and corrects the estimation result based on the color image data.

[0018] 1, the processing device 1 includes, for example, a control unit 2, a storage unit 3, an interface 4, and an interface 5. The processing device 1 can also be considered, for example, a processing circuit. The processing device 1 can also be considered, for example, a computer device.

[0019] The interface 4 is capable of communicating with the sensor device 10. The interface 4 may communicate with the sensor device 10 via wired or wireless communication. The interface 4 may also be referred to as, for example, an interface circuit, a communication unit, or a communication circuit. The interface 4 acquires the depth image 11 and the color image 12 generated by the sensor device 10. The interface 4 acquires the depth image data and the color image data from the sensor device 10, thereby acquiring the depth image 11 and the color image 12 generated by the sensor device 10. The depth image 11 and the color image 12 acquired by the interface 4 are input to the control unit 2. In other words, the depth image data and the color image data received by the interface 4 are input to the control unit 2.

[0020] The interface 5 is capable of communicating with devices (also referred to as external devices) external to the processing device 1 other than the sensor device 10. The interface 5 may communicate with the external device via wired or wireless communication. The interface 5 may also be referred to as, for example, an interface circuit, a communication unit, or a communication circuit. A signal including data and the like that is received by the interface 5 from the external device is input to the control unit 2.

[0021] The control unit 2 can generally manage the operation of the processing device 1 by controlling the other components of the processing device 1. The control unit 2 can also be referred to as a control circuit, for example. The control unit 2 includes at least one processor to provide control and processing power for performing various functions, as described in more detail below.

[0022] According to various embodiments, the at least one processor may be implemented as a single integrated circuit (IC) or as multiple communicatively connected integrated circuits ICs and / or discrete circuits. The at least one processor may be implemented according to various known techniques.

[0023] In one embodiment, a processor includes one or more circuits or units configured to perform one or more data computational procedures or processes, for example, by executing instructions stored in associated memory. In other embodiments, a processor may be firmware (e.g., discrete logic components) configured to perform one or more data computational procedures or processes.

[0024] According to various embodiments, the processor may include one or more processors, controllers, microprocessors, microcontrollers, application specific integrated circuits (ASICs), digital signal processors, programmable logic devices, field programmable gate arrays, or any combination of these devices or configurations, or other known devices and configurations, to perform the functions described below.

[0025] The control unit 2 may include, for example, a CPU (Central Processing Unit) as a processor. The storage unit 3 may include a non-transitory recording medium readable by the CPU of the control unit 2, such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage unit 3 stores, for example, a program 30 for controlling the processing device 1. Various functions of the control unit 2 are realized, for example, by the CPU of the control unit 2 executing the program 30 in the storage unit 3.

[0026] The configuration of the control unit 2 is not limited to the above example. For example, the control unit 2 may include multiple CPUs. The control unit 2 may also include at least one DSP (Digital Signal Processor). All or some of the functions of the control unit 2 may be realized by a hardware circuit that does not require software to realize the function. The storage unit 3 may also include a computer-readable non-transitory recording medium other than ROM and RAM. The storage unit 3 may also include, for example, a small hard disk drive or SSD (Solid State Drive).

[0027] The control unit 2 includes, for example, a detection unit 25 as a functional block. The detection unit 25 is realized by the CPU of the control unit 2 executing a program 30 in the storage unit 3. The detection unit 25 detects a partial surface included in the surface of the object 15 as a target surface based on surface information related to the surface of the object 15 acquired based on distance information to the object 15 indicated by the depth image 11 and color information indicated by the captured image 12 (e.g., the color image 12 or the grayscale image 12). This makes it possible to appropriately detect the target surface, i.e., the partial surface included in the surface of the object 15. In the present disclosure, the color information may include shading information between white and black indicated by the grayscale image.

[0028] The detection unit 25 includes, for example, an estimation unit 20 and a correction unit 21. The estimation unit 20 acquires surface information related to the surface of the object 15 based on distance information indicated by the depth image 11, and estimates the object surface (in other words, a partial surface) based on the acquired surface information. The correction unit 21 corrects the estimation result by the estimation unit 20 based on color information indicated by the captured image 12 (e.g., color image 12 or grayscale image 12), and the corrected estimation result is used as the detection result of the object surface. Hereinafter, the term "surface information" simply refers to surface information related to the surface of the object 15.

[0029] The target surface (in other words, a partial surface) of the object 15 estimated by the estimation unit 20 is, for example, a plane included in the area of ​​the surface of the object 15 that can be photographed by the sensor device 10. The target surface of the object 15 can also be said to be a plane included in the area of ​​the surface of the object 15 that can be seen from the sensor device 10. The target surface of the object 15 can also be said to be a plane included in the area of ​​the surface of the object 15 that appears in the color image 12. Hereinafter, the area of ​​the surface of the object 15 that can be photographed by the sensor device 10 may be referred to as the object measurement surface. Furthermore, simply referring to the target surface means the target surface of the estimated target of the object 15.

[0030] Note that some or all of the functions of the detection unit 25 may be implemented by a hardware circuit that does not require software to implement the function. The same applies to the estimation unit 20 and the correction unit 21.

[0031] <Examples of Depth Image and Color Image> Fig. 3 is a schematic diagram showing an example of a color image 12 (in other words, a captured image 12). In Fig. 3, for convenience of illustration, the color image 12 is shown roughly in a line diagram. The color image 12 shows an object 15 including an object surface 15a. The color image 12 also shows the background of the object 15. The color image 12 shows the object 15 and the surroundings of the object 15. The color image 12 shows the object surface 15a and the surroundings of the object surface 15a. The color image 12 includes an object region 120 that is an image of the object 15. The object region 120 can also be said to be an image in which only the object 15 is shown. The object region 120 can also be said to be an object image.

[0032] In this example, the object 15 is, for example, a roughly T-shaped metal part. The object 15 is placed on, for example, a tray. The color image 12 also includes a tray region 125 (also referred to as a background region 125), which is an image of the tray as the background. In the color image 12, the portion other than the object region 120 is the tray region 125. The tray region 125 can be called either a tray image or a background image. Note that the actual color image 12 may also include a shadow appearing on the object 15. The actual color image 12 may also include a shadow appearing on the tray.

[0033] The object region 120 includes an object surface region 121, which is an image of the object surface 15a. The object surface region 121 can also be called an object surface image. In FIG. 3, the object surface region 121 is indicated by diagonal lines. In this example, the object surface 15a is, for example, the plane with the largest area as seen from the sensor device 10 on the object measurement surface. In other words, the object surface 15a is the plane with the largest image area on the object measurement surface shown in the captured image 12.

[0034] The color information represented by the color image 12 indicates a plurality of gradations, for example, 256 gradations. The color information represented by the color image 12 includes color tone differences. That is, the color image 12 indicates color shading and brightness depending on the shooting environment of the color image 12, etc. The color information represented by the color image 12 includes color distribution. Similarly, the color information represented by the grayscale image 12 indicates a plurality of gradations, for example, 256 gradations. Furthermore, the color information represented by the grayscale image 12 includes color tone differences. Furthermore, the color information represented by the grayscale image 12 includes color distribution.

[0035] The color information indicated by the captured image 12 (e.g., color image 12 or grayscale image 12) includes the hue difference and color distribution of the object 15. The color information indicated by the captured image 12 also includes color information within the object surface 15a (also referred to as first color information) and color information around the object surface 15a (also referred to as second color information). The first color information includes the hue difference and color distribution within the object surface 15a.

[0036] The correction unit 21 may correct the estimation result of the object surface 15a by the estimation unit 20 based on the first color information and the second color information indicated by the captured image 12 (e.g., the color image 12 or the grayscale image 12). The correction unit 21 may also correct the estimation result of the object surface 15a based on the hue difference or color distribution of the object 15 indicated by the captured image 12. The correction unit 21 may also correct the estimation result of the object surface 15a based on the hue difference or color distribution within the object surface 15a indicated by the captured image 12.

[0037] Fig. 4 is a schematic diagram showing an example of the depth image 11. The actual depth image 11 is a grayscale image, but for convenience of illustration, Fig. 4 shows the depth image 11 with white parts, hatched parts, and black parts.

[0038] Each pixel in the white portion included in the depth image 11 corresponds to a measurement point that is relatively close to the sensor device 10. The pixel value of each pixel in the white portion included in the depth image 11 indicates a relatively small value. The white portion included in the depth image 11 roughly corresponds to the target surface 15a of the target object 15.

[0039] Each pixel in the shaded portion of the depth image 11 corresponds to a measurement point that is relatively far from the sensor device 10. The pixel value of each pixel in the shaded portion of the depth image 11 indicates a relatively large value. The shaded portion of the depth image 11 generally corresponds to the tray on which the object 15 is placed (in other words, the background).

[0040] Each pixel in the black portion included in the depth image 11 corresponds to a measurement point where the sensor device 10 was unable to measure the distance due to the influence of the shadows cast by the object 15 and the tray, etc. The pixel value of each pixel in the black portion included in the depth image 11 is zero.

[0041] <Example of Operation of Control Unit> Fig. 5 is a flowchart showing an example of the operation of the estimation unit 20 and the correction unit 21. As shown in Fig. 5, in step s1, the estimation unit 20 generates a mask image 200 (also referred to as object mask image 200) of the object 15 based on the color image 12 (in other words, the captured image 12). The object mask image 200 represents the position and shape of the object 15 in the measurement space 50. It can also be said that the object mask image 200 represents the position and shape of the object 15 as seen from the sensor device 10. The object mask image 200 is, for example, a binary image.

[0042] 6 is a schematic diagram showing an example of an object mask image 200. The object mask image 200 is composed of an object region 201 representing the object 15 and a remaining region 202 (also referred to as the other region 202). The pixel value of each pixel in the object region 201 is, for example, "1," and the pixel value of each pixel in the other region 202 is, for example, "0."

[0043] The number of pixels in the row and column directions of the object mask image 200 is equal to, for example, the number of pixels in the row and column directions of the color image 12. Also, the number of pixels in the row and column directions of the object mask image 200 is equal to, for example, the number of pixels in the row and column directions of the depth image 11.

[0044] The multiple pixels that make up the object mask image 200 correspond to the multiple measurement points in the measurement space 50. Between the object mask image 200 and the color image 12, pixels at the same pixel position correspond to the same measurement point in the measurement space 50. Also, between the object mask image 200 and the depth image 11, pixels at the same pixel position correspond to the same measurement point in the measurement space 50.

[0045] The multiple pixels that make up the object region 201 correspond to the multiple measurement points located on the object measurement surface. The position of the object region 201 in the object mask image 200 represents the position of the object 15 in the measurement space 50, and the shape of the object region 201 represents the shape of the object 15. The position and shape of the object region 201 in the object mask image 200 are the same as the position and shape of the object region 120 in the color image 12. In the object mask image 200, the pixel value of each pixel in the portion representing the object 15 (i.e., the object region 201) is "1," and the pixel value of each pixel in the portion representing other than the object 15 (other region 202) is "0."

[0046] Here, in the depth image 11, the region corresponding to the object 15 (in other words, the partial image corresponding to the object 15) is called the object-equivalent region. The object-equivalent region represents the distance to the object 15. The object-equivalent region can also be said to be an object distance image representing the distance to the object. The multiple pixels that make up the object-equivalent region correspond to multiple measurement points located on the object measurement surface. The object-equivalent region represents the distance to each measurement point located on the object measurement surface. The position and shape of the object region 201 in the object mask image 200 are the same as the position and shape of the object-equivalent region in the depth image 11.

[0047] The estimation unit 20 can generate the object mask image 200 using, for example, machine learning with the color image 12 as input data. The estimation unit 20 has, for example, a neural network that realizes instance segmentation, and generates the object mask image 200 using the neural network. For example, Mask Scoring R-CNN is adopted as the neural network that realizes instance segmentation. R-CNN is an abbreviation for Region Based Convolutional Neural Networks.

[0048] The estimation unit 20 inputs the color image 12 to the input layer of the trained neural network. The trained neural network recognizes the object 15 appearing in the color image 12 and outputs an object mask image 200 representing the object 15 from the output layer. The trained neural network has been trained so that it can generate and output the object mask image 200 based on the color image 12.

[0049] Once the object mask image 200 is generated in step s1, step s2 is executed. In step s2, the estimation unit 20 uses the object mask image 200 to perform a first mask process to extract an object-corresponding region from the depth image 11. In the first mask process, the estimation unit 20 identifies, in the depth image 11, a plurality of pixels at the same pixel positions as the pixel positions of a plurality of pixels constituting the object region 201 in the object mask image 200. That is, the estimation unit 20 identifies, in the depth image 11, a plurality of pixels at the same pixel positions as the pixel positions of a plurality of pixels having a pixel value of "1" in the object mask image 200. Then, the estimation unit 20 extracts, from the depth image 11, a region consisting of the identified plurality of pixels as the object-corresponding region.

[0050] Next, in step s3, the estimation unit 20 generates a point cloud representing the surface of the object 15 as surface information based on the extracted object-equivalent region. The point cloud representing the surface of the object 15 (also referred to as an object point cloud) is composed of, for example, a plurality of measurement points corresponding to a plurality of pixels constituting the object-equivalent region. The object point cloud is expressed by the three-dimensional position coordinates of the plurality of measurement points constituting the object point cloud within the measurement space 50. The three-dimensional position coordinates of the measurement points are expressed, for example, by position coordinates in a three-dimensional Cartesian coordinate system (also referred to as a camera coordinate system) set in a three-dimensional camera serving as the sensor device 10. The estimation unit 20 generates point cloud data including the three-dimensional position coordinates of the plurality of measurement points constituting the object point cloud based on the performance, etc., of the three-dimensional camera serving as the sensor device 10.

[0051] In this way, the estimation unit 20 generates an object point cloud, i.e., a point cloud representing the surface of the object 15, based on the depth image 11. Hereinafter, each of the multiple measurement points constituting the object point cloud may be referred to as an object point.

[0052] After step s3, in step s4, the estimation unit 20 sets multiple candidate surfaces in the camera coordinate system based on the generated object point cloud. The candidate surfaces are surfaces that are candidates for surfaces (also called specific surfaces) that are in the same plane as the object surface. The specific surfaces can also be said to be surfaces that include the object surface, for example. The candidate surfaces are, for example, flat surfaces. After step s4, in step s5, the estimation unit 20 estimates a candidate surface that is in the same plane as the object surface, i.e., a specific surface, from the multiple candidate surfaces based on the comparison result between the distance from the candidate surface to the object point and a threshold value. The processing of steps s3 and s4 is realized, for example, using RANSAC (random sample consensus), which is a plane estimation algorithm.

[0053] In step s4, the estimation unit 20 randomly selects three object points from the object points constituting the object point cloud. The estimation unit 20 then determines a plane passing through the selected three object points as a candidate plane. The estimation unit 20 repeats this process to set multiple candidate planes in the camera coordinate system.

[0054] In step s4, once a candidate surface has been set, the estimation unit 20 calculates the number of object points located around the candidate surface (also referred to as peripheral object points) whose distance from the set candidate surface is equal to or less than a threshold value. The distance from the candidate surface to an object point is the length of a perpendicular line drawn from the object point to the candidate surface. Hereinafter, the threshold value used to compare the distance of an object point from the candidate surface is referred to as the specific surface estimation threshold value.

[0055] In step s4, the estimation unit 20 calculates the number of peripheral object points for each of the multiple candidate surfaces. In step s5, the estimation unit 20 identifies the candidate surface with the largest number of peripheral object points from the multiple candidate surfaces. The candidate surface with the largest number of peripheral object points is likely to be on the same plane as the plane with the largest area (i.e., target surface 15a) as seen from the sensor device 10 on the object measurement surface. Therefore, the estimation unit 20 estimates the candidate surface with the largest number of peripheral object points as the specific surface that is on the same plane as target surface 15a.

[0056] In step s4, the estimation unit 20 may randomly select four or more object points from the plurality of object points constituting the object point cloud. In this case, the estimation unit 20 may identify a plane that has the smallest sum of distances from the selected four or more object points using the least squares method, and set the identified plane as one candidate plane. Alternatively, the number of object points whose distance from the candidate plane is less than a threshold value may be set as the number of peripheral object points.

[0057] Once the specific surface is estimated in step s5, step s6 is executed. In step s6, the estimation unit 20 estimates multiple object points representing the target surface 15a included in the target point group based on the comparison result between the distance from the estimated specific surface to the object point and a threshold value. The estimation unit 20 estimates multiple object points whose distance from the specific surface is equal to or less than the threshold value as multiple object points representing the target surface 15a. Hereinafter, each of the multiple object points representing the target surface 15a may be referred to as an object surface point. Furthermore, the threshold value used in step s6, i.e., the threshold value used when estimating the object surface points, may be referred to as an object surface point estimation threshold value. The multiple object surface points representing the target surface 15a can also be considered multiple measurement points on the target surface 15a. The object surface point estimation threshold value may be the same as or different from the specific surface estimation threshold value.

[0058] In this example, the specific surface is estimated based on RANSAC, but the specific surface may be estimated based on a plane estimation algorithm other than RANSAC. For example, the Randomized Hough Transform (RHT) or Region Growing Segmentation may be used to estimate the specific surface. Furthermore, multiple object points whose distance from the specific surface is less than a threshold may be estimated as multiple object surface points representing the target surface 15a.

[0059] After step s6, in step s7, the estimation unit 20 generates a mask image of the target surface 15a (also referred to as a target surface mask image) based on the estimated plurality of target surface points and the depth image 11. The target surface mask image represents the position and shape of the target surface 15a in the measurement space 50. It can also be said that the target surface mask image represents the position and shape of the target surface 15a as seen from the sensor device 10. The target surface mask image is, for example, a binary image. The estimation unit 20 generates, for example, a target surface mask image as an estimation result of the target surface 15a (also referred to as a target surface estimation result).

[0060] In step s7, the estimation unit 20 identifies a plurality of pixels corresponding to the estimated plurality of target surface points (in other words, a plurality of measurement points) from a plurality of pixels constituting the depth image 11. The region consisting of the identified plurality of pixels corresponds to the target surface 15a. The pixel values ​​of the identified plurality of pixels represent the distances to the plurality of measurement points on the target surface 15a, respectively. The estimation unit 20 resets the pixel values ​​of the identified plurality of pixels to "1" in the depth image 11 and resets the pixel values ​​of the other pixels to "0", resulting in an object surface mask image.

[0061] 7 is a schematic diagram showing an example of a target surface mask image 250. The target surface mask image 250 is composed of a first region 251 (white region in FIG. 7 ) where pixel values ​​indicate "1" and a second region 252 (black region in FIG. 7 ) where pixel values ​​indicate "0". The multiple pixels that make up the first region 251 correspond to the multiple target surface points estimated in step s6. The first region 251 becomes a target surface region 255 that represents the target surface 15a. The position of the target surface region 255 in the target surface mask image 250 represents the position of the target surface 15a in the measurement space 50, and the shape of the target surface region 255 represents the shape of the target surface 15a.

[0062] As described above, when the estimation unit 20 estimates the target surface 15 a of the object 15 based on the depth image 11, an error may be included in the result of the target surface estimation by the estimation unit 20 due to a measurement error in the distance to each measurement point in the sensor device 10. Therefore, as shown in Fig. 7 , the shape of the target surface region 255 included in the target surface mask image 250 generated based on the depth image 11 may differ from the actual shape of the target surface 15 a of the object 15 (the shaded portion in Fig. 3 ).

[0063] Therefore, in this example, the correction unit 21 corrects the object surface estimation result by the estimation unit 20 based on the captured image 12. This reduces the error in the object surface estimation result by the estimation unit 20, and the shape of the object surface region 255 included in the object surface mask image 250 comes closer to the actual shape of the object surface 15 a. An example of the operation of the correction unit 21 will be described below.

[0064] In step s11 of FIG. 5 , the correction unit 21 performs a segmentation process to divide the color image 12 (in other words, the captured image 12) into multiple small regions such that the boundaries between the small regions are aligned along edges. In the segmentation process, the correction unit 21 divides the color image 12 into k small regions (k is an integer equal to or greater than 2) using, for example, Simple Linear Iterative Clustering (SLIC). The small regions are called, for example, clusters or superpixels. Step s11 will be described in detail below. Note that the edge may be an edge within an object formed by the intersection of multiple faces within the object, or an edge indicating the boundary between the object and the background.

[0065] 8 is a flowchart showing an example of step s11 in detail. In this example, an x-y Cartesian coordinate system is set as the image coordinate system for images such as the color image 12 and the depth image 11. The pixel position of a pixel in an image is represented by a two-dimensional position in the image coordinate system. That is, the pixel position of a pixel in an image is represented by an x-coordinate and a y-coordinate in the image coordinate system. For example, the x-direction of the image coordinate system is set to the row direction of the image (in other words, the left-right direction), and the y-direction of the image coordinate system is set to the column direction of the image (in other words, the up-down direction).

[0066] 8 , in step s11, step s110 is first executed. In step s110, the modifying unit 21 color-converts the R, G, and B components that make up the pixel value of each pixel of the color image 12 into lightness L, chromaticity a, and chromaticity b. Hereinafter, the color image 12 in which the R, G, and B components of each pixel have been converted into lightness L, chromaticity a, and chromaticity b may be referred to as the color-converted color image 12 or the Lab image 12. Furthermore, the original color image 12, i.e., the color image 12 before color conversion, may be referred to as the RGB image 12.

[0067] After step s110, in step s111, the modifying unit 21 initially sets the center of gravity positions of k small regions in the image coordinate system for the color-converted color image 12. For example, the modifying unit 21 sets the k center of gravity positions in a matrix at equal intervals. k may be set to, for example, 80 or more, 90 or more, or 100 or more. Hereinafter, one center of gravity position of the object of explanation will be referred to as a center of gravity position of interest.

[0068] Next, in step s112, the modification unit 21 adjusts the k initially set centroid positions based on the color gradient. When adjusting the target centroid position, the modification unit 21 sets (3 × 3) pixels consisting of the pixel at the target centroid position and its surrounding eight pixels (i.e., nine pixels) as adjustment pixels to be used for adjusting the target centroid position. Then, the modification unit 21 individually calculates the color gradient (in other words, the Lab gradient) for each of the nine adjustment pixels. The modification unit 21 moves the target centroid position to the pixel position of the adjustment pixel with the smallest calculated color gradient among the nine adjustment pixels. In other words, the modification unit 21 sets the pixel position of the adjustment pixel with the smallest color change from the surrounding pixels among the nine adjustment pixels for the target centroid position as the target centroid position after adjustment. In this way, the modification unit 21 individually adjusts each of the k centroid positions in the image coordinate system.

[0069] After step s112, step s113 is executed. In step s113, the modifying unit 21 arranges multiple pixels constituting the color-converted color image 12 in a five-dimensional space (also referred to as a five-dimensional coordinate system) with axes representing lightness L, chromaticity a, chromaticity b, x-coordinate, and y-coordinate. The modifying unit 21 also sets the positions in the five-dimensional space of k pixels present at the k adjusted centroid positions in the image coordinate system as centroid positions of k small regions in the five-dimensional space. Hereinafter, one pixel to be described among the multiple pixels arranged in the five-dimensional space will be referred to as a pixel of interest.

[0070] Next, the modifying unit 21 provisionally determines, in the five-dimensional space, the small region having the centroid position closest to the pixel of interest among the k centroid positions, as the small region to which the pixel of interest belongs. Similarly, the modifying unit 21 provisionally determines, for each pixel in the five-dimensional space, the small region to which the pixel belongs. In this way, the ranges of the k small regions in the five-dimensional space are provisionally determined.

[0071] Next, in step s114, the correction unit 21 recalculates the center of gravity of each of the k small regions in the five-dimensional space. For example, the correction unit 21 determines the average value of the positions of multiple pixels belonging to each small region in the five-dimensional space as the new center of gravity of the small region.

[0072] Thereafter, the correction unit 21 executes step s113 again, and provisionally re-determines, from among the k re-calculated center positions, the small region having the center position closest to the pixel of interest as the small region to which the pixel of interest belongs. If the pixels belonging to a small region change, the shape of the small region also changes. In the same manner, the correction unit 21 provisionally re-determines, for each pixel in the five-dimensional space, the small region to which the pixel belongs. Thereafter, in step s114, the correction unit 21 re-determines the center positions of the k small regions in the five-dimensional space. Thereafter, the estimation unit 20 repeatedly executes steps s113 and s114.

[0073] For example, when there is at least one small region among the k small regions in which the center of gravity position calculated in step s114 has not changed significantly from the center of gravity position calculated in the previous step s114, the correction unit 21 may end the processing shown in Fig. 8. Alternatively, for example, the correction unit 21 may end the processing shown in Fig. 8 when step s114 has been executed a predetermined number of times.

[0074] The range of the small region at the time when the process shown in Fig. 8 is completed becomes the range of the final small region. In other words, when the process shown in Fig. 8 is completed, the small region to which each of the multiple pixels constituting the color-converted color image 12 belongs is finally determined. In other words, for each of the k small regions, the pixels of the color-converted color image 12 that belong to that small region are finally determined. As a result, the color-converted color image 12 (in other words, the Lab image 12) is divided into k small regions so that the boundaries between the small regions are along the edges.

[0075] The correction unit 21 divides the original color image 12 into k small regions by superimposing the contours of the k small regions of the color-converted color image 12 onto the original color image 12 (in other words, the RGB image 12).

[0076] Fig. 9 is a schematic diagram showing an example of how the color image 12 shown in Fig. 3 is divided into a plurality of small regions 300. In Fig. 9, the outlines of the small regions 300 are indicated by thin lines. As shown in Fig. 9, the color image 12 is divided into a plurality of small regions 300 so that the boundaries between the small regions 300 run along the edges. Dividing the color image 12 into a plurality of small regions 300 can also be said to mean dividing each of the object region 120 and the tray region 125 included in the color image 12 into a plurality of small regions 300 so that the boundaries between the small regions 300 run along the edges.

[0077] As shown in Fig. 5, the correction unit 21 performs a segmentation process in step s11 and then performs step s12. In step s12, the correction unit 21 performs a second mask process using the object mask image 200 generated in step s1 to extract an object region 120 (also referred to as a divided object region 120) divided into a plurality of small regions from the color image 12 (also referred to as a divided color image 12) divided into a plurality of small regions as shown in Fig. 9. In other words, the second mask process extracts the object region 120, in which the outlines of the plurality of small regions are indicated, from the color image 12 in which the outlines of the plurality of small regions are indicated.

[0078] In the second mask processing, the correction unit 21 identifies a plurality of pixels in the divided color image 12 that are located at the same pixel positions as the pixel positions of the plurality of pixels that make up the object region 201 in the object mask image 200. That is, the correction unit 21 identifies a plurality of pixels in the divided color image 12 that are located at the same pixel positions as the pixel positions of the plurality of pixels that have a pixel value of "1" in the object mask image 200. Then, the correction unit 21 extracts an area consisting of the identified plurality of pixels from the divided color image 12 as the divided object region 120.

[0079] In the process consisting of steps s11 and s12, the correction unit 21 divides the region of the object 15 included in the captured image 12 (i.e., the object region 120) into multiple small regions with boundaries along the edges of the object 15 based on the color information indicated in the captured image 12. It can also be said that the correction unit 21 divides the object region 120 into multiple small regions based on first color information within the object surface 15a indicated in the captured image 12 and second color information around the object surface 15a. It can also be said that the correction unit 21 divides the object region 120 into multiple small regions based on the hue difference or color distribution of the object 15 indicated in the captured image 12. It can also be said that the correction unit 21 divides the object region 120 into multiple small regions based on the hue difference or color distribution within the object surface 15a indicated in the captured image 12.

[0080] Once the segmentation target region 120 is extracted in step s12, step s13 is executed. In step s13, the correction unit 21 shapes the target surface region 255 of the target surface mask image 250 based on the multiple small regions that make up the segmentation target region 120. That is, the shape of the target surface region 255 is adjusted. For example, for each of the multiple small regions that make up the segmentation target region 120, the correction unit 21 calculates the occupancy rate of the target surface region 255 within the outline of the small region when the outline of the small region is superimposed on the target surface mask image 250. The correction unit 21 then shapes the target surface region 255 based on the occupancy rates calculated for the multiple small regions. Hereinafter, in the description of step s13, a small region refers to a small region that makes up the segmentation target region 120. Furthermore, hereinafter, one of the multiple small regions that make up the segmentation target region 120 that is the subject of the description will be referred to as a small region of interest.

[0081] When calculating the occupancy rate of a small region of interest, the correction unit 21 first superimposes the outline of the small region of interest on the target surface mask image 250. The correction unit 21 superimposes the outline of the small region of interest on the target surface mask image 250 so that the position of the outline of the small region of interest in the target surface mask image 250 is the same as the position of the outline of the small region of interest in the color image 12. In other words, the correction unit 21 superimposes the outline of the small region of interest on the target surface mask image 250 so that the pixel positions of multiple pixels within the outline of the small region of interest in the target surface mask image 250 match the pixel positions of multiple pixels within the outline of the small region of interest in the color image 12, respectively.

[0082] Next, the correction unit 21 calculates, for example, the total number of pixels, which is the number of all pixels present in the small region of interest overlaid on the target surface mask image 250. The correction unit 21 also calculates, for example, the number of pixels of the target surface region 255 present in the small region of interest overlaid on the target surface mask image 250, which is the number of occupied pixels. The correction unit 21 then determines, for example, the value obtained by dividing the number of occupied pixels by the total number of pixels as the occupancy rate of the target surface region 255 within the outline of the small region of interest. In this way, the correction unit 21 calculates the occupancy rate for each small region.

[0083] In step s13, the correction unit 21 determines, for each small region, whether the area within the outline of the small region in the target surface mask image 250 corresponds to the target surface 15a, for example, based on the occupancy rate of the small region. For example, in the target surface mask image 250 in which the outlines of multiple small regions are superimposed (also referred to as the contour-superimposed target surface mask image 250), the correction unit 21 determines that the area within the outline of a small region whose occupancy rate is greater than a threshold value corresponds to the target surface 15a. On the other hand, for example, in the contour-superimposed target surface mask image 250, the correction unit 21 determines that the area within the outline of a small region whose occupancy rate is less than the threshold value does not correspond to the target surface 15a. The correction unit 21 may determine that the area within the outline of a small region whose occupancy rate matches the threshold value in the contour-superimposed target surface mask image 250 corresponds to the target surface 15a or does not correspond to the target surface 15a. Hereinafter, a small region whose occupancy rate is greater than the threshold value will be referred to as a small region with a large occupancy rate, and a small region whose occupancy rate is less than the threshold value will be referred to as a small region with a small occupancy rate.

[0084] For each small region with a large occupancy rate, the correction unit 21 regards the entire area within the outline of the small region in the contour-superimposed target surface mask image 250 as an area corresponding to the target surface 15a. On the other hand, for each small region with a small occupancy rate, the correction unit 21 regards the entire area within the outline of the small region in the contour-superimposed target surface mask image 250 as an area not corresponding to the target surface 15a. The threshold value may be, for example, 0.7 or more, 0.8 or more, 0.9 or more, or another value.

[0085] 10 and 11 are schematic diagrams for explaining an example of the operation of the correction unit 21. The upper part of Fig. 10 shows an example of an area 260 within the outline of a small area with a large occupancy rate (also referred to as an area within the outline of a large occupancy rate) in the mask image 250 of the surface on which the contour is superimposed. The upper part of Fig. 11 shows an example of an area 261 within the outline of a small area with a small occupancy rate (also referred to as an area within the outline of a small occupancy rate) in the mask image 250 of the surface on which the contour is superimposed.

[0086] The correction unit 21 regards the entire large occupancy rate intra-contour area 260 in the contour-superimposed target surface mask image 250 as an area corresponding to the target surface 15a, and resets the pixel values ​​of all pixels constituting the large occupancy rate intra-contour area 260 to "1." The lower part of Fig. 10 shows an example of the large occupancy rate intra-contour area 260 in which the pixel values ​​of all pixels have been reset to "1." The large occupancy rate intra-contour area 260 in which the pixel values ​​of all pixels have been reset to "1" constitutes part of the shaped target surface area 255.

[0087] Furthermore, the correction unit 21 regards all of the small occupancy rate intra-contour area 261 in the contour superimposed target surface mask image 250 as areas that do not correspond to the target surface 15a, and resets the pixel values ​​of all pixels that make up the small occupancy rate intra-contour area 261 to "0." An example of the small occupancy rate intra-contour area 261 in which the pixel values ​​of all pixels have been reset to "0" is shown in the lower part of Fig. 11.

[0088] The correction unit 21 resets the pixel values ​​of all pixels in each of the large-occupancy intra-contour regions 260 in the contour-superimposed target surface mask image 250 to "1," and resets the pixel values ​​of all pixels in each of the small-occupancy intra-contour regions 261 to "0." This causes the target surface region 255 (i.e., the region with a pixel value of "1") included in the contour-superimposed target surface mask image 250 to be shaped in accordance with the actual shape of the target surface 15a. As a result, the shape of the target surface region 255 after shaping becomes closer to the actual shape of the target surface 15a. When the contours of each small region are erased from the contour-superimposed target surface mask image 250 in which the target surface region 255 has been shaped, a target surface mask image 250 in which the target surface region 255 has been shaped is obtained. Hereinafter, the target surface mask image 250 in which the target surface region 255 has been shaped will be referred to as the shaped target surface mask image 250a.

[0089] Fig. 12 is a schematic diagram showing an example of a target surface mask image 250a after shaping. Fig. 12 shows an example of how the target surface region 255 has been shaped in the target surface mask image 250 shown in Fig. 7. The shape of the target surface region 255 after shaping shown in Fig. 12 is closer to the actual shape of the target surface 15a (the shaded portion in Fig. 3) than the shape of the target surface region 255 shown in Fig. 7.

[0090] In the above example, SLIC is used to divide the color image 12 into a plurality of small regions such that the boundaries between the small regions are along the edges. However, a segmentation algorithm other than SLIC may be used to divide the color image 12 into a plurality of small regions. For example, Watershed or Quickshift may be used to segment the color image 12.

[0091] Furthermore, the color image 12 divided into multiple small regions 300 shown in FIG. 9 does not necessarily have to be generated. In this case, in step s12, an object region representing the object 15 divided into multiple small regions may be extracted from the color-converted color image 12 divided into multiple small regions (also referred to as the divided color-converted color image 12). For example, the correction unit 21 identifies multiple pixels in the divided color-converted color image 12 that are located at the same pixel positions as the pixel positions of the multiple pixels constituting the object region 201 in the object mask image 200. Then, the correction unit 21 extracts a region consisting of the identified multiple pixels in the divided color-converted color image 12 as an object region divided into multiple small regions. This object region divided into multiple small regions is referred to as a second divided object region. In step s13, the second divided object region is used instead of the divided object region 120 to shape the object surface region 255 in the object surface mask image 250.

[0092] In the process of FIG. 8 , the color image 12 is divided into multiple small regions after color conversion. However, the color image 12 may be divided into multiple small regions without color conversion. In this case, step s110 is not executed, and in step s111, the positions of the centers of gravity of k small regions are initially set for the color image 12, which is an RGB image. Then, step s112 is similarly executed. In step s113, the correction unit 21 arranges multiple pixels constituting the color image 12 (i.e., the RGB image 12) in a five-dimensional space with the R component, G component, B component, x coordinate, and y coordinate as axes. Then, for each pixel in the five-dimensional space, the correction unit 21 tentatively determines the small region to which the pixel belongs. Then, step s114 is similarly executed. Thereafter, steps s113 and s114 are repeatedly executed, and the color image 12, which is an RGB image, is divided into k small regions so that the boundaries between the small regions are aligned along the edges.

[0093] Furthermore, in the above example, the segmented object regions are extracted from the segmented color image 12 based on the object mask image 200. However, the segmented object regions do not necessarily have to be extracted. That is, the second mask process of step s12 does not have to be performed. In this case, step s13 may calculate the occupancy rate for each small region of the entire segmented color image 12. Furthermore, step s13 may use an object surface mask image 250 (also known as the second contour-superimposed object surface mask image 250) in which the outlines of the multiple small regions constituting the entire segmented color image 12 are superimposed. Then, in step s13, the area within the outline of each small region whose occupancy rate is greater than a threshold value in the second contour-superimposed object surface mask image 250 may be regarded as an area corresponding to the object surface 15a, and the area within the outline of each small region whose occupancy rate is less than the threshold value may be regarded as an area not corresponding to the object surface 15a.

[0094] Furthermore, in the above example, the color image 12 is divided into multiple small regions, and then the object region is extracted from the color image 12; however, the object region may be divided into multiple small regions after being extracted from the color image 12.

[0095] The processing device 1 may also estimate each of multiple object planes included in the surface of the object 15. Here, the plane with the largest area as seen from the sensor device 10 among the object planes described so far, i.e., the object measurement plane, is defined as the first object plane. After performing step s7, the estimation unit 20 corrects the object point cloud by removing the multiple object surface points representing the first object plane estimated in step s6 from the object point cloud generated in step s3. The estimation unit 20 then performs steps s4 to s6 using the corrected object point cloud to estimate multiple object surface points representing the second object plane. The second object plane is the plane with the second largest area as seen from the sensor device 10 among the object measurement planes. The estimation unit 20 then generates a second object plane mask image 250 representing the second object plane in step s7. Then, in step s13, the correction unit 21 similarly shapes the second object plane region representing the second object plane included in the second object plane mask image 250 based on the multiple small regions.

[0096] When the processing device 1 estimates a third object surface separate from the first and second object surfaces, the object point cloud is further modified by removing multiple object surface points representing the estimated second object surface from the modified object point cloud. The estimation unit 20 then executes steps s4 to s6 using the further modified object point cloud to estimate multiple object surface points representing the third object surface. The third object surface is the plane with the third largest area as seen from the sensor device 10 on the object measurement surface. The estimation unit 20 then generates a third object surface mask image 250 representing the third object surface in step s7. The correction unit 21 then reshapes the third object surface region representing the third object surface, included in the third object surface mask image 250, based on multiple small regions in step s13. Thereafter, the processing device 1 can estimate further object surfaces in a similar manner.

[0097] <Example of Use of Estimation Result of Target Surface> The target surface estimation result corrected by the processing device 1 can be used in various situations. For example, when the suction unit suctions the target surface 15 a, the suction position on the target surface 15 a where the suction unit suctions may be determined based on the target surface estimation result corrected by the correction unit 21.

[0098] 13 is a schematic diagram showing an example of a robot system 900 including a robot 500 having a suction portion that suctions a target surface 15a of an object 15. The robot system 900 includes, for example, the robot 500, a robot control device 800 that controls the robot 500, and a processing device 1.

[0099] The robot 500 is, for example, an arm-type robot, and includes an arm 510 and an end effector 520 connected to the arm 510. The end effector 520 includes a suction pad 521 that functions as a suction unit. The suction pad 521 is capable of suctioning an object surface 15 a of the object 15 and holding the object 15.

[0100] 14 is a schematic diagram showing an example of the configuration of a robot control device 800. The robot control device 800 includes, for example, a control unit 810, a storage unit 820, an interface 830, and an interface 840. The robot control device 800 can also be considered, for example, a robot control circuit. The robot control device 800 can also be considered, for example, a computer device.

[0101] The interface 830 is capable of communicating with the interface 5 of the processing device 1. The interface 840 is capable of communicating with the robot 500. The interface 840 may communicate with the robot 500 by wired communication or wireless communication.

[0102] The control unit 810 can comprehensively manage the operation of the robot control device 800 by controlling the other components of the robot control device 800. The control unit 810 has, for example, the same configuration as the control unit 2 provided in the processing device 1, and the control unit 810 may have, for example, a CPU as a processor.

[0103] The storage unit 820 has, for example, the same configuration as the storage unit 3 of the processing device 1. The storage unit 820 stores, for example, a program 821 for controlling the robot control device 800. The various functions of the control unit 810 are realized, for example, by the CPU of the control unit 810 executing the program 821 in the storage unit 820. The control unit 810 can control the robot 500 through the interface 840. The control unit 810 can control, for example, the arm 510 and the end effector 520. Note that the robot control device 800 may be composed of a robot control device that controls the arm 510 and a robot control device that controls the end effector 520.

[0104] The adsorption position on the target surface 15a may be determined by the processing device 1 or by a device other than the processing device 1. Fig. 15 is a schematic diagram showing an example of the configuration of the processing device 1 that determines the adsorption position.

[0105] 15 , the control unit 2 includes a determination unit 22 as a functional block. The determination unit 22 is realized by the CPU of the control unit 2 executing the program 30 in the storage unit 3. The determination unit 22 determines the suction position on the target surface 15 a where the suction pad 521 will suction, based on the target surface estimation result corrected by the correction unit 21. Note that all or some of the functions of the determination unit 22 may be realized by a hardware circuit that does not require software to realize the function.

[0106] The determination unit 22 determines the suction position based on, for example, the shaped target surface mask image 250a. For example, the determination unit 22 calculates the shortest straight-line distance from an edge for each of a plurality of pixels constituting the target surface region 255 included in the shaped target surface mask image 250a (i.e., the shaped target surface region 255). The edge can also be said to be the contour or edge of the target surface region 255. Then, the determination unit 22 replaces the pixel value of each pixel of the target surface region 255 in the shaped target surface mask image 250a with the value of the shortest straight-line distance calculated for that pixel, thereby generating a distance-transformed image. The distance-transformed image is, for example, a grayscale image.

[0107] Fig. 16 is a schematic diagram showing an example of a distance transformed image 270. The actual distance transformed image 270 is a grayscale image, but for convenience of illustration, Fig. 16 shows the distance transformed image 270 as a white portion 271 and a black portion 272.

[0108] The white portion 271 is an area in the distance transformed image 270 where the pixel values ​​are relatively large. The black portion 272 is an area in the distance transformed image 270 where the pixel values ​​are relatively small. The white portion 271 corresponds to an area in the target surface region 255 of the shaped target surface mask image 250a where the shortest straight-line distance from the edge is relatively large. The white portion 271 represents a central region in the target surface region 255 that is far from the edge (in other words, the contour). It can also be said that the white portion 271 represents a central region in the target surface that is far from the edge.

[0109] The determination unit 22 determines the position of the center of gravity of the target surface region 255 of the shaped target surface mask image 250a. For example, the determination unit 22 determines the average value of the x coordinates of a plurality of pixels constituting the target surface region 255 and the average value of the y coordinates of the plurality of pixels. Then, the determination unit 22 determines the average value of the determined x coordinates as the x coordinate of the center of gravity, and the average value of the determined y coordinates as the y coordinate of the center of gravity.

[0110] Next, the determination unit 22 sets a center of gravity point in the distance transformed image 270 at the same position as the center of gravity position of the target surface region 255. Then, the determination unit 22 identifies, from among the multiple pixels constituting the distance transformed image 270, a pixel whose pixel value is equal to or greater than a threshold and which is closest to the center of gravity, and sets the pixel position of the identified pixel as an attraction pixel position. The attraction pixel position represents a position on the target surface that is far from the edge and close to the center of gravity. Then, the determination unit 22 sets an attraction point 280 at the attraction pixel position in the color image 12 in which the target surface appears.

[0111] 17 is a schematic diagram showing an example of a color image 12 in which an adsorption point 280 is set. The adsorption point 280 is located at a position far from the edge and close to the center of gravity on the target surface 15a shown in the color image 12. The determination unit 22 determines the position at which the adsorption point 280 is set on the target surface 15a shown in the color image 12 as the adsorption position of the adsorption pad 521.

[0112] The processing device 1 notifies the robot control device 800 of the suction position determined by the determination unit 22 via the interface 5. In the robot control device 800, the control unit 810 controls the position of the suction pad 521 based on the suction position notified to the interface 830. For example, the control unit 810 adsorbs the suction pad 521 to the target surface 15a so that the suction position notified to the interface 830 coincides with the center of the suction pad 521.

[0113] 17 shows an outline 285 of the suction pad 521 when the suction pad 521 adsorbs the target surface 15a so that the center of the suction pad 521 coincides with the adsorption position determined by the determination unit 22. In the example of Fig. 17, the entire suction pad 521 is in contact with the target surface 15a, and the suction pad 521 can appropriately adsorb the target surface 15a.

[0114] Fig. 18 is a schematic diagram showing an example of a color image 12 in which an adsorption point 280 is set, which is determined by the determination unit 22 in a similar manner based on the unshaped target surface mask image 250 shown in Fig. 7. In the example of Fig. 18, the adsorption point 280 is set near the edge of the target surface 15a shown in the color image 12. Therefore, a part of the adsorption pad 521 that adsorbs the target surface 15a protrudes from the target surface 15a, making it difficult for the adsorption pad 521 to adsorb the target surface 15a.

[0115] In this way, the suction position at which the suction unit suctions the target surface 15a is set based on the target surface estimation result corrected by the correction unit 21, thereby enabling the suction unit to properly suction the target surface 15a.

[0116] 19 , the control unit 810 of the robot control device 800 may include a determination unit 811 similar to the determination unit 22 of the processing device 1. In this case, the processing device 1 notifies the robot control device 800 of the target surface estimation result corrected by the correction unit 21. In the robot control device 800, the determination unit 811 determines, similar to the determination unit 22, the suction position on the target surface 15 a where the suction pad 521 will suction, based on the corrected target surface estimation result (in other words, the detection result of the target surface 15 a by the detection unit 25). Then, the control unit 810 controls the position of the suction pad 521 based on the suction position determined by the determination unit 811. When the control unit 810 includes the determination unit 811, it can be said that the control unit 810 controls the robot 500 based on the detection result of the target surface 15 a by the detection unit 25.

[0117] Furthermore, the processing device 1 may control the robot 500 based on the detection result of the target surface 15a by the detection unit 25. FIG. 20 is a schematic diagram showing an example of the configuration of the processing device 1 in this case. In the example of FIG. 20 , the control unit 2 includes a robot control unit 23 that controls the robot 500. The robot control unit 23 can control the robot 500, for example, through the interface 5. Similar to the control unit 810, the robot control unit 23 controls the position of the suction pad 521 based on the suction position determined by the determination unit 22. The processing device 1 shown in FIG. 20 can be said to be a robot control device 1 that controls the robot 500.

[0118] In addition, if the target surface is a placement surface on which the robot places the object it is holding, the control unit 2 or the control unit 810 may determine the placement position on the placement surface on which the robot places the object based on the corrected target surface estimation result.

[0119] <Example of Setting the Threshold for Specific Surface Estimation and the Threshold for Object Surface Point Estimation> For example, consider a case where the threshold for specific surface estimation used in step s5 and the threshold for object surface point estimation used in step s6 are the same. In this case, if the threshold for specific surface estimation and the threshold for object surface point estimation are large, it may be impossible to properly estimate a specific surface that is on the same plane as the object surface. On the other hand, if the threshold for specific surface estimation and the threshold for object surface point estimation are small, it may be possible to properly estimate a specific surface, but it may be impossible to properly estimate multiple object surface points that represent the object surface included in the object point cloud. This point will be explained below.

[0120] For example, consider the case where the color image 12 shown in Fig. 21 is used. The color image 12 (also referred to as color image 12A) shown in Fig. 21 shows an object 15 with a step on its upper surface. A higher surface 150 of the upper surface is the object surface 15a, and a lower surface 151 of the upper surface is not the object surface 15a.

[0121] 22 and 23 are schematic diagrams showing an example of a target surface mask image 250 generated by the estimation unit 20 based on the color image 12A and the corresponding depth image. Fig. 22 shows the target surface mask image 250 when the specific surface estimation threshold value and the target surface point estimation threshold value are set to relatively large values ​​(e.g., 1.0 mm). Fig. 23 shows the target surface mask image 250 when the specific surface estimation threshold value and the target surface point estimation threshold value are set to relatively small values ​​(e.g., 0.5 mm).

[0122] The target surface mask image 250 shown in FIG. 22 shows not only the higher surface 150 as the target surface 15a, but also a lower surface 151 that is not the target surface 15a. This is because the specific surface was not properly estimated. On the other hand, the target surface mask image 250 shown in FIG. 23 does not show the lower surface 151 that is not the target surface 15a. However, the shape of the higher surface 150 as the target surface 15a that appears in the target surface mask image 250 is significantly different from the actual shape. This is because the multiple object points that represent the higher surface 150 as the target surface 15a were not properly estimated.

[0123] 24 is a schematic diagram illustrating an example of the specific surface estimation process when the specific surface estimation threshold and the target surface point estimation threshold are 1.0 mm. FIG. 24 roughly illustrates an example of a range 310 (also referred to as a point cloud existence range 310) in which the object point cloud generated in step s3 exists in the camera coordinate system. The steps on the upper surface of the point cloud existence range 310 correspond to the steps on the upper surface of the object 15. The higher surface on the upper surface of the point cloud existence range 310 corresponds to the higher surface 150 as the target surface 15a, and the lower surface on the upper surface of the point cloud existence range 310 corresponds to the lower surface 151 that is not the target surface 15a.

[0124] 24 shows examples of candidate surfaces 320a and 320b set in the camera coordinate system in step s4. Candidate surface 320a passes diagonally through the higher surface of the top surface of point cloud existence range 310 and the lower surface of the top surface of point cloud existence range 310. Candidate surface 320b includes the higher surface of the top surface of point cloud existence range 310.

[0125] 24 also shows, with a bold line, an example of a range 315aa in which, among the plurality of object points (in other words, measurement points) on the higher surface 150 of the upper surface of the object 15, there are object points whose distance from the candidate surface 320a is equal to or less than the specific surface estimation threshold value (i.e., 1.0 mm). Also, FIG. 24 also shows, with a bold line, an example of a range 315ab in which, among the plurality of object points on the lower surface 151 of the upper surface of the object 15, there are object points whose distance from the candidate surface 320a is equal to or less than the specific surface estimation threshold value. Finally, FIG. 24 also shows, with a bold line, an example of a range 315b in which, among the plurality of object points on the higher surface 150 of the upper surface of the object 15, there are object points whose distance from the candidate surface 320b is equal to or less than the specific surface estimation threshold value.

[0126] Ideally, it would be desirable to estimate candidate surface 320b as a specific surface that is coplanar with the target surface. However, in the example of Figure 24, the number of object points whose distance from candidate surface 320a is less than the specific surface estimation threshold (i.e., 1.0 mm) is greater than the number of object points whose distance from candidate surface 320b is less than the specific surface estimation threshold. Therefore, candidate surface 320a, rather than candidate surface 320b, is estimated as a specific surface that is coplanar with the target surface. In this case, as shown in Figure 22, the target surface mask image 250 includes not only the higher surface 150 that is the target surface 15a, but also a lower surface 151 that is not the target surface 15a.

[0127] FIG. 25 is a schematic diagram illustrating an example of a specific surface estimation process when the specific surface estimation threshold and the target surface point estimation threshold are 0.5 mm. The content illustrated in FIG. 25 is the same as that illustrated in FIG. 24. In the example of FIG. 25, the number of object points whose distance from the candidate surface 320b is equal to or less than the specific surface estimation threshold (i.e., 0.5 mm) is greater than the number of object points whose distance from the candidate surface 320a is equal to or less than the specific surface estimation threshold. Therefore, the candidate surface 320b is estimated as a specific surface that is coplanar with the target surface. In this case, as illustrated in FIG. 23, the lower surface 151, which is not the target surface 15a, does not appear in the target surface mask image 250, but the higher surface 150, which is the target surface 15a, appears.

[0128] On the other hand, if, due to a distance measurement error in the sensor device 10, the multiple object points representing the target surface 15a included in the target point group include a relatively large number of object points whose distance from the candidate surface 320b is greater than 0.5 mm, the multiple object points representing the target surface 15a will not be estimated properly, and as shown in Figure 23, the shape of the higher surface 150 as the target surface 15a shown in the target surface mask image 250 will be significantly different from the actual shape.

[0129] 26 is a schematic diagram showing an example of a state in which a relatively large number of object points representing the target surface 15a are located at a distance from the candidate surface 320b greater than 0.5 mm. In FIG. 26, a thick line roughly indicates an example of a range 316b in which, among the object points on the upper surface 150 of the object 15 (in other words, among the object points representing the target surface 15a), object points located at a distance of 0.5 mm or less from the candidate surface 320b are located. Also, in FIG. 26, a thinner line roughly indicates a range 316bb in which, among the object points on the upper surface 150 of the object 15 (in other words, among the object points representing the target surface 15a), object points located at a distance greater than 0.5 mm from the candidate surface 320b are located.

[0130] 27 is a schematic diagram showing an example of a target surface mask image 250 when the specific surface estimation threshold is 0.5 mm and the target surface point estimation threshold is 1.0 mm. The target surface mask image 250 shown in FIG. 27 does not show a lower surface 151 that is not the target surface 15a, and the shape of the higher surface 150 that is shown in the target surface mask image 250 and serves as the target surface 15a is close to the actual shape. This is because the specific surface is correctly estimated, and as shown in FIG. 28, the multiple object points representing the target surface 15a include almost no object points that are farther away from the candidate surface 320b than 1.0 mm.

[0131] In this way, the accuracy of estimating the target surface is improved by separately setting the threshold value for estimating the specific surface and the threshold value for estimating the target surface points. Furthermore, by increasing the threshold value for estimating the target surface points, the robustness of the target surface estimation process against distance measurement errors in the sensor device 10 is improved. In other words, the robustness of the target surface estimation process against errors (in other words, noise) contained in the depth image 11 is improved.

[0132] Next, a specific example of a method for setting the threshold value for specific surface estimation and the threshold value for target surface point estimation will be described. The methods for setting the threshold value for specific surface estimation and the threshold value for target surface point estimation are based on mutually different setting standards. First, a specific example of a method for setting the threshold value for target surface point estimation will be described.

[0133] As can be understood from the above explanation, if the threshold value for estimating the target surface point is too large, there is a possibility that object points (in other words, measurement points) that are not on the same plane as the target surface will be estimated as target surface points. On the other hand, if the threshold value for estimating the target surface point is too small, the robustness of the target surface estimation process against errors contained in the depth image 11 will decrease.

[0134] Therefore, the threshold value for estimating the target surface point is set based on, for example, the maximum error in the distance represented by the depth image 11. This reduces the possibility that an object point that is not on the same plane as the target surface is estimated as a target surface point while maintaining the robustness of the target surface estimation process against errors contained in the depth image 11. In other words, the accuracy of estimating the target surface can be improved.

[0135] When the target surface point estimation threshold is set, for example, a depth image 11 generated by the sensor device 10 in the real environment where the sensor device 10 is actually used is used. This depth image 11 is called the depth image 11 in the real environment. Then, the maximum error of the distance represented by the depth image 11 in the real environment is obtained by experiment or simulation. For example, the error between each of a plurality of measured distances represented by the pixel values ​​of a plurality of pixels constituting the depth image 11 in the real environment and the actual distance is obtained by experiment or simulation. Then, the maximum error among the errors obtained for the plurality of measured distances is set as the maximum error of the distance represented by the depth image 11 in the real environment. The target surface point estimation threshold may be set to a value equal to the maximum error of the distance represented by the depth image 11 in the real environment, or may be set to a value obtained by multiplying the maximum error by a predetermined adjustment coefficient.

[0136] Next, a specific example of a method for setting the specific surface estimation threshold will be described. As can be understood from the above description, the smaller the specific surface estimation threshold, the more accurate the estimation of the specific surface. On the other hand, when RANSAC, for example, is used to estimate the specific surface as described above, in order to properly estimate the specific surface, the object point cloud generated from the depth image 11 must contain three or more object points whose distance from the object surface is equal to or less than the specific surface estimation threshold. If the specific surface estimation threshold is too small, the number of object points whose distance from the object surface is equal to or less than the specific surface estimation threshold may be less than three, and the specific surface may not be properly estimated.

[0137] Therefore, the threshold value for estimating the specific surface is set based on, for example, the error distribution of the distance represented by the depth image 11. This makes it possible to improve the accuracy of estimating the specific surface.

[0138] When a threshold value for estimating a specific surface is set, for example, the distribution of errors at multiple measurement distances represented by the pixel values ​​of multiple pixels constituting the depth image 11 in the real environment is calculated as the error distribution of the distances represented by the depth image 11 (also referred to as the distance error distribution). Specifically, based on the errors of multiple measurement distances corresponding to the multiple pixels constituting the depth image 11 in the real environment, the relationship between each error value and the number of pixels corresponding to the measurement distances having an error of that value is calculated. In other words, for each value of the error of the measurement distance in the depth image 11 in the real environment, the number of pixels corresponding to the measurement distances having an error of that value is calculated.

[0139] Next, the probability that the error in the measured distance corresponding to one pixel included in the depth image 11 is equal to or less than the threshold value for estimating a specific surface is calculated. Here, the threshold value for estimating a specific surface is represented by x. Also, the probability that the error in the measured distance corresponding to one pixel included in the depth image 11 is equal to or less than x is represented by r(x). The probability r(x) is expressed by the following equation (1).

[0140]

[0141] In formula (1), S means the number of pixels constituting the depth image 11 (i.e., the total number of pixels in the depth image 11). Also, in formula (1), m means the number of pixels corresponding to the measurement distance having an error of x or less in the obtained distance error distribution.

[0142] Next, a probability P1(x) is calculated that, among the plurality of pixels included in the depth image 11 corresponding to the target surface 15a, the number of pixels corresponding to the measurement distance with an error of x or less will be 2 or less. The plurality of pixels included in the depth image 11 corresponding to the target surface 15a can also be considered to be a plurality of pixels corresponding to the plurality of measurement points included in the depth image 11 on the target surface 15a (in other words, a plurality of measurement points representing the target surface 15a). The probability P1(x) is the sum of the probability that, among the plurality of pixels included in the depth image 11 corresponding to the target surface 15a, the number of pixels corresponding to the measurement distance with an error of x or less is 0, the probability that the number of pixels corresponding to the measurement distance with an error of x or less is 1, and the probability that the number of pixels corresponding to the measurement distance with an error of x or less is 2. The probability P1(x) is expressed by the following equation (2) using the combination C.

[0143]

[0144] In equation (2), N represents the number of pixels included in the depth image 11 and corresponding to the target surface 15a.

[0145] Next, the probability P1(x) is used to calculate the probability P2(x) that the number of pixels corresponding to the measurement distance with an error of x or less will be 3 or more among the multiple pixels corresponding to the target surface 15a included in the depth image 11. The probability P2(x) is expressed by the following equation (3). The larger the specific surface estimation threshold value x, the larger the probability P2(x).

[0146]

[0147] For example, if N=2500 and r (0.1 mm)=0.1, P2 (0.1 mm) is expressed by the following equation (4).

[0148]

[0149] The specific surface estimation threshold value x is set to a value that is as small as possible while still providing a sufficiently large value for the probability P2(x). For example, if a value of 0.999 or greater is sufficient for the probability P2(x), the specific surface estimation threshold value x is set to a value that provides a probability P2(x) of 0.999.

[0150] In this way, the specific surface estimating threshold value and the target surface point estimating threshold value are set according to mutually different setting standards, so that, for example, the target surface point estimating threshold value is set to be larger than the specific surface estimating threshold value x.

[0151] Furthermore, since the threshold value for estimating a specific surface and the threshold value for estimating a target surface point are set using different setting criteria, depending on the distance measurement error in the sensor device 10, the threshold value for estimating a specific surface and the threshold value for estimating a target surface point may be set to the same value, or the threshold value for estimating a specific surface may be set to a value greater than the threshold value for estimating a target surface point.

[0152] Although the processing device and the program have been described in detail above, the above description is merely illustrative in all respects and does not limit the scope of this disclosure. Furthermore, the various examples described above can be combined and applied as long as they are not mutually inconsistent. It is understood that countless examples not illustrated can be envisioned without departing from the scope of this disclosure.

[0153] For example, in the above example, an arm-type robot is described as an application example of the processing device 1. However, the processing device 1 may also be applied to a self-propelled robot such as an AGV (Automatic Guided Vehicle) or an AMR (Autonomous Mobile Robot), or to a humanoid robot. The processing device 1 may also be applied to a care robot, a remote-controlled robot, or the like. In the above example, the object 15 is an industrial product, but this is not limiting. The object 15 may be, for example, food such as vegetables or bread, furniture such as a chest of drawers, or household goods such as a toothbrush or a cup.

[0154] This disclosure includes the following:

[0155] In one embodiment, (1) the processing device includes a detection unit that detects a partial surface included in the surface based on surface information regarding the surface of the object obtained based on distance information to the object and color information indicated by a color image or grayscale image showing the object and its surroundings.

[0156] (2) In the processing device of (1) above, the color information includes first color information within the partial surface and second color information around the partial surface.

[0157] (3) In the processing device of (2) above, the first color information includes a color tone difference within the partial surface.

[0158] (4) In the processing device according to any one of (1) to (3), the color information includes a color tone difference of the object.

[0159] (5) A processing device according to any one of (1) to (4) above, wherein the detection unit has an estimation unit that acquires the surface information based on the distance information and estimates the partial surface based on the acquired surface information, and a correction unit that corrects the estimation result of the estimation unit based on the color information and sets the corrected estimation result as the detection result of the partial surface.

[0160] (6) In the processing device of (5) above, the correction unit divides the area of ​​the object contained in the color image or the grayscale image into multiple small areas based on the color information so that the boundaries follow the edges of the object, and corrects the estimation result based on the multiple small areas.

[0161] (7) A processing device according to (6) above, wherein the estimation unit generates a mask image of the partial surface as the estimation result, and the correction unit calculates, for each of the plurality of small regions, the occupancy rate of the partial surface area within the contour when the contour of the small region is superimposed on the mask image, and shapes the partial surface area based on the occupancy rate for the plurality of small regions.

[0162] (8) In the processing device of (7) above, the correction unit determines whether or not the area within the outline of the small region in the mask image corresponds to the partial surface based on the occupancy rate for the small region.

[0163] (9) A processing device according to any one of (1) to (8), wherein the estimation unit generates a point cloud representing the surface of the object as the surface information based on a depth image indicating the distance information, sets a plurality of candidate surfaces based on the point cloud, estimates a candidate surface that is on the same plane as the object surface from the plurality of candidate surfaces based on a comparison result between the distance from the candidate surface to a point included in the point cloud and a first threshold value, and estimates a plurality of points representing the partial surface included in the point cloud based on a comparison result between the distance from the candidate surface estimated to be on the same plane as the object surface to a point included in the point cloud and a second threshold value.

[0164] (10) In the processing device of (9) above, the second threshold value is equal to or greater than the first threshold value.

[0165] (11) In the processing device of (9) or (10) above, the first threshold value and the second threshold value are based on different setting criteria.

[0166] (12) In the processing device of (11), the first threshold is based on an error distribution of the distance represented by the depth image.

[0167] (13) In the processing device of (11) or (12), the second threshold is based on a maximum error in the distance represented by the depth image.

[0168] (14) A processing device according to any one of (1) to (13) above, further comprising a determination unit that determines an adsorption position on the partial surface at which the adsorption unit adsorbs, based on the detection result of the partial surface by the detection unit.

[0169] (15) The processing device according to (14) above, further comprising a control unit that controls the position of the suction unit based on the suction position determined by the determination unit.

[0170] (16) The robot control device includes a control unit that controls the robot based on the detection result of the partial surface by the detection unit provided in any one of the processing devices (1) to (13) above.

[0171] (17) A robot system includes the robot control device of (16) above and a robot controlled by the control unit provided in the robot control device.

[0172] (18) The program is a program for causing a computer device to function as any one of the processing devices (1) to (15) above.

[0173] (19) The program is a program for causing a computer device to function as the robot control device described above in (16).

[0174] REFERENCE SIGNS LIST 1 Processing device 11 Depth image 12 (12A) Photographed image, color image, grayscale image 15 Object 15a Object surface (partial surface) 20 Estimation unit 21 Correction unit 22, 811 Determination unit 25 Detection unit 30, 821 Program 50 Measurement space 120 Object region 250 Object surface mask image 251 Object surface region 300 Small region 320a, 320b Candidate surface 521 Adsorption unit 800 Robot control device 810 Control unit 900 Robot system

Claims

1. A processing device comprising a detection unit that detects partial surfaces included in the surface based on surface information regarding the surface of an object acquired based on distance information to the object and color information indicated by a color image or grayscale image showing the object and its surroundings.

2. 2. The processing device according to claim 1, The processing device, wherein the color information includes first color information within the subsurface and second color information surrounding the subsurface.

3. 3. The processing device according to claim 2, The processing device, wherein the first color information includes a color tone difference within the partial surface.

4. 2. The processing device according to claim 1, The processing device, wherein the color information includes color tone differences of the object.

5. 2. The processing device according to claim 1, The detection unit an estimation unit that acquires the surface information based on the distance information and estimates the partial surface based on the acquired surface information; a correction unit that corrects the estimation result of the estimation unit based on the color information and sets the corrected estimation result as the detection result of the partial surface; A processing device comprising:

6. 6. The processing device according to claim 5, The correction unit Dividing the area of ​​the object included in the color image or the grayscale image into a plurality of small areas based on the color information so that boundaries follow edges of the object; A processing device that modifies the estimation result based on the plurality of small regions.

7. 7. The processing device according to claim 6, the estimation unit generates a mask image of the partial surface as the estimation result, The correction unit For each of the plurality of small regions, an occupancy rate of the partial surface area within the contour of the small region when the contour of the small region is superimposed on the mask image is calculated; A processing device that shapes the area of ​​the partial surface based on the occupancy rates for the plurality of small areas.

8. 8. The processing device according to claim 7, The correction unit determines whether or not an area within the outline of the small region in the mask image corresponds to the partial surface based on the occupancy rate for the small region.

9. 2. The processing device according to claim 1, The estimation unit generating a point cloud representing a surface of the object as the surface information based on a depth image indicating the distance information; setting a plurality of candidate surfaces based on the point cloud; estimating a candidate surface that is on the same plane as the target surface from the plurality of candidate surfaces based on a comparison result between a distance from the candidate surface to a point included in the point cloud and a first threshold value; A processing device that estimates a plurality of points representing the partial surface included in the point cloud based on a comparison result between a distance from a candidate surface estimated to be on the same plane as the target surface to a point included in the point cloud and a second threshold value.

10. 10. The processing device according to claim 9, The second threshold is greater than or equal to the first threshold.

11. 10. The processing device according to claim 9, The processing device, wherein the first threshold and the second threshold are based on different setting criteria.

12. 12. The processing device according to claim 11, The processing device, wherein the first threshold is based on an error distribution of the distance represented by the depth image.

13. 12. The processing device according to claim 11, The processing device, wherein the second threshold is based on a maximum error in distance represented by the depth image.

14. 2. The processing device according to claim 1, a determining unit that determines a suction position on the partial surface at which the suction unit will suction, based on a detection result of the partial surface by the detecting unit.

15. 15. The processing device of claim 14, a control unit that controls a position of the suction unit based on the suction position determined by the determination unit.

16. A robot control device comprising: a control unit that controls a robot based on a detection result of the partial surface by the detection unit included in the processing device according to any one of claims 1 to 15.

17. The robot control device according to claim 16 ; a robot controlled by the control unit included in the robot control device; A robot system comprising:

18. A program for causing a computer device to function as the processing device according to any one of claims 1 to 15.

19. A program for causing a computer device to function as the robot control device according to claim 16.