Processing device, processing system, processing method, and recording medium

Abstract

This processing device comprises: a first identification means (4031) that uses data obtained by observing an object from a plurality of viewpoints and identifies the type of the object from which noise generated by specular reflection is removed; and a determination means (4033) that determines an operation to be applied to the object on the basis of the type of the object identified by the first identification means.

Description

Processing device, processing system, processing method, and recording medium

[0001] The present disclosure relates to a processing device, a processing system, a processing method, and a recording medium.

[0002] Image processing techniques are used to identify objects in various fields such as logistics. Japanese Patent Laid-Open No. 2011-144992 discloses, as a related technique, an image processing technique for performing robust position and orientation measurement even in a noisy environment.

[0003] JP 2015-090560 A

[0004] Incidentally, in the field of image processing related to Patent Document 1, there is a demand for a technique that can identify the state of a specularly reflective object.

[0005] One of the objectives of each aspect of the present disclosure is to provide a processing device, a processing system, a processing method, and a recording medium that can solve the above-mentioned problems.

[0006] According to one aspect of the present disclosure, the processing device includes a first identification means that identifies the type of object from which noise caused by specular reflection has been removed using data in which the object is observed from multiple viewpoints, and a determination means that determines an action to be performed on the object based on the type of object identified by the first identification means.

[0007] According to another aspect of the present disclosure, a processing system includes the processing device described above and a robot that operates under the control of the processing device.

[0008] According to another aspect of the present disclosure, a processing method includes identifying a type of object from which noise caused by specular reflection has been removed using data in which the object is observed from multiple viewpoints, and determining an action to be performed on the object based on the identified type of object.

[0009] According to another aspect of the present disclosure, a recording medium stores a program that causes a computer to perform the following steps: identify a type of object from which noise caused by specular reflection has been removed using data in which the object is observed from multiple viewpoints; and determine an action to be performed on the object based on the identified type of the object.

[0010] According to each aspect of the present disclosure, the state of a specularly reflected object can be identified.

[0011] FIG. 1 is a diagram illustrating an example of a configuration of a processing system according to some embodiments of the present disclosure. FIG. 2 is a diagram illustrating an example of a configuration of a robot according to some embodiments of the present disclosure. FIG. 3 is a diagram illustrating an example of a configuration of a control device according to some embodiments of the present disclosure. FIG. 4 is a diagram illustrating an example of a configuration of a computing device according to some embodiments of the present disclosure. FIG. 5 is a diagram illustrating an example of a 3D CAD model included in a database 401 according to some embodiments of the present disclosure. FIG. 6 is a first diagram illustrating an example of a processing flow of a processing system according to some embodiments of the present disclosure. FIG. 7 is a second diagram illustrating an example of a processing flow of a processing system according to some embodiments of the present disclosure. FIG. 8 is a first diagram illustrating processing performed by a reconstructing means according to some embodiments of the present disclosure. FIG. 9 is a second diagram illustrating processing performed by a reconstructing means according to some embodiments of the present disclosure. FIG. 10 is a diagram illustrating processing performed by an executing means according to some embodiments of the present disclosure. FIG. 11 is a diagram illustrating an example of a configuration of a processing device according to some embodiments of the present disclosure. FIG. 12 is a diagram illustrating an example of a processing flow of a processing device according to some embodiments of the present disclosure. FIG. 13 is a schematic block diagram illustrating a configuration of a computer according to at least one embodiment.

[0012] Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0013] <Embodiment> A processing system 1 according to an embodiment of the present disclosure is a system that identifies the type of an object M manufactured by a manufacturing apparatus 10 (described later) and enables a robot 20 (described later) to grasp the object M.

[0014] (Configuration of Processing System) Fig. 1 is a diagram showing an example of the configuration of a processing system 1 according to some embodiments of the present disclosure. As shown in Fig. 1, the processing system 1 includes a manufacturing apparatus 10, a robot 20, a control device 30, a computing device 40, and transfer mechanisms 50a and 50b. The transfer mechanisms 50a and 50b may be collectively referred to as the transfer mechanism 50.

[0015] The manufacturing apparatus 10 manufactures the object M under the control of the control device 30. Note that "manufacturing" in this disclosure is not limited to creating a new object from raw materials. For example, "manufacturing" may also include "processing," such as changing the shape. The object M includes objects M1 and M2 of different types. The object M is an object whose shape and state are difficult to identify due to its tendency to specular reflection, making it difficult to identify the type of the object M. For example, with a typical still camera, it is difficult to capture the shape and state of the object M due to specular reflection. Note that the shape may include size information. Furthermore, the state may include posture and position.

[0016] Fig. 2 is a diagram illustrating an example of the configuration of a robot 20 according to some embodiments of the present disclosure. Fig. 2 also illustrates a computing device 40. As illustrated in Fig. 2, the robot 20 includes a movement mechanism 201, an elevator 202, a robot arm 203, a sensor 204, and a computer 205.

[0017] The moving mechanism 201 moves the robot 20 under the control of the computer 205. For example, the moving mechanism 201 includes a motor and wheels, and by rotating the motor under the control of the computer 205, the wheels rotate in conjunction with the motor, causing the robot 20 to move.

[0018] As shown in Fig. 2, the elevator 202 is provided above the moving mechanism 201. As shown in Fig. 2, a robot arm 203 is provided above the elevator 202. The elevator 202 extends and contracts in the vertical direction under the control of a computer 205, thereby changing the position of the robot arm 203 in the vertical direction according to the height at which the target object M is output from the manufacturing apparatus 10.

[0019] The robot arm 203 includes a plurality of joints 2031 and a gripping mechanism 2032. Each of the plurality of joints 2031 includes a drive mechanism 2031a. Each of the drive mechanisms 2031a operates in response to a control signal from the computer 205. This allows the robot arm 203 to change to various angles and orientations. As a result, the gripping mechanism 2032 can assume a desired position and posture, enabling it to grip the object M.

[0020] The gripping mechanism 2032 grips the object M under the control of the computer 205. Then, under the control of the computer 205, the gripping mechanism 2032 moves the object M onto the transport mechanism 50 corresponding to the type of the object M and releases it. For example, if the type of object M is object M1, the gripping mechanism 2032 moves the object M1 onto the transport mechanism 50a and releases the object M1. Furthermore, if the type of object M is object M2, the gripping mechanism 2032 moves the object M2 onto the transport mechanism 50b and releases the object M2.

[0021] It should be noted that the gripping mechanism 2032 is not limited to one that clamps the object M. For example, the gripping mechanism 2032 may be one that sucks the object M. In other words, gripping here may also include suction.

[0022] The sensor 204 is a sensor capable of acquiring depth information that indicates the depth in the imaging direction. The sensor 204 is, for example, a depth camera.

[0023] The computer 205 transmits the depth information acquired by the sensor 204 to the calculation device 40. The computer 205 also controls the robot 20. For example, the computer 205 controls the robot 20 to move the sensor 204 to a position identified by the planning means 4033. For example, the computer 205 also controls the robot 20 to move the object M from the manufacturing device 10 to a transport mechanism 50 corresponding to the type of the object M. The calculation device 40 and the computer 205 are provided in a processing device 400 as shown in FIG. 2 .

[0024] 3 is a diagram illustrating an example of the configuration of the control device 30 according to some embodiments of the present disclosure. As shown in FIG. 3, the control device 30 includes a database 301 and a control unit 302.

[0025] The database 301 includes various information necessary for the manufacturing apparatus 10 to manufacture the objects M. For example, the database 301 includes information such as the number of objects M to be manufactured, the type of objects M, the shape of the objects M, and control details including setting values ​​of the manufacturing apparatus 10 when manufacturing the objects M.

[0026] The control unit 302 controls the manufacturing apparatus 10 based on the information contained in the database 301 .

[0027] Fig. 4 is a diagram showing an example of the configuration of the computing device 40 according to some embodiments of the present disclosure. Fig. 4 also shows a sensor 204 and a computer 205. As shown in Fig. 4, the computing device 40 includes a database 401, a reconstruction means 402, and a position determination means 403.

[0028] The database 401 includes a three-dimensional CAD (Computer Added Design) model. The three-dimensional CAD model is a model corresponding to information on the shape of the object M included in the database 301. The three-dimensional CAD model is a model in which the outer shape of the object M is represented by point cloud data. The three-dimensional CAD model includes a model represented by point cloud data for each type of object M (e.g., for each of objects M1 and M2). FIG. 5 is a diagram illustrating an example of a three-dimensional CAD model included in the database 401 according to some embodiments of the present disclosure. Part (a) of FIG. 5 illustrates an example of point cloud data that is a three-dimensional CAD model. Part (b) of FIG. 5 illustrates an example of the shape of the object M indicated by information included in the corresponding database 301.

[0029] The reconstructor 402 uses the depth information acquired by the sensor 204 to generate a three-dimensional model of the object M in which noise caused by specular reflections has been removed.

[0030] As shown in FIG. 4, the position determining means 403 includes an estimating means 4031, an executing means 4032, and a planning means 4033.

[0031] The estimation means 4031 estimates the state of the object M based on the 3D CAD model included in the database 401 and the 3D model generated by the reconstruction means 402. The state of the object M includes at least the 3D posture of the object M. The estimation means 4031 specifies the estimation accuracy of the estimated state of the object M. Then, the estimation means 4031 determines whether the specified estimation accuracy is equal to or greater than a threshold. If the estimation means 4031 determines that the specified estimation accuracy is equal to or greater than the threshold, it determines that the estimated state of the object M is the current correct state. Furthermore, if the estimation means 4031 determines that the specified estimation accuracy is less than the threshold, it determines that there is a possibility that the estimated state of the object M is not the current correct state. Then, the processing system 1 repeats a series of processes described below until it is determined that the state of the object M estimated by the estimation means 4031 is the current correct state.

[0032] The execution means 4032 sets parameters for a ray tracing simulation for a plurality of candidate objects M based on the three-dimensional model generated by the reconstruction means 402 and the state of the object M estimated by the estimation means 4031. Then, the execution means 4032 executes a ray tracing simulation using the parameters set for the plurality of candidate objects M. Note that the ray tracing simulation may be executed using a well-known ray tracing simulator.

[0033] The planning means 4033 identifies a position from among a plurality of positions from which highly accurate data can be obtained, based on the results of the simulation by the execution means 4032. The planning means 4033 then instructs the computer 205 to move the sensor 204 to the identified position. In this case, the computer 205 controls the robot 20 to move the sensor 204 to the position identified by the planning means 4033.

[0034] The transport mechanism 50a is a destination transport mechanism to which the robot 20 moves the object M1. The transport mechanism 50b is a destination transport mechanism to which the robot 20 moves the object M2. The transport mechanism 50 moves the object M to the next work location (e.g., a location where packaging work is performed). For example, the transport mechanism 50 is a belt conveyor.

[0035] Note that the processing performed by the processing system 1 according to an embodiment of the present disclosure is not limited to the above-described processing. For example, the processing system 1 may perform the processing described below.

[0036] (Processing Performed by Processing System) Fig. 6 is a first diagram illustrating an example of a processing flow of the processing system 1 according to some embodiments of the present disclosure. Fig. 7 is a second diagram illustrating an example of a processing flow of the processing system 1 according to some embodiments of the present disclosure. Next, processing performed by the processing system 1 will be described with reference to Figs. 6 and 7 . Note that, here, processing will be described in which, in the processing system 1, the arithmetic device 40 identifies the type of the target object M manufactured by the manufacturing apparatus 10, and the robot 20 moves the target object M identified by the arithmetic device 40 from the manufacturing apparatus 10 to the transport mechanism 50.

[0037] First, the computer 205 moves the sensor 204 to a position where depth information including the depth of at least a portion of the object M can be acquired (step S1). The computer 205 transmits imaging position information, which is information indicating the position of the sensor 204, to the reconstruction means 402. The sensor 204 acquires depth information including the depth of at least a portion of the object M.

[0038] The reconstruction means 402 acquires the photographing position information transmitted by the computer 205 (step S2), and also acquires the depth information acquired by the sensor 204 from the sensor 204 (step S3).

[0039] Next, the computer 205 moves the sensor 204 to a position different from the position in step S1 where depth information including the depth of a portion that overlaps with at least a portion of the object M in step S1 can be acquired (step S4. The computer 205 transmits shooting position information, which is information indicating the position of the sensor 204, to the reconstruction means 402. The sensor 204 acquires depth information including the depth of a portion that overlaps with at least a portion of the object M.

[0040] The reconstruction means 402 acquires the photographing position information transmitted by the computer 205 (step S5). The reconstruction means 402 also acquires the depth information acquired by the sensor 204 from the sensor 204 (step S6).

[0041] The reconstruction means 402 uses the depth information acquired by the sensor 204 at the two positions to generate a three-dimensional model of the object M from which noise caused by specular reflection has been removed (step S7).

[0042] Here, a specific example of the process performed by the reconstructing means 402 to generate a three-dimensional model of the object M from which noise caused by specular reflection has been removed will be described. FIG. 8 is a first diagram for describing the process performed by the reconstructing means 402 according to some embodiments of the present disclosure. FIG. 9 is a second diagram for describing the process performed by the reconstructing means 402 according to some embodiments of the present disclosure. FIG. 8 is a diagram for describing the process performed when the reconstructing means 402 determines that a real image has been observed. FIG. 9 is a diagram for describing the process performed when the reconstructing means 402 determines that a virtual image has been observed. The reconstructing means 402 determines whether a real image or a virtual image has been observed using depths acquired by the sensor 204 at two different positions. Here, the earlier of the two different positions of the sensor 204 is referred to as a first position, and the later position is referred to as a second position.

[0043] The computer 205 controls the robot 20 so that the position of the sensor 204 is at a first position. The sensor 204 acquires depth information including the depth of at least a portion of the object M at the first position. The reconstruction means 402 acquires information indicating the first position from the computer 205. The reconstruction means 402 also acquires the depth information acquired by the sensor 204 at the first position.

[0044] Next, the computer 205 controls the robot 20 so that the position of the sensor 204 is the second position. The sensor 204 acquires depth information including the depth of a portion at the second position that has a portion overlapping with at least a portion of the object M. The reconstruction means 402 acquires information indicating the second position from the computer 205. The reconstruction means 402 also acquires the depth information acquired by the sensor 204 at the second position.

[0045] When the reconstructing means 402 observes from the second position the same point A as that observed from the first position, if the depth information acquired by the sensor 204 at the first position and the depth information acquired by the sensor 204 at the second position overlap at the point A within the allowable error range as shown in Fig. 8, it determines that a real image has been observed. This real image is a point that indicates part of the outline of the object M.

[0046] Furthermore, when the same point A as that observed from the first position is observed from the second position, the reconstruction means 402 determines that a virtual image has been observed if the depth information acquired by the sensor 204 at the first position and the depth information acquired by the sensor 204 at the second position do not overlap at the point A within the allowable error range, as shown in Fig. 9. This virtual image is an example of noise.

[0047] The reconstruction means 402 generates a three-dimensional model showing the shape of the object M by using only the overlapping points that are within the allowable error range when it is determined that a real image has been observed (i.e., by removing the virtual image).

[0048] As will be described later, the sensor 204 acquires depth information each time it moves. Each time the sensor 204 acquires depth information, the reconstruction means 402 uses the depth information acquired by the sensor 204 to determine whether an image is a real image or a virtual image as described above. The reconstruction means 402 then adds points that overlap within the allowable error range when it is determined that a new real image has been observed to points that overlap within the allowable error range when it is determined that a real image has been observed up to that point. This increases the number of points that indicate the outline of the object M, and the shape of the object M gradually becomes clearer.

[0049] The estimation means 4031 estimates the state of the object M based on the 3D CAD model included in the database 401 and the 3D model generated by the reconstruction means 402 (step S8). The estimation means 4031 determines whether the estimation accuracy of the estimated state of the object M is equal to or greater than a threshold (step S9). If the estimation means 4031 determines that the estimation accuracy of the estimated state of the object M is equal to or greater than the threshold (YES in step S9), the computer 205 controls the robot 20 to grasp the object M in the state estimated by the estimation means 4031 (step S10). Then, the computer 205 ends the series of processes. On the other hand, if the estimation means 4031 determines that the estimation accuracy of the estimated state of the object M is less than the threshold (NO in step S9), the computer 205 proceeds to the process of step S11, which will be described later.

[0050] For example, the estimation unit 4031 calculates an evaluation value based on the 3D model generated by the reconstruction unit 402 and each 3D CAD model included in the database 401. The evaluation value is a value indicating the similarity between the 3D model and the 3D CAD model (i.e., the type of the 3D model (which 3D CAD model the 3D model corresponds to)) and the accuracy of the estimation of the state of the 3D model. Specifically, for example, the estimation unit 4031 calculates the evaluation value by comparing the 3D model generated by the reconstruction unit 402 with each 3D CAD model included in the database 401 (for example, by using ICP (Iterative Closest Point) matching technology). The ICP matching technology is a technology that estimates the state of the object M by repeatedly calculating the alignment of two point clouds. Note that a high evaluation value indicates a high accuracy of the estimation of the state of the object M and at the same time, it also indicates that the type of the object M is estimated. The estimation means 4031 determines whether the calculated evaluation value is equal to or greater than a threshold value. If the estimation means 4031 determines that the calculated evaluation value is less than the threshold value, the process proceeds to step S11, which will be described later. If the estimation means 4031 determines that the calculated evaluation value is equal to or greater than the threshold value, the estimation means 4031 determines that the type of the three-dimensional model is the three-dimensional CAD model used in calculating the evaluation value, and that the state of the three-dimensional model is the state used in calculating the evaluation value. Then, the estimation means 4031 ends the series of processes.

[0051] The execution means 4032 sets parameters for a ray tracing simulation for multiple candidate objects M based on the 3D model generated by the reconstruction means 402 and the state of the object M estimated by the estimation means 4031 (step S11). For example, when the sensor 204 is at two different positions, an earlier position is a first position and a later position is a second position, the execution means 4032 sets multiple positions near the second position. For example, the execution means 4032 may randomly set multiple positions near the second position. Then, the execution means 4032 sets parameters for a ray tracing simulator simulating a real environment so that the sensor 204 is located at each of the multiple set positions, and the orientation includes many points with low image realism and many portions of the object M for which the sensor 204 has not yet acquired depth. The image realism is a measure of the likelihood that the image is a real image. For example, the image realism is indicated as a larger value the more times the reconstruction means 402 determines that a real image has been observed.

[0052] There are multiple possible orientations that include many points with low realism and many portions of the object M for which the sensor 204 has not yet acquired depth information. Therefore, the execution unit 4032 may determine the orientations using the priority determination method described below. For example, the execution unit 4032 identifies a predetermined number of orientations of the sensor 204 that include at least one of points with low realism and portions of the object M for which the sensor 204 has not yet acquired depth information, at each of the multiple positions that have been set (e.g., three directions). For example, assuming that the number of points with low realism is Nreal, the number of points in the real environment for which the depth information has not yet been acquired is Nunknown, and a priority parameter is k, the execution unit 4032 calculates an evaluation value E for each orientation at each of the multiple positions near the second position using Equation (1).

[0053]

[0054] The priority parameter k may be determined from the degree of specular reflection and the geometric shape of the object M to be observed. The priority parameter k may also be determined experimentally. The execution means 4032 specifies, at each of a plurality of positions near the second position, the orientation corresponding to the highest calculated evaluation value E as the orientation at each of the plurality of positions near the second position. Then, the execution means 4032 may set parameters in a ray tracing simulator that simulates a real environment so as to achieve the specified orientation.

[0055] The execution means 4032 sets parameters for a plurality of candidate objects M. Then, the execution means 4032 executes a ray tracing simulation using the set parameters (step S12).

[0056] FIG. 10 is a diagram illustrating processing performed by the execution unit 4032 according to some embodiments of the present disclosure. The thick lines in parts (a) and (b) of FIG. 10 indicate areas determined to be real images. Part (c) of FIG. 10 indicates objects M1 and M2, which are candidates for object M. For example, even if the actual object M is object M1, if only the thick line part in part (a) of FIG. 10 is determined to be a real image, it is impossible to determine whether the actual object M is object M1 or object M2. Therefore, as shown in part (b) of FIG. 10, the sensor 204 is set in an orientation that includes many parts of the object M whose depths have not yet been acquired, so as to increase the number of real images of parts of the object M whose depths have not yet been acquired by the sensor 204.

[0057] In this way, the portion of the object M for which the sensor 204 has not yet acquired depth information is likely to contain important information for determining whether the object M is the object M1 or the object M2 (i.e., determining the type of the object M). Furthermore, the portion of the object M for which the realism is small is likely to contain important information for determining whether the object M is the object M1 or the object M2 (i.e., determining the type of the object M). Therefore, the execution unit 4032 sets parameters in the ray tracing simulator simulating a real environment so that the sensor 204 is positioned at each of the multiple positions that have been set, so that the ray tracing simulator is oriented to include many points with low realism and many portions of the object M for which the sensor 204 has not yet acquired depth information. Note that including many points with low realism and including many portions of the object M for which the sensor 204 has not yet acquired depth information may not be compatible. Therefore, it is possible to prioritize or weight the image based on whether it contains many points with low image quality and many parts of the object M for which the sensor 204 has not yet acquired the depth.

[0058] Based on the results of the simulation by the execution means 4032, the planning means 4033 identifies a position from among the multiple positions set by the execution means 4032 where highly accurate data is likely to be obtained (step S13). For example, the planning means 4033 identifies a position corresponding to a simulation result indicating that many real images can be obtained from among the results of the simulation by the execution means 4032 for each of the multiple positions set by the execution means 4032. Then, the planning means 4033 instructs the computer 205 to move the sensor 204 to the identified position.

[0059] The computer 205 controls the robot 20 to move the sensor 204 to the position specified by the planning means 4033 (step S14). The computer 205 transmits imaging position information, which is information indicating the position of the sensor 204, to the reconstruction means 402. The robot 20 operates in accordance with the control of the computer 205. As a result, the sensor 204 moves to the position specified by the planning means 4033. The sensor 204 acquires depth information including the depth of at least a portion of the object M.

[0060] The reconstruction means 402 acquires the photographing position information transmitted by the computer 205 (step S15). The reconstruction means 402 also acquires the depth information acquired by the sensor 204 from the sensor 204 (step S16).

[0061] The reconstruction means 402 sets the second position in the previous processing as the first position in the current processing, sets the position indicated by the shooting position information newly acquired in the processing of step S15 as the second position in the current processing, and uses the depth information acquired by the sensor 204 at the first position and the second position in the current processing to generate a 3D model of the object M from which noise caused by specular reflection has been removed (step S17). For example, the reconstruction means 402 generates a new 3D model of the object M from which noise caused by specular reflection has been removed by adding data indicating the real image obtained in the current processing to data indicating the real image obtained up to the previous processing. Then, the reconstruction means 402 proceeds to the processing of step S8.

[0062] (Advantages) The processing system 1 according to an embodiment of the present disclosure has been described above. In the processing device 400 of the processing system 1, the estimation unit 4031 (an example of a first identification unit) identifies the type of the object M (an example of an object) from which noise caused by specular reflection has been removed, using data obtained by observing the object M (an example of an object) from multiple viewpoints. The planning unit 4033 (an example of a determination unit) determines an action to be performed on the object M based on the type of the object M identified by the estimation unit 4031.

[0063] The processing device 400 of this processing system 1 can identify the state of a specularly reflective object.

[0064] <Modification of the embodiment> In the processing system 1 according to an embodiment of the present disclosure, the arithmetic device 40 and the computer 205 have been described as separate devices. However, in a modification of the embodiment of the present disclosure, part or all of the arithmetic device 40 may be provided in the computer 205. Alternatively, part or all of the computer 205 may be provided in the arithmetic device 40.

[0065] A processing device 400 according to some embodiments of the present disclosure will be described. Fig. 11 is a diagram illustrating an example of the configuration of the processing device 400 according to some embodiments of the present disclosure. As shown in Fig. 11, the processing device 400 according to some embodiments of the present disclosure includes a first identification unit 501 and a determination unit 502.

[0066] The first identification unit 501 identifies the type of the object from which noise caused by specular reflection has been removed using data obtained by observing the object from multiple viewpoints. The determination unit 502 determines an action to be performed on the object based on the type of the object identified by the first identification unit 501.

[0067] The first identifying means 501 can be realized, for example, by using the function of the estimating means 4031 illustrated in Fig. 4. The determining means 502 can be realized, for example, by using the function of the planning means 4033 illustrated in Fig. 4.

[0068] Next, a process performed by the processing device 400 according to some embodiments of the present disclosure will be described. Fig. 12 is a diagram showing an example of a processing flow of the processing device 400 according to some embodiments of the present disclosure. Here, the process performed by the processing device 400 will be described with reference to Fig. 12.

[0069] The first identification unit 501 identifies the type of the object from which noise caused by specular reflection has been removed using data obtained by observing the object from multiple viewpoints (step S101). The determination unit 502 determines an action to be performed on the object based on the type of the object identified by the first identification unit 501 (step S102).

[0070] Such a processing device 400 can identify the state of a specularly reflecting object.

[0071] The order of the processes in the embodiments of the present disclosure may be changed as long as the processes are performed appropriately.

[0072] Although the embodiments of the present disclosure have been described, the processing system 1, manufacturing apparatus 10, robot 20, control device 30, arithmetic device 40, processing device 400, and other control devices may have a computer device inside. The above-described processing steps are stored in the form of a program on a computer-readable recording medium, and the above processing is performed by the computer reading and executing this program. Specific examples of computers are shown below.

[0073] FIG. 13 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. As shown in FIG. 13 , the computer 5 includes a CPU (Central Processing Unit) 6, a main memory 7, a storage 8, and an interface 9. For example, the processing system 1, the manufacturing apparatus 10, the robot 20, the control device 30, the arithmetic device 40, the processing device 400, and other control devices are each implemented in the computer 5. The operation of each of the processing units described above is stored in the storage 8 in the form of a program. The CPU 6 reads the program from the storage 8, loads it into the main memory 7, and executes the above-described processing in accordance with the program. The CPU 6 also allocates storage areas in the main memory 7 corresponding to each of the storage units described above in accordance with the program.

[0074] Examples of storage 8 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and a semiconductor memory. Storage 8 may be an internal medium directly connected to the bus of computer 5, or an external medium connected to computer 5 via interface 9 or a communication line. Furthermore, if the program is distributed to computer 5 via a communication line, computer 5 that receives the program may load the program into main memory 7 and execute the above-described processing. In at least one embodiment, storage 8 is a non-transitory tangible storage medium.

[0075] The program may also implement some of the functions described above. Furthermore, the program may be a file that can implement the functions described above in combination with a program already stored in the computer device, a so-called differential file (differential program).

[0076] Although several embodiments of the present disclosure have been described, these embodiments are merely examples and do not limit the scope of the disclosure. Various additions, omissions, substitutions, and modifications may be made to these embodiments without departing from the spirit of the disclosure.

[0077] Note that part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0078] (Supplementary Note 1) A processing device comprising: a first identification means for identifying a type of object from which noise caused by specular reflection has been removed using data obtained by observing the object from multiple viewpoints; and a determination means for determining an action to be performed on the object based on the type of the object identified by the first identification means.

[0079] (Supplementary Note 2) The processing device according to Supplementary Note 1, further comprising: an execution means for executing a simulation relating to the specular reflection to identify a viewpoint from among a plurality of candidate viewpoints for observing the location of the object where the noise occurs.

[0080] (Supplementary Note 3) The processing device according to Supplementary Note 2, further comprising: a second specifying means for specifying one viewpoint from among the candidates based on a result of the simulation.

[0081] (Supplementary Note 4) The processing device according to Supplementary Note 3, wherein the second identification means sets the ability to observe a portion of the object that was not observed in the observation as one of the evaluation items, assigns a high evaluation value to it, and identifies the candidate with the highest evaluation value for the evaluation item as the one viewpoint.

[0082] (Supplementary Note 5) The processing device according to Supplementary Note 3 or Supplementary Note 4, wherein the second identification means sets the ability to observe a location of an object that is observed in the observation but where the noise is occurring as one of the evaluation items, assigns a high evaluation value to it, and identifies the candidate with the highest evaluation value for the evaluation item as the one viewpoint.

[0083] (Supplementary Note 6) The processing device according to any one of Supplementary Note 3 to Supplementary Note 5, comprising: an observation means for observing the object; and a first control means for moving the observation means to a position corresponding to the viewpoint identified by the second identification means.

[0084] (Supplementary Note 7) The processing device according to any one of Supplementary Note 1 to Supplementary Note 6, further comprising: second control means for causing the robot to execute the action to be performed on the object.

[0085] (Supplementary Note 8) A processing system comprising: the processing device according to any one of Supplementary Notes 1 to 7; and a robot that operates based on the control of the processing device.

[0086] (Supplementary Note 9) A processing method including: identifying a type of an object from which noise caused by specular reflection has been removed using data obtained by observing the object from multiple viewpoints; and determining an action to be performed on the object based on the identified type of the object.

[0087] (Supplementary Note 10) The processing method according to Supplementary Note 9, comprising: executing a simulation of the specular reflection to identify a viewpoint from among a plurality of candidate viewpoints for observing the location of the object where the noise occurs.

[0088] (Supplementary Note 11) The processing method according to Supplementary Note 10, comprising: identifying one viewpoint from among the candidates based on a result of the simulation.

[0089] (Supplementary Note 12) The processing method according to Supplementary Note 11, comprising: setting the ability to observe a portion of the object that is not observed in the observation as one of the evaluation items, assigning a high evaluation value to the candidate, and identifying the candidate with the highest evaluation value for the evaluation item as the one viewpoint.

[0090] (Supplementary Note 13) The processing method according to Supplementary Note 11 or Supplementary Note 12, comprising: setting the ability to observe a location of an object that is observed in the observation but where the noise is occurring as one of the evaluation items, assigning a high evaluation value to the location, and identifying the candidate with the highest evaluation value for the evaluation item as the one viewpoint.

[0091] (Supplementary Note 14) The processing method according to any one of Supplementary Note 11 to Supplementary Note 13, comprising: observing the object; and moving an observation means for observing the object to a position corresponding to the identified viewpoint.

[0092] (Supplementary Note 15) The processing method according to any one of Supplementary Note 9 to Supplementary Note 14, comprising: causing a robot to perform the action to be performed on the object.

[0093] (Supplementary Note 16) A recording medium storing a program that causes a computer to execute the following steps: identifying a type of an object from which noise caused by specular reflection has been removed using data obtained by observing the object from multiple viewpoints; and determining an action to be performed on the object based on the identified type of the object.

[0094] (Supplementary Note 17) A recording medium according to Supplementary Note 16, storing a program for causing a computer to execute the following: performing a simulation of the specular reflection to identify a viewpoint from among a plurality of candidate viewpoints for observing the location of the object where the noise occurred.

[0095] (Supplementary Note 18) The recording medium according to Supplementary Note 17, storing a program for causing a computer to execute the steps of: identifying one viewpoint from the candidates based on a result of the simulation.

[0096] (Supplementary Note 19) A recording medium according to Supplementary Note 18, storing a program that causes a computer to execute the following: setting the ability to observe a portion of the object that was not observed in the observation as one of the evaluation items, assigning a high evaluation value to the candidate with the highest evaluation value for the evaluation item, and identifying the one viewpoint.

[0097] (Supplementary Note 20) A recording medium according to Supplementary Note 18 or Supplementary Note 19, storing a program that causes a computer to execute the following: setting the ability to observe a portion of an object that is observed in the observation but where the noise is occurring as one of the evaluation items, assigning a high evaluation value to it, and identifying the candidate with the highest evaluation value for the evaluation item as the one viewpoint.

[0098] (Appendix 21) A recording medium according to any one of Appendices 18 to 20, storing a program that causes a computer to execute the following steps: observe the object; and move an observation means that observes the object to a position corresponding to the identified viewpoint.

[0099] (Supplementary Note 22) The recording medium according to any one of Supplementary Note 16 to Supplementary Note 21, storing a program for causing a computer to execute the following: causing a robot to execute an action to be performed on the object.

[0100] According to each aspect of the present disclosure, the state of a specularly reflected object can be identified.

[0101] DESCRIPTION OF SYMBOLS 1 Processing system 5 Computer 6 CPU 7 Main memory 8 Storage 9 Interface 10 Manufacturing device 20 Robot 30 Control device 40 Arithmetic device 50, 50a, 50b Conveyance mechanism 50a, 50b Conveyance system 201 Moving mechanism 202 Elevator 203 Robot arm 204 Sensor 205 Computer 301 Database 302 Control unit 400 Processing device 401 Database 402 Reconstruction means 403 Position determination means 501 First identification means 502 Determination means 2031 Joint 2031 2032 Grasping mechanism 4031 Estimation means 4032 Execution means 4033 Planning means

Claims

1. A processing device comprising: a first identification means for identifying the type of object from which noise caused by specular reflection has been removed using data obtained by observing the object from multiple viewpoints; and a determination means for determining an action to be performed on the object based on the type of object identified by the first identification means.

2. The processing device according to claim 1, further comprising: an execution means for executing a simulation relating to the specular reflection in order to identify a viewpoint from among a plurality of candidate viewpoints for observing the location of the object where the noise occurs.

3. The processing device according to claim 2, further comprising: second specifying means for specifying one viewpoint from among the candidates based on the result of the simulation.

4. The processing device described in claim 3, wherein the second identification means uses the ability to observe a portion of the object that was not observed in the observation as one of the evaluation items, assigns a high evaluation value to it, and identifies the candidate with the highest evaluation value for the evaluation item as the one viewpoint.

5. The processing device described in claim 3, wherein the second identification means uses the ability to observe a portion of an object that is observed in the observation but where the noise is occurring as one of the evaluation items, assigns a high evaluation value to it, and identifies the candidate with the highest evaluation value for the evaluation item as the one viewpoint.

6. A processing device according to claim 3, comprising: an observation means for observing the object; and a first control means for moving the observation means to a position corresponding to the viewpoint identified by the second identification means.

7. The processing device according to claim 1, further comprising: second control means for causing the robot to execute the action to be performed on the object.

8. A processing system comprising: the processing device according to claim 1; and a robot that operates under the control of said processing device.

9. A processing method comprising: identifying the type of object from which noise caused by specular reflection has been removed using data obtained by observing the object from multiple viewpoints; and determining an action to be performed on the object based on the identified type of object.

10. A recording medium storing a program that causes a computer to perform the following steps: identifying the type of an object from which noise caused by specular reflection has been removed using data obtained by observing the object from multiple viewpoints; and determining an action to be performed on the object based on the identified type of the object.

Images