Processing device and program

Abstract

To provide a technique which enable appropriate identification of a surface of an object facing an image capturing device from among multiple surfaces with similar appearances.SOLUTION: In a processing device is provided, comprising a control unit configured to acquire a first image of an object having multiple surfaces with similar appearances in a first posture from an image capturing device, identify a target surface facing the image capturing device from among the multiple surfaces, acquire a second image of the object in a second posture different from the first posture, and identify the target surface based on the first image and the second image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a technique for identifying a surface of an object facing the imaging device side.

Background Art

[0002] Patent Document 1 describes a technique for determining whether the adsorption posture of a component adsorbed by adsorption noise is the reverse of the front and back.

Prior Art Document

Patent Document

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is room for improvement in the technique for identifying the surface of an object facing the imaging device side.

Means for Solving the Problems

[0005] A processing device and a program are disclosed. In one embodiment, the processing device acquires a first captured image of an object having a plurality of surfaces with similar appearances from an imaging device in a first posture, and among the plurality of surfaces Is shown in the first photographed image is provided with a control unit for identifying a target surface facing the imaging device side. The control unit acquires a second captured image of the object in a second posture different from the first posture. The control unit identifies the target surface Which of the plurality of surfaces it is based on the first captured image and the second captured image.

[0006] Also, in one embodiment, the program is a program for causing a computer device to function as the above-described processing device.

Effects of the Invention

[0007] Among a plurality of surfaces of an object that have similar appearances, it is possible to appropriately identify the surface facing the imaging device side.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Modes for Carrying Out the Invention

[0009] FIG. 1 is a schematic diagram showing an example of the configuration of the processing device 1. The processing device 1 can identify the surface of the object 10 facing the imaging device 8 side based on the image generated by the imaging device 8 that images the object 10. FIG. 2 is a schematic diagram showing an example of how the object 10 is imaged by the imaging device 8.

[0010] As shown in FIG. 2, the imaging device 8 images, for example, the object 10 held by the robot 9. The imaging device 8 is also called a camera. The imaging device 8 can generate, for example, a two-dimensional color image. The color image is also called an RGB image, for example. Hereinafter, the image generated by the imaging device 8 may be simply referred to as an imaging image.

[0011] The robot 9 includes, for example, an arm 90 and an end effector 91 connected to the arm 90. The arm 90 has a plurality of joints. The posture of the arm 90 changes as the rotation amount of at least one of the plurality of joints changes. Then, as the posture of the arm 90 changes, the position and posture of the end effector 91 change.

[0012] The end effector 91 can hold the object 10. It can be said that the end effector 91 is a holding mechanism for holding the object 10. The position and posture of the object 10 held by the end effector 91 change according to the change in the position and posture of the end effector 91. It can also be said that the position and posture of the object 10 change according to the movement of the arm 90. The end effector 91 may, for example, adsorb and hold the object 10 or hold the object 10 with a plurality of fingers.

[0013] The object 10 is disposed, for example, inside a container. The robot 9 holds the object 10 inside the container with the end effector 91, moves the arm 90, and transports the held object 10 in front of the imaging device 8. That is, the robot 9 changes the posture of the arm 90 and transports the object 10 to a position where the held object 10 is imaged by the imaging device 8. At this time, the robot 9 arranges the object 10, for example, on the outer surface of the object 10 such that the surface opposite to the surface held by the end effector 91 faces the imaging device 8 side. The processing device 1 identifies the surface facing the imaging device 8 side on the outer surface of the object 10 based on the captured image in which the object 10 appears. By identifying the surface facing the imaging device 8 side on the outer surface of the object 10, the holding posture of the object 10 by the robot 9 is identified. In this example, since the surface opposite to the surface held by the end effector 91 faces the imaging device 8 side on the outer surface of the object 10, by identifying the surface facing the imaging device 8 side on the outer surface of the object 10, which surface of the object 10 the robot 9 is holding is identified. That is, the processing device 1 can identify the surface facing the imaging device 8 side on the outer surface of the object 10 and determine that the robot 9 is holding the surface opposite to the identified surface.

[0014] The processing device 1 can, for example, control the robot 9. The processing device 1 may control the operation of the robot 9 according to the holding posture of the object 10 by the robot 9. For example, when the end effector 91 holds a certain surface of the object 10, the processing device 1 moves the object 10 to a certain location by the robot 9, and when the end effector 91 holds another surface of the object 10, the processing device 1 moves the object 10 to another location by the robot 9. Note that the examples of the operation of the robot 9 are not limited to this. Also, the object 10 may be held by means other than the robot 9.

[0015] FIG. 3 is a schematic view showing an example of the object 10. The object 10 is, for example, a plate-like member bent in a U-shape. The object 10 has, for example, a first main surface 101 and a second main surface 102 facing each other. The outer shapes of the first main surface 101 and the second main surface 102 form, for example, a somewhat elongated trapezoid with a height larger than the lower base. The first main surface 101 is bent in a U-shape so as to protrude outward at the central portion in its longitudinal direction (in other words, the height direction of the trapezoid formed by the outer shape of the first main surface 101). The second main surface 102 is bent in a U-shape so as to be recessed inward at the central portion in its longitudinal direction.

[0016] On the far left of FIG. 3, an example of how the object 10 looks when viewed from the side of the first main surface 101 is shown. In the third position from the left in FIG. 3, an example of how the object 10 looks when viewed from the side of the second main surface 102 is shown. In the second position from the left in FIG. 3, an example of how the object 10 shown on the far left looks when viewed from the right side of the figure is shown. On the far right of FIG. 3, an example of how the object 10 shown in the third position from the left looks when viewed from the right side of the figure is shown. Hereafter, when explaining the shape of the object 10, the top, bottom, right, and left of the object 10 mean the top, bottom, right, and left when the object 10 is viewed in the state shown on the far left of FIG. 3, respectively. Also, the first main surface 101 may be called the front surface 101, and the second main surface 102 may be called the back surface 102. The front surface 101 can also be said to be the front side surface 101.

[0017] In addition to the front surface 101 and the back surface 102, the object 10 includes a right surface 103, a left surface 104, an upper surface 105, and a lower surface 106. The right surface 103, the left surface 104, the upper surface 105, and the lower surface 106 can also be said to be the right end surface 103, the left end surface 104, the upper end surface 105, and the lower end surface 106, respectively. The object 10 has a shape that is symmetric about the left and right and has a small thickness. Hereafter, when there is no need to particularly distinguish between the right surface 103 and the left surface 104, they may each be called a side surface. The right surface 103 can also be said to be the right side surface 103, and the left surface 104 can also be said to be the left side surface 104.

[0018] When the robot 9 holds the object 10 in the container, for example, it holds either the front surface 101 or the back surface 102. For example, when the object 10 is arranged in the container such that the front surface 101 of the object 10 faces the opening side (upper side) of the container, the robot 9 holds the front surface 101 of the object 10. On the other hand, when the object 10 is arranged in the container such that the back surface 102 of the object 10 faces the opening side of the container, the robot 9 holds the back surface 102 of the object 10. When the robot 9 holds the object 10 in the container, it does not recognize which of the front surface 101 and the back surface 102 of the object 10 faces the opening side of the container. Therefore, when the robot 9 holds the object 10, it does not recognize which surface of the front surface 101 and the back surface 102 it is holding.

[0019] The object 10 has a plurality of surfaces whose appearances are similar to each other. When the plurality of surfaces are viewed without changing the posture of the object 10, they have surfaces with similar appearances to each other. Each of the plurality of surfaces of the object 10 that have similar appearances to each other may be held by the robot 9, for example. The processing device 1 identifies the surface (also referred to as the target surface) facing the imaging device 8 side among the plurality of surfaces of the object 10 that have similar appearances to each other based on the captured image in which the object 10 appears. For example, when the posture of the object is not uniquely determined when the object is photographed by the imaging device 8, the object may be considered as the object 10 having a plurality of surfaces with similar appearances to each other.

[0020] In this example, the front surface 101 and the back surface 102 of the object 10 are similar to each other. The appearance of the object 10 seen from the front surface 101 side and the appearance of the object 10 seen from the back surface 102 side with the same posture are similar to each other. In other words, the appearance of the object 10 when viewed from the front surface 101 side and the appearance of the object 10 when viewed from the back surface 102 side with the same posture are similar to each other. The processing device 1 identifies the target surface facing the imaging device 8 side on the front surface 101 and the back surface 102 based on the captured image in which the object 10 appears.

[0021] Note that the shape of the object 10 is not limited to the example in FIG. 3. For example, the object 10 may have three or more surfaces with similar appearances.

[0022] Hereafter, a plurality of surfaces of the object 10 that have similar appearances may be referred to as a plurality of specific surfaces. It can also be said that the processing device 1 identifies the specific surface facing the imaging device 8 side among the plurality of specific surfaces. The appearances of the object 10 seen from each specific surface are similar to each other.

[0023] When the robot 9 holds the object 10, the object 10 is arranged such that the surface opposite to the held surface faces the imaging device 8 side. Therefore, when the front surface 101 of the object 10 is held by the robot 9, the object 10 is arranged such that the back surface 102 faces the imaging device 8 side. The state where the back surface 102 of the object 10 faces the imaging device 8 side can be regarded as the state where the imaging device 8 is located on the back surface 102 side. When the back surface 102 of the object 10 faces the imaging device 8 side, the imaging device 8 images the object 10 from the back surface 102 side. On the other hand, as shown in FIG. 2, when the back surface 102 of the object 10 is held by the robot 9, the object 10 is arranged such that the front surface 101 faces the imaging device 8 side. The state where the front surface 101 of the object 10 faces the imaging device 8 side can be regarded as the state where the imaging device 8 is located on the front surface 101 side. When the front surface 101 of the object 10 faces the imaging device 8 side, the imaging device 8 images the object 10 from the front surface 101 side.

[0024] When the front surface 101 of the object 10 is held by the robot 9, in the processing device 1, the back surface 102 is identified as the surface facing the imaging device 8 side, that is, as the target surface. On the other hand, when the back surface 102 of the object 10 is held by the robot 9, in the processing device 1, the front surface 101 is identified as the surface facing the imaging device 8 side, that is, as the target surface.

[0025] The processing device 1 is, for example, a type of computer device. The processing device 1 can not only identify the target surface of the object 10 based on the captured image, but also control the robot 9. The processing device 1 also functions as a robot control device for controlling the robot 9. Note that, separately from the processing device 1, a robot control device for controlling the robot 9 may be provided.

[0026] As shown in FIG. 1, the processing device 1 includes, for example, a control unit 2, a storage unit 3, an interface 4, and an interface 5. The processing device 1 can also be referred to as a processing circuit, for example.

[0027] The interface 4 can communicate with the imaging device 8. The control unit 2 can acquire the image generated by the imaging device 8 through the interface 4. The interface 4 can also be referred to as an interface circuit, a communication unit, or a communication circuit, for example. The interface 4 may perform wired communication or wireless communication with the imaging device 8.

[0028] The interface 5 can communicate with the robot 9. The control unit 2 can control the robot 9 through the interface 5. The interface 5 can also be referred to as an interface circuit, a communication unit, or a communication circuit, for example. The interface 5 may perform wired communication or wireless communication with the robot 9.

[0029] The control unit 2 can comprehensively manage the operation of the processing device 1 by controlling other components of the processing device 1. The control unit 2 can also be referred to as a control circuit, for example. The control unit 2 includes at least one processor to provide control and processing capabilities for executing various functions, as described in more detail below.

[0030] According to various embodiments, at least one processor may be implemented as a single integrated circuit (IC), or as multiple communicatively connected integrated circuit ICs and / or discrete circuits. The at least one processor can be implemented according to various known techniques.

[0031] In one embodiment, a processor includes one or more circuits or units configured to perform one or more data calculation procedures or processes, for example, by executing instructions stored in an associated memory. In other embodiments, the processor may be firmware (e.g., discrete logic components) configured to perform one or more data calculation procedures or processes.

[0032] According to various embodiments, a processor may include one or more processors, controllers, microprocessors, microcontrollers, application specific integrated circuits (ASICs), digital signal processing devices, programmable logic devices, field programmable gate arrays, or any combination of these devices or configurations, or combinations of other known devices and configurations, and may perform the functions described below.

[0033] The control unit 2 may include, for example, a CPU (Central Processing Unit) as a processor. The storage unit 3 may include a non-transitory recording medium readable by the CPU of the control unit 2, such as a ROM (Read Only Memory) and a RAM (Random Access Memory). For example, a program 30 for controlling the processing device 1 is stored in the storage unit 3. Various functions of the control unit 2 are realized, for example, by the CPU of the control unit 2 executing the program 30 in the storage unit 3.

[0034] Note that the configuration of the control unit 2 is not limited to the above example. For example, the control unit 2 may include a plurality of CPUs. Further, the control unit 2 may include at least one DSP (Digital Signal Processor). Also, all or some of the functions of the control unit 2 may be realized by a hardware circuit that does not require software for realizing the functions. Further, the storage unit 3 may include a non-transitory computer-readable recording medium other than ROM and RAM. The storage unit 3 may include, for example, a small hard disk drive and an SSD (Solid State Drive).

[0035] The control unit 2 is capable of performing a target surface identification process for identifying a target surface of the object 10 (that is, the surface facing the imaging device 8 side) based on the captured image. The control unit 2 includes, for example, an estimation unit 20 and an identification unit 21. The estimation unit 20 and the identification unit 21 are functional blocks formed in the control unit 2, for example, when the CPU of the control unit 2 executes the program 30 in the storage unit 3. Note that all or some of the functions of the estimation unit 20 or some of the functions of the identification unit 21 may be realized by a hardware circuit that does not require software for realizing the functions. The same applies to the identification unit 21.

[0036] The estimation unit 20 estimates the posture of the object 10 based on the captured image. The identification unit 21 identifies the surface facing the imaging device 8 side (that is, the target surface) among a plurality of surfaces of the object 10 that have similar appearances (that is, a plurality of specific surfaces). In this example, the identification unit 21 identifies the surface facing the imaging device 8 side on the front surface 101 and the back surface 102 that have similar appearances to each other. Hereinafter, the state in which the front surface 101 of the object 10 faces the imaging device 8 side may be referred to as the front-facing state, and the state in which the back surface 102 of the object 10 faces the imaging device 8 side may be referred to as the back-facing state. It can also be said that the identification unit 21 determines whether the object 10 is in the front-facing state or the back-facing state. The fact that the object 10 is in the front-facing state can also be said to be a state in which the imaging device 8 captures the object 10 from the front surface 101 side. Also, the fact that the object 10 is in the back-facing state can also be said to be a state in which the imaging device 8 captures the object 10 from the back surface 102 side.

[0037] <An example of target surface identification processing> In this example, the front surface 101 and the back surface 102 of the object 10 are similar in appearance to each other. Therefore, with only one captured image in which the object 10 appears, it is difficult for the control unit 2 to determine whether the object 10 is facing forward or backward.

[0038] For example, consider a case where only the front surface 101 of the outer surface of the object 10 appears in the captured image. In this case, if the control unit 2 performs image processing on the captured image and can identify that the front surface 101 appears in the captured image, the control unit 2 can identify that the object 10 is facing forward. However, since the front surface 101 and the back surface 102 are similar in appearance to each other, the control unit 2 may not be able to identify whether the surface appearing in the captured image is the front surface 101 or the back surface 102. Therefore, the control unit 2 may not be able to identify that the object 10 is facing forward.

[0039]

[0040] Therefore, in the target surface identification processing of this example, the target surface of the object 10 is not identified using only one captured image, but is identified based on two captured images (the first image and the second image) in which the postures of the object 10 appearing therein are different from each other. Thereby, the identification accuracy of the target surface can be improved.FIG. 4 is a flowchart showing an example of the target surface identification process executed by the control unit 2. As shown in FIG. 4, in step s1, the control unit 2 controls the robot 9 that holds the object 10 to move the object 10 in front of the imaging device 8. At this time, the robot 9 arranges the object 10 in front of the imaging device 8 so that only the surface on the side opposite to the holding surface on the outer surface of the object 10 is imaged by the imaging device 8 under the control of the control unit 2. For example, when the robot 9 holds the back surface 102, the object 10 is arranged so that only the front surface 101 is imaged by the imaging device 8. In this case, in the captured image, the object 10 appears as shown on the far left in FIG. 3. That is, when the object 10 is viewed from the imaging device 8, the object 10 appears as shown on the far left in FIG. 3. On the other hand, when the robot 9 holds the front surface 101, the object 10 is arranged so that only the back surface 102 is imaged by the imaging device 8. In this case, in the captured image, the object 10 appears as shown third from the left in FIG. 3. That is, when the object 10 is viewed from the imaging device 8, the object 10 appears as shown third from the left in FIG. 3.

[0041] Next, in step s2, the control unit 2 causes the imaging device 8 to image the object 10. The posture of the object 10 as viewed from the imaging device 8 when the object 10 is imaged in step s2 is referred to as the first posture. The first posture can be said to be the posture of the object 10 when the object 10 arranged so that only the surface on the side opposite to the surface held by the robot 9 on the outer surface of the object 10 is imaged by the imaging device 8 is viewed from the imaging device 8. By executing step s2, the imaging device 8 images the object 10 in the first posture. The imaging device 8 generates a captured image (also referred to as a first captured image) in which the object 10 in the first posture appears and outputs it to the control unit 2. When the robot 9 holds the back surface 102 of the object 10, the front-facing object 10 appears in the first captured image, and specifically, only the front surface 101 of the outer surface of the object 10 appears. On the other hand, when the robot 9 holds the front surface 101 of the object 10, the back-facing object 10 appears in the first captured image, and specifically, only the back surface 102 of the outer surface of the object 10 appears.

[0042] The estimation unit 20 is a plurality of types of posture estimation processes for estimating the posture of the object 10 based on the captured images, each corresponding to a plurality of specific surfaces, and can execute a plurality of types of posture estimation processes suitable for the case where the corresponding specific surface faces the imaging device 8 side. After step s2, in step s3, the estimation unit 20 executes a plurality of types of posture estimation processes based on the first captured image. Hereinafter, each of the plurality of estimated postures estimated by the plurality of types of posture estimation processes based on the first captured image is referred to as a first estimated posture.

[0043] In this example, the estimation unit 20 can execute a posture estimation process (also referred to as a front surface posture estimation process) that corresponds to the front surface 101 and is suitable for the case where the front surface 101 faces the imaging device 8 side. Further, the estimation unit 20 can execute a posture estimation process (also referred to as a back surface posture estimation process) that corresponds to the back surface 102 and is suitable for the case where the back surface 102 faces the imaging device 8 side. When the front surface 101 of the object 10 faces the imaging device 8 side, in other words, when the imaging device 8 captures the object 10 from the front surface 101 side, the possibility of appropriately estimating the posture of the object 10 is increased by executing the front surface posture estimation process. On the other hand, when the back surface 102 of the object 10 faces the imaging device 8 side, in other words, when the imaging device 8 captures the object 10 from the back surface 102 side, the possibility of appropriately estimating the posture of the object 10 is increased by executing the back surface posture estimation process. When the object 10 is facing forward, the posture estimation accuracy of the object 10 is improved when the front surface posture estimation process is executed rather than when the back surface posture estimation process is executed. And when the object 10 is facing backward, the posture estimation accuracy of the object 10 is improved when the back surface posture estimation process is executed rather than when the front surface posture estimation process is executed. Hereinafter, the front surface posture estimation process may be simplified and referred to as the front surface process, and the back surface posture estimation process may be simplified and referred to as the back surface process.

[0044] In the posture estimation process, for example, template matching is performed in which a captured image is compared with a plurality of templates each associated with the posture of the object 10. A template, also called a template image, is a captured image generated in advance by the imaging device 8. The storage unit 3 stores a plurality of front templates used in the front processing and a plurality of back templates used in the back processing.

[0045] The plurality of front templates are composed of a number of captured images obtained by the imaging device 8 capturing the object 10 from various angles on the front surface 101 side. The plurality of back templates are composed of a number of captured images obtained by the imaging device 8 capturing the object 10 from various angles on the back surface 102 side.

[0046] For each template, the posture of the object 10 when the template is generated by the imaging device 8 is associated therewith. That is, for each template, the posture of the object 10 shown in the template is associated therewith. The posture associated with the template is represented, for example, in the coordinate system of the imaging device 8 (also referred to as the imaging device coordinate system). The imaging device coordinate system is a three-dimensional orthogonal coordinate system set in the imaging device 8. That is, the imaging device coordinate system is a coordinate system that represents the real space as seen from the imaging device 8 as a coordinate space.

[0047] In the posture estimation process, when the estimation unit 20 compares the captured image with the template, the estimation unit 20 performs image processing including binarization processing on the captured image and the template, extracts the contour of the object 10 shown in the captured image as the first contour, and extracts the contour of the object 10 shown in the template as the second contour. The contour of the object 10 is also called the outer contour. Then, the estimation unit 20 compares the first contour and the second contour and calculates the similarity between the two. This similarity represents the similarity between the captured image and the template. The similarity between the captured image and the template is also called the matching score. Hereinafter, the similarity between the captured image and the template may be referred to as the matching score.

[0048] In the front-side posture estimation process, when the estimation unit 20 compares the captured image with a plurality of front-side templates, it obtains the matching scores between the captured image and each of the plurality of front-side templates as described above. Then, the estimation unit 20 sets the posture associated with the front-side template having the highest matching score with the captured image among the plurality of front-side templates as the estimated posture of the object 10 shown in the captured image. Similarly, in the back-side posture estimation process, when the estimation unit 20 compares the captured image with a plurality of back-side templates, it obtains the matching scores between the captured image and each of the plurality of back-side templates. Then, the estimation unit 20 sets the posture associated with the back-side template having the highest matching score with the captured image among the plurality of back-side templates as the estimated posture of the object 10 shown in the captured image.

[0049] In step s3, the estimation unit 20 performs a front-side posture estimation process and a back-side posture estimation process based on the first captured image obtained in step s2. In the front-side posture estimation process based on the first captured image, the estimation unit 20 obtains the matching scores between the first captured image and each of the plurality of front-side templates. Then, the estimation unit 20 sets the posture associated with the front-side template having the highest matching score with the first captured image among the plurality of front-side templates as the first estimated posture of the object 10 shown in the first captured image. This first estimated posture is referred to as the first estimated posture when using the front-side process.

[0050] Similarly, in the back-side posture estimation process based on the first captured image, the estimation unit 20 obtains the matching scores between the first captured image and each of the plurality of back-side templates. Then, the estimation unit 20 sets the posture associated with the back-side template having the highest matching score with the first captured image among the plurality of back-side templates as the first estimated posture of the object 10 shown in the first captured image. This first estimated posture is referred to as the first estimated posture when using the back-side process. The first estimated posture is represented in the imaging device coordinate system.

[0051] At the stage when step s3 is executed, the control unit 2 does not grasp which of the front surface 101 and the back surface 102 is facing the imaging device 8 side. That is, the control unit 2 does not grasp whether the object 10 is facing forward or backward. As described above, the execution of the front-facing posture estimation process by the estimation unit 20 can also be regarded as the estimation unit 20 assuming that the front surface 101 of the object 10 is facing the imaging device 8 side (in other words, assuming that the object 10 is facing forward) and estimating the posture of the object 10. Further, the execution of the back-facing posture estimation process by the estimation unit 20 can also be regarded as the estimation unit 20 assuming that the back surface 102 of the object 10 is facing the imaging device 8 side (in other words, assuming that the object 10 is facing backward) and estimating the posture of the object 10.

[0052] For example, consider a case where the object 10 is facing forward and only the front surface 101 of the outer surface of the object 10 appears in the first captured image. In this case, the first estimated posture when using the front-facing process is a posture close to the actual posture of the object 10 as seen from the imaging device 8 (that is, the first posture). On the other hand, the first estimated posture when using the back-facing process is a posture significantly different from the actual posture of the object 10 as seen from the imaging device 8. In this example, since the front surface 101 and the back surface 102 are similar in appearance to each other, the first estimated posture when using the back-facing process is the same as the first estimated posture obtained by performing the back-facing process when only the back surface 102 of the outer surface of the object 10 appears in the first captured image. The first estimated posture when using the back-facing process is a posture close to the posture of the object 10 when the object 10 arranged so that only the back surface 102 is imaged by the imaging device 8 is seen from the imaging device 8.

[0053] Also, consider the case where the object 10 is facing backward and only the back surface 102 of the outer surface of the object 10 is shown in the first captured image. In this case, the first estimated posture when using the back-side processing is a posture close to the actual posture of the object 10 as seen from the imaging device 8. On the other hand, the first estimated posture when using the front-side processing is a posture significantly different from the actual posture of the object 10 as seen from the imaging device 8. The first estimated posture when using the front-side processing is the same as the first estimated posture obtained by performing the front-side processing when only the front surface 101 of the outer surface of the object 10 is shown in the first captured image. The first estimated posture when using the front-side processing is a posture close to the posture of the object 10 when the object 10 arranged such that only the front surface 101 is imaged by the imaging device 8 is seen from the imaging device 8.

[0054] In step s3, when the first estimated posture when using the front-side processing and the first estimated posture when using the back-side processing are obtained, in step s4, the control unit 2 rotates the object 10 with the robot 9. In step s4, the robot 9 moves the arm 90 to rotate the object 10 by a predetermined angle W around a predetermined rotation axis J (see FIG. 2). The object 10 rotates together with the end effector 91. That is, the object 10 and the end effector 91 rotate integrally. In the robot 9, due to the movement of the arm 90, the end effector 91 that holds the object 10 rotates by an angle W around the rotation axis J, so that the object 10 rotates by an angle W around the rotation axis J. The posture of the object 10 as seen from the imaging device 8 after the object 10 has rotated by an angle W around the rotation axis J is referred to as the second posture. Thereafter, in step s5, the control unit 2 causes the imaging device 8 to image the object 10 in the second posture. The captured image generated by the imaging device 8 in step s5 is referred to as the second captured image. The second captured image shows the object 10 in the second posture rotated from the first posture.

[0055] In the present disclosure, the object 10 and the imaging device 8 only need to rotate relative to each other. As will be described later, the imaging device 8 may rotate, or both the object 10 and the imaging device 8 may rotate. Further, a plurality of imaging devices 8 may be prepared, and the plurality of imaging devices 8 may be arranged such that the angle when the plurality of imaging devices 8 and the object 10 are connected by a virtual straight line becomes a predetermined angle W. A first imaging image may be acquired by the first imaging device 8, and a second imaging image may be acquired by the second imaging device 8.

[0056] FIGS. 5 and 6 are schematic views showing an example of the state in which the object 10 rotates by an angle W around the rotation axis J. On the left side of FIG. 5, the state of the object 10 in the first attitude facing outward as seen from the imaging device 8 is shown. On the right side of FIG. 5, the state of the object 10 in the second attitude facing outward as seen from the imaging device 8 is shown. That is, on the right side of FIG. 5, an example of the state of the object 10 in the first attitude facing outward after rotating by a predetermined angle W around the rotation axis J as seen from the imaging device 8 is shown. On the left side of FIG. 6, the state of the object 10 in the first attitude facing inward as seen from the imaging device 8 is shown. On the right side of FIG. 6, the state of the object 10 in the second attitude facing inward as seen from the imaging device 8 is shown. That is, on the right side of FIG. 6, an example of the state of the object 10 in the first attitude facing inward after rotating by a predetermined angle W around the rotation axis J as seen from the imaging device 8 is shown. The state of the object 10 as seen from the imaging device 8 can also be said to be the state of the object 10 in the imaging image in which the object 10 is imaged.

[0057] The rotation axis J is set, for example, to pass through the object 10. Further, the rotation axis J is set, for example, in a direction along the vertical direction of the object 10 when the object 10 in the first posture and the rotation axis J are viewed from the imaging device 8. Further, the rotation axis J is set to pass through the center in the left - right direction of the object 10 when the object 10 in the first posture and the rotation axis J are viewed from the imaging device 8. The rotation axis J is set, for example, in a direction perpendicular to the optical axis 80 (see FIG. 2) of the imaging device 8, or in a direction substantially perpendicular to the optical axis 80. The control unit 2 sets the rotation axis J based on the first captured image. Further, assuming that each specific surface faces the imaging device 8, the rotation axis J may be set in advance so that when the object 10 is rotated around the rotation axis J, the appearances of the object 10 after rotation are as different as possible from each other on each specific surface. Hereinafter, simply referring to the rotation of the object 10 means the rotation around the rotation axis J of the object 10.

[0058] The angle W is set to a value such that when the object 10 in the first posture with a certain surface (front surface 101 or back surface 102) facing the imaging device 8 side rotates by the angle W, only the certain surface and the side surface (right side surface 103 or left side surface 104) connected to the certain surface among the outer surfaces of the object 10 are imaged by the imaging device 8. That is, the angle W is set to a value such that when the object 10 in the first posture with a certain surface facing the imaging device 8 side rotates by the angle W, the certain surface faces the imaging device 8 side and the side surface connected to the certain surface is imaged by the imaging device 8.

[0059] In the present disclosure, the clockwise rotation angle of the rotation axis J when viewed from above the object 10 is represented by + (plus). Also, the counterclockwise rotation angle of the rotation axis J when viewed from above the object 10 is represented by - (minus). In step s4, the robot 9 rotates the object 10 by an angle +W around the rotation axis J, for example, to change the posture of the object 10 from the first posture to the second posture. Note that the clockwise rotation angle of the rotation axis J when viewed from above the object 10 may be represented by - (minus), and the counterclockwise rotation angle of the rotation axis J when viewed from above the object 10 may be represented by + (plus).

[0060] When the object 10 in the front-facing first posture rotates by an angle +W, as shown in FIG. 5, only the front surface 101 and the right surface 103 (in other words, the right side surface 103) of the outer surface of the object 10 can be seen from the imaging device 8. Therefore, in step s5, when the object 10 in the front-facing second posture is imaged by the imaging device 8, only the front surface 101 and the right surface 103 of the outer surface of the object 10 are shown in the second captured image.

[0061] When the object 10 in the back-facing first posture rotates by an angle +W, as shown in FIG. 6, only the back surface 102 and the left surface 104 (in other words, the left side surface 104) of the outer surface of the object 10 can be seen from the imaging device 8. Therefore, in step s5, when the object 10 in the back-facing second posture is imaged by the imaging device 8, only the back surface 102 and the left surface 104 of the outer surface of the object 10 are shown in the second captured image.

[0062] As shown in FIGS. 5 and 6, the state of viewing the object 10 in the front-facing first posture from the imaging device 8 and the state of viewing the object 10 in the back-facing first posture from the imaging device 8 are similar to each other. That is, the state of the object 10 in the front-facing first posture in the first captured image and the state of the object 10 in the back-facing first posture in the first captured image are similar to each other. On the other hand, the state of viewing the object 10 in the front-facing second posture from the imaging device 8 and the state of viewing the object 10 in the back-facing second posture from the imaging device 8 are significantly different from each other. That is, the state of the object 10 in the front-facing second posture in the second captured image and the state of the object 10 in the back-facing second posture in the second captured image are significantly different from each other.

[0063] Thus, no matter which specific surface faces the imaging device 8 when the first captured image is generated, the appearance of the object 10 captured in the first captured image will be similar. And depending on the specific surface that faces the imaging device 8 when the first captured image is generated, the appearance of the object 10 captured in the second captured image will be very different. For example, no matter whether the specific surface that faces the imaging device 8 when the first captured image is generated is the front surface 101 or the back surface 102, the appearance of the object 10 captured in the first captured image will be similar (the left side of FIG. 5 and the left side of FIG. 6). And the appearance of the object 10 captured in the second captured image when the specific surface that faces the imaging device 8 when the first captured image is generated is the front surface 101 (the right side of FIG. 5) and the appearance of the object 10 captured in the second captured image when the specific surface that faces the imaging device 8 when the first captured image is generated is the back surface 102 (the right side of FIG. 6) are very different from each other.

[0064] After step s5, in step s6, the estimation unit 20 executes a plurality of types of posture estimation processes based on the second captured image in the same manner as in step s3. Hereinafter, each of the plurality of estimated postures estimated by each of the plurality of types of posture estimation processes based on the second captured image is referred to as a second estimated posture.

[0065] In this example, in step s6, the estimation unit 20 performs a front-side posture estimation process and a back-side posture estimation process based on the second captured image. In the front-side posture estimation process based on the second captured image, the estimation unit 20 obtains a matching score with the second captured image for each of the plurality of front-side templates. Then, the estimation unit 20 sets the posture associated with the front-side template having the highest matching score with the second captured image among the plurality of front-side templates as the second estimated posture of the object 10 captured in the second captured image. This second estimated posture is referred to as the second estimated posture when using the front-side process.

[0066] Similarly, in the backside posture estimation process based on the second captured image, the estimation unit 20 obtains a matching score between the second captured image and each of a plurality of backside templates. Then, the estimation unit 20 sets the posture associated with the backside template having the highest matching score with the second captured image among the plurality of backside templates as the second estimated posture of the object 10 shown in the second captured image. This second estimated posture is referred to as the second estimated posture when using the backside process. The second estimated posture is represented in the imaging device coordinate system.

[0067] For example, consider the case where the object 10 in the front-facing second posture (see FIG. 5) is shown in the second captured image. In this case, the second estimated posture when using the frontside process is a posture close to the actual posture of the object 10 as seen from the imaging device 8 (that is, the front-facing second posture). On the other hand, the second estimated posture when using the backside process is a posture significantly different from the actual posture of the object 10 as seen from the imaging device 8. Here, the posture of the object 10 when the object 10 in the first posture is rotated by an angle -W around the rotation axis J and then viewed from the imaging device 8 is referred to as the third posture. The front-facing third posture is the posture of the object 10 when the object 10 in the front-facing first posture is rotated by an angle -W around the rotation axis J and then viewed from the imaging device 8. The back-facing third posture is the posture of the object 10 when the object 10 in the back-facing first posture is rotated by an angle -W around the rotation axis J and then viewed from the imaging device 8. FIG. 7 is a schematic diagram showing the state of the object 10 in the front-facing second posture as seen from the imaging device 8 (left side of FIG. 7) and the state of the object 10 in the back-facing third posture as seen from the imaging device 8 (right side of FIG. 7). As shown in FIG. 7, in this example, the state of the object 10 in the front-facing second posture as seen from the imaging device 8 and the state of the object 10 in the back-facing third posture as seen from the imaging device 8 are particularly similar in terms of the points of the outer contour. That is, the front-facing second posture and the back-facing third posture are similar to each other. Therefore, the second estimated posture obtained by using the state of the front-facing second posture in the backside posture estimation process, which is the second estimated posture when using the backside process, is a posture close to the back-facing third posture.

[0068] Next, consider the case where the object 10 in the second, rear-facing posture (see FIG. 6) appears in the second captured image. In this case, the second estimated posture when using the rear-side processing is a posture close to the actual posture of the object 10 as seen from the imaging device 8 (i.e., the second, rear-facing posture). On the other hand, the second estimated posture when using the front-side processing is a posture significantly different from the actual posture of the object 10 as seen from the imaging device 8. FIG. 8 is a schematic diagram showing how the object 10 in the third, front-facing posture is seen from the imaging device 8 (left side of FIG. 8) and how the object 10 in the second, rear-facing posture is seen from the imaging device 8 (right side of FIG. 8). As shown in FIG. 9, in this example, how the object 10 in the second, rear-facing posture is seen from the imaging device 8 and how the object 10 in the third, front-facing posture is seen from the imaging device 8 are particularly similar in terms of the outer contour points. That is, the second, rear-facing posture and the third, front-facing posture are similar to each other. Therefore, the second estimated posture at the time of front-side processing obtained by using the front-side posture estimation process for the state of the second, rear-facing posture is a posture close to the third, front-facing posture.

[0069] In step S6, when the second estimated posture at the time of front-side processing and the second estimated posture at the time of rear-side processing are obtained, in step S7, the specifying unit 21 obtains a plurality of first rotation estimated postures obtained by rotating the plurality of first estimated postures obtained in step S3 in the same manner as when the posture of the object 10 rotates from the first posture to the second posture. In this example, the specifying unit 21 obtains a first rotation estimated posture (also referred to as the first rotation estimated posture at the time of front-side processing) obtained by rotating the first estimated posture at the time of front-side processing in the same manner as when the posture of the object 10 rotates from the first posture to the second posture. Further, the specifying unit 21 obtains a first rotation estimated posture (also referred to as the first rotation estimated posture at the time of rear-side processing) obtained by rotating the first estimated posture at the time of rear-side processing in the same manner as when the posture of the object 10 rotates from the first posture to the second posture. The specifying unit 21 sets, in the imaging device coordinate system, the posture obtained by rotating the first estimated posture at the time of front-side processing by an angle +W around the rotation axis J as the first rotation estimated posture at the time of front-side processing. Further, the specifying unit 21 sets, in the imaging device coordinate system, the posture obtained by rotating the first estimated posture at the time of rear-side processing by an angle +W around the rotation axis J as the first rotation estimated posture at the time of rear-side processing.

[0070] For example, consider a case where the object 10 in the front-facing first posture is shown in the first captured image. In this case, as described above, the first estimated posture when using the front-side processing is a posture close to the front-facing first posture. Therefore, similar to the case where the posture of the object 10 rotates from the first posture to the second posture, the first rotation estimated posture in which the first estimated posture when using the front-side processing rotates, that is, the first rotation estimated posture when using the front-side processing is a posture close to the front-facing second posture. And, as described above, since the second estimated posture when using the front-side processing is a posture close to the front-facing second posture, the first rotation estimated posture when using the front-side processing is a posture close to the second estimated posture when using the front-side processing. On the other hand, the first estimated posture when using the back-side processing is a posture close to the back-facing first posture. Therefore, similar to the case where the posture of the object 10 rotates from the first posture to the second posture, the first rotation estimated posture in which the first estimated posture when using the back-side processing rotates, that is, the first rotation estimated posture when using the back-side processing is a posture close to the back-facing second posture (the right side in FIG. 6). And, since the second estimated posture when using the back-side processing is a posture close to the back-facing third posture (the right side in FIG. 7), the first rotation estimated posture when using the back-side processing is a posture significantly different from the second estimated posture when using the back-side processing.

[0071] Next, consider the case where the object 10 in the reverse first posture is shown in the first captured image. In this case, as described above, the first estimated posture when using the reverse-side process is a posture close to the reverse first posture. Therefore, when the posture of the object 10 rotates from the first posture to the second posture, the first rotation estimated posture when using the reverse-side process, that is, the first rotation estimated posture when using the reverse-side process, is a posture close to the reverse second posture. And since the second estimated posture when using the reverse-side process is a posture close to the reverse second posture, the first rotation estimated posture when using the reverse-side process is a posture close to the second estimated posture when using the reverse-side process. On the other hand, since the first estimated posture when using the front-side process is a posture close to the front first posture, when the posture of the object 10 rotates from the first posture to the second posture, the first rotation estimated posture when using the front-side process, that is, the first rotation estimated posture when using the front-side process, is a posture close to the front second posture (the right side in FIG. 5). And since the second estimated posture when using the front-side process is a posture close to the front third posture (the left side in FIG. 8), the first rotation estimated posture when using the front-side process is a posture significantly different from the second estimated posture when using the front-side process.

[0072] Thus, when the first captured image showing the object 10 in the front first posture is generated, the first rotation estimated posture when using the front-side process is a posture close to the second estimated posture when using the front-side process. On the other hand, the first rotation estimated posture when using the reverse-side process is a posture significantly different from the second estimated posture when using the reverse-side process.

[0073] Also, when the first captured image showing the object 10 in the reverse first posture is generated, the first rotation estimated posture when using the reverse-side process is a posture close to the second estimated posture when using the reverse-side process. On the other hand, the first rotation estimated posture when using the front-side process is a posture significantly different from the second estimated posture when using the front-side process.

[0074] After step s7, in step s8, the specifying unit 21 performs a first comparison process of comparing the plurality of first rotation estimated postures obtained in step s7 with the plurality of second estimated postures obtained in step s6 for each corresponding specific surface. In this example, the specifying unit 21 performs a first comparison process of comparing the first rotation estimated postures during front - side processing use and the first rotation estimated postures during back - side processing use with the second estimated postures during front - side processing use and the second estimated postures during back - side processing use for each corresponding specific surface. Since the first rotation estimated posture during front - side processing use and the second estimated posture during front - side processing use correspond to the front surface 101, in the first comparison process, the first rotation estimated posture during front - side processing use and the second estimated posture during front - side processing use are compared. Since the first rotation estimated posture during back - side processing use and the second estimated posture during back - side processing use correspond to the back surface 102, in the first comparison process, the first rotation estimated posture during back - side processing use and the second estimated posture during back - side processing use are compared.

[0075] In the first comparison process, the specifying unit 21, for example, obtains an evaluation value (also referred to as a first similarity evaluation value) indicating the similarity between each first rotation estimated posture and the second estimated posture corresponding to the same specific surface as the specific surface corresponding to the first rotation estimated posture. As a result of the first comparison process, a plurality of first similarity evaluation values corresponding to the plurality of specific surfaces are obtained. The first similarity evaluation value indicating the similarity between the first rotation estimated posture corresponding to a certain specific surface and the second estimated posture corresponding to the certain specific surface is the first similarity evaluation value corresponding to the certain specific surface.

[0076] In this example, the specifying unit 21 obtains a first similarity evaluation value (also referred to as the first similarity evaluation value during front - side processing use) indicating the similarity between the first rotation estimated posture during front - side processing use and the second estimated posture during front - side processing use in the first comparison process. Further, the specifying unit 21 obtains a first similarity evaluation value (also referred to as the first similarity evaluation value during back - side processing use) indicating the similarity between the first rotation estimated posture during back - side processing use and the second estimated posture during back - side processing use in the first comparison process. The first similarity evaluation value during front - side processing use and the first similarity evaluation value during back - side processing use are the result of the first comparison process. An example of a method for calculating the first similarity evaluation value will be described below.

[0077] For example, the posture of the object 10 in the imaging device coordinate system is represented by the rotation (αn, βn, γn) with respect to the reference posture (n is a variable). αn means the rotation angle around the x-axis of the imaging device coordinate system (also referred to as the roll angle). βn means the rotation angle around the y-axis of the imaging device coordinate system (also referred to as the pitch angle). γn means the rotation angle around the z-axis of the imaging device coordinate system (also referred to as the yaw angle). For example, assume that the first rotation estimated posture during the front-side processing is represented by the rotation (α11, β11, γ11), and the second estimated posture during the front-side processing is represented by the rotation (α12, β12, γ12). Also, for example, assume that the first rotation estimated posture during the back-side processing is represented by the rotation (α21, β21, γ21), and the second estimated posture during the back-side processing is represented by the rotation (α22, β22, γ22). Further, in the imaging device coordinate system, assume that the vector starting from the origin is represented by (x, y, z). x indicates the position on the x-axis of the imaging device coordinate system, y indicates the position on the y-axis of the imaging device coordinate system, and z indicates the position on the z-axis of the imaging device coordinate system.

[0078] The specific part 21 considers the vector T(0, 0, 1) in the imaging device coordinate system. When the specific part 21 obtains the first similarity evaluation value during the front-side processing, it first rotates the vector T by α11 around the x-axis, then rotates it by β11 around the y-axis, and finally rotates it by γ11 around the z-axis, and sets the vector obtained by these rotations as the vector T11. Also, the specific part 21 first rotates the vector T by α12 around the x-axis, then rotates it by β12 around the y-axis, and finally rotates it by γ12 around the z-axis, and sets the vector obtained by these rotations as the vector T12. Then, the specific part 21 obtains the Euclidean distance between the end point of the vector T11 and the end point of the vector T12, and sets the obtained Euclidean distance as the first similarity evaluation value during the front-side processing. In this example, the smaller the first similarity evaluation value during the front-side processing, the higher the similarity between the first rotation estimated posture and the second estimated posture during the front-side processing.

[0079] Similarly, when the specific unit 21 obtains the first similarity evaluation value during the backside processing use, the vector T is first rotated by α21 around the x-axis, then rotated by β21 around the y-axis, and finally rotated by γ21 around the z-axis, and the vector obtained by these rotations is defined as vector T21. Further, the specific unit 21 rotates the vector T first by α22 around the x-axis, then by β22 around the y-axis, and finally by γ22 around the z-axis, and the vector obtained by these rotations is defined as vector T22. Then, the specific unit 21 calculates the Euclidean distance between the end point of vector T21 and the end point of vector T22, and sets the calculated Euclidean distance as the first similarity evaluation value during the backside processing use. In this example, the smaller the first similarity evaluation value during the backside processing use, the higher the similarity between the first rotation estimated posture and the second estimated posture during the backside processing use.

[0080] After step s8, in step s9, based on the result of the first comparison process in step s8, when the first imaging image in which the object 10 in the first posture is captured is generated by the imaging device 8, the specific unit 21 specifies, as the target surface, the specific surface that faces the imaging device 8 side among a plurality of specific surfaces of the object 10. That is, the specific unit 21 specifies, as the target surface, the specific surface that faces the imaging device 8 side when the object 10 in the first posture is imaged by the imaging device 8 based on the result of the first comparison process. In step s9, the specific unit 21 specifies, for example, the minimum first similarity evaluation value among a plurality of first similarity evaluation values as the result of the first comparison process. Then, the specific unit 21 sets the specific surface corresponding to the specified minimum first similarity evaluation value as the target surface.

[0081] In this example, when the first captured image is generated by the imaging device 8 (i.e., in step s2), the specific part 21 identifies the surfaces facing the imaging device 8 on the front surface 101 and the back surface 102 based on the first similarity evaluation value during the use of the front surface process and the first similarity evaluation value during the use of the back surface process, which are the results of the first comparison process. For example, if the first similarity evaluation value during the use of the front surface process is smaller than the first similarity evaluation value during the use of the back surface process, the specific part 21 determines that the specific surface corresponding to the first similarity evaluation value during the use of the front surface process, that is, the front surface 101, is the surface facing the imaging device 8 in step s2. It can also be said that if the first similarity evaluation value during the use of the front surface process is smaller than the first similarity evaluation value during the use of the back surface process, the specific part 21 determines that the object 10 is facing forward in step s2. On the other hand, for example, if the first similarity evaluation value during the use of the back surface process is smaller than the first similarity evaluation value during the use of the front surface process, the specific part 21 identifies the specific surface corresponding to the first similarity evaluation value during the use of the back surface process, that is, the back surface 102, as the surface facing the imaging device 8 in step s2. It can also be said that if the first similarity evaluation value during the use of the back surface process is smaller than the first similarity evaluation value during the use of the front surface process, the specific part 21 determines that the object 10 is facing backward in step s2. Note that when the first similarity evaluation value during the use of the front surface process is the same as the first similarity evaluation value during the use of the back surface process, the specific part 21 may use the front surface 101 as the target surface or the back surface 102 as the target surface.

[0082] As described above, when the first captured image showing the object 10 in the front-facing first posture is generated, the first rotation estimated posture during the front-side processing use is a posture close to the second estimated posture during the front-side processing use. On the other hand, the first rotation estimated posture during the back-side processing use is a posture significantly different from the second estimated posture during the back-side processing use. Therefore, when the front surface 101 faces the imaging device 8 when the first captured image is generated, the first similarity evaluation value during the front-side processing use becomes small, and the first similarity evaluation value during the back-side processing use becomes large. Thus, when the front surface 101 faces the imaging device 8 when the first captured image is generated, there is a high possibility that the first similarity evaluation value during the front-side processing use is smaller than the first similarity evaluation value during the back-side processing use. Therefore, when the first similarity evaluation value during the front-side processing use is smaller than the first similarity evaluation value during the back-side processing use, by determining that the surface facing the imaging device 8 is the front surface 101, it is possible to appropriately identify the surface facing the imaging device 8 when the first captured image is generated.

[0083] Also, as described above, when the first captured image showing the object 10 in the back-facing first posture is generated, the first rotation estimated posture during the back-side processing use is a posture close to the second estimated posture during the back-side processing use. On the other hand, the first rotation estimated posture during the front-side processing use is a posture significantly different from the second estimated posture during the front-side processing use. Therefore, when the back surface 102 faces the imaging device 8 when the first captured image is generated, the first similarity evaluation value during the back-side processing use becomes small, and the first similarity evaluation value during the front-side processing use becomes large. Thus, when the back surface 102 faces the imaging device 8 when the first captured image is generated, there is a high possibility that the first similarity evaluation value during the back-side processing use is smaller than the first similarity evaluation value during the front-side processing use. Therefore, when the first similarity evaluation value during the back-side processing use is smaller than the first similarity evaluation value during the front-side processing use, by determining that the surface facing the imaging device 8 is the back surface 102, it is possible to appropriately identify the surface facing the imaging device 8 when the first captured image is generated.

[0084] When the control unit 2 identifies the target surface, it determines that the surface on the opposite side of the identified target surface is held by the robot 9. For example, when the target surface is the front surface 101, the control unit 2 determines that the back surface 102 is held by the robot 9. On the other hand, when the target surface is the back surface 102, the control unit 2 determines that the front surface 101 is held by the robot 9. The control unit 2, which can control the robot 9 through the interface 5, may change the operation of the robot 9 according to the surface of the object 10 held by the robot 9. For example, the control unit 2 may cause the robot 9 holding the front surface 101 to execute a certain operation, and cause the robot 9 holding the back surface 102 to execute an operation different from the certain operation.

[0085] Note that the setting direction of the rotation axis J is not limited to the above example. For example, the rotation axis J may be set in a direction slightly oblique to the direction along the vertical direction of the object 10 when the object 10 in the first posture and the rotation axis J are viewed from the imaging device 8. Further, the rotation axis J may be set so as to pass through a position slightly deviated from the center in the left - right direction of the object 10 when the object 10 in the first posture and the rotation axis J are viewed from the imaging device 8. Further, the rotation axis J may be set so as not to pass through the object 10.

[0086] As can be understood from the above description, in the present disclosure, when the object 10 in the front-facing first posture is rotated by an angle +W, the appearance of the object 10 as seen from the imaging device 8 of the object 10 in the front-facing second posture obtained (the right side of FIG. 5), and the appearance of the object 10 as seen from the imaging device 8 of the object 10 in the back-facing second posture obtained by rotating the object 10 in the back-facing first posture by an angle +W (the right side of FIG. 6) are made to be significantly different from each other, whereby the identification accuracy of the target surface, that is, the surface facing the imaging device 8 side, can be improved. In other words, by making the front-facing second posture and the back-facing second posture significantly different from each other, the identification accuracy of the target surface can be improved. Further, in other words, when the specific surface facing the imaging device 8 side when the first imaging image is generated is the front surface 101, the appearance of the object 10 shown in the second imaging image (the right side of FIG. 5), and when the specific surface facing the imaging device 8 side when the first imaging image is generated is the back surface 102, the appearance of the object 10 shown in the second imaging image (the right side of FIG. 6) are made to be significantly different from each other, whereby the identification accuracy of the target surface can be improved. When the rotation axis J is set as in the examples of FIGS. 5 and 6, since the front-facing second posture and the back-facing second posture are significantly different from each other, the identification accuracy of the target surface is improved.

[0087] On the other hand, when the rotation axis J is set, for example, in the direction along the left-right direction of the object 10 when the object 10 and the rotation axis J are viewed from the imaging device 8 as shown in FIG. 9, the difference between the front-facing second posture and the back-facing second posture becomes small. Therefore, in this case, it becomes more difficult to identify the target surface compared to the case where the rotation axis J is set as in the examples of FIGS. 5 and 6.

[0088] In this example, the robot 9 rotates the object 10 by moving the arm 90. When the robot 9 holds the object 10 and transports the object 10 in front of the imaging device 8, as shown on the left side of FIG. 10, it is conceivable that the rotation axis M capable of rotating the object 10 only by the movement of the arm 90 is different from the rotation axis J about which the object 10 should rotate. In this case, before executing step s2, the robot 9 may cause the entire end effector 91 that holds the object 10 or only the tip of the end effector 91 to rotate relative to the arm 90 so that the rotation axis J coincides with the rotation axis M, as shown on the right side of FIG. 10. This operation of the robot 9 is realized by the control unit 2 of the processing device 1 controlling the robot 9. The control unit 2 causes the imaging device 8 to image the object 10 between step s1 and step s2. Then, based on the imaging image generated by the imaging device 8, the control unit 2 specifies the rotation of the object 10 around the optical axis of the imaging device 8, and based on the specification result, controls the robot 9 so that the entire end effector 91 or the tip of the end effector 91 of the robot 9 rotates relative to the arm 90 so that the rotation axis J coincides with the rotation axis M.

[0089] As described above, in this example, based on the plurality of first estimation results respectively estimated by a plurality of types of pose estimation processes based on the first imaging image in which the object 10 in the first pose is imaged, and the plurality of second estimation results respectively estimated by a plurality of types of pose estimation processes based on the second imaging image in which the object 10 in the second pose rotated from the first pose is imaged, when the first imaging image is generated by the imaging device 8, the specific surface facing the imaging device 8 on a plurality of specific surfaces of the object 10 is specified. Thereby, when the appearance of the object 10 imaged in the second imaging image varies greatly depending on the specific surface facing the imaging device 8 side when the first imaging image is generated, the specific surface facing the imaging device 8 side when the first imaging image is generated can be appropriately specified.

[0090] After identifying the target surface, the specifying unit 21 may use, as the final estimated result of the posture of the object 10 (i.e., the object 10 in the first posture) when the first captured image is generated by the imaging device 8, the first estimated posture corresponding to the identified target surface among the plurality of first estimated postures obtained in step s3. For example, when the specifying unit 21 identifies the front surface 101 as the target surface, the first estimated posture during the front surface processing may be used as the final estimated result of the posture of the object 10 when the first captured image is generated. On the other hand, when the specifying unit 21 identifies the back surface 102 as the target surface, the first estimated posture during the back surface processing may be used as the final estimated result of the posture of the object 10 when the first captured image is generated.

[0091] Further, after identifying the target surface, the specifying unit 21 may use, as the final estimated result of the posture of the object 10 (i.e., the object 10 in the second posture) when the second captured image is generated by the imaging device 8, the second estimated posture corresponding to the identified target surface among the plurality of second estimated postures obtained in step s6. For example, when the specifying unit 21 identifies the front surface 101 as the target surface, the second estimated posture during the front surface processing may be used as the final estimated result of the posture of the object 10 when the second captured image is generated. On the other hand, when the specifying unit 21 identifies the back surface 102 as the target surface, the second estimated posture during the back surface processing may be used as the final estimated result of the posture of the object 10 when the second captured image is generated.

[0092] Furthermore, after identifying the target surface, the specifying unit 21 may use, as the final estimated result of the posture of the object 10 (i.e., the object 10 in the second posture) when the second captured image is generated by the imaging device 8, the first rotational estimated posture corresponding to the identified target surface among the plurality of first rotational estimated postures obtained in step s7. For example, when the specifying unit 21 identifies the front surface 101 as the target surface, the first rotational estimated posture during the front surface processing may be used as the final estimated result of the posture of the object 10 when the second captured image is generated. On the other hand, when the specifying unit 21 identifies the back surface 102 as the target surface, the first rotational estimated posture during the back surface processing may be used as the final estimated result of the posture of the object 10 when the second captured image is generated.

[0093] <Another Example of Target Surface Identification Processing> In the above example, the first and second captured images are generated by the imaging device 8 when the posture of the object 10 is rotated when the imaging device 8 captures the object 10. However, the first and second captured images may also be generated by the imaging device 8 when the posture of the imaging device 8 when capturing the object 10 is rotated. FIG. 11 is a flowchart showing an example of the target surface identification processing in this case. When the target surface identification processing shown in FIG. 11 is executed, as shown in FIG. 12, a drive mechanism 800 capable of rotating the imaging device 8 is provided.

[0094] The drive mechanism 800 can rotate the imaging device 8 around the rotation axis J. As the drive mechanism 800, for example, a robot similar to the robot 9 may be adopted. The drive mechanism 800 is controlled by the control unit 2 of the processing device 1. The processing device 1 is provided with an interface capable of communicating with the drive mechanism 800. The control unit 2 controls the drive mechanism 800 through the interface to rotate the imaging device 8 around the rotation axis J by the drive mechanism 800.

[0095] In the target surface identification processing shown in FIG. 11, first, the above steps s1, s2, and s3 are executed. Then, in step s11, the control unit 2 rotates the imaging device 8 by the drive mechanism 800. In step s11, the drive mechanism 800 rotates the imaging device 8 by an angle -W around the rotation axis J. Then, the above step s5 is executed, and the imaging device 8 captures the object 10. In step s5, as in the above example, a second captured image in which the object 10 in the second posture rotated from the first posture is captured is generated. Thereafter, the above steps s6, s7, s8, and s9 are sequentially executed.

[0096] Thus, even when the first and second captured images are generated by the imaging device 8 when the posture of the imaging device 8 when capturing the object 10 is rotated, the control unit 2 can appropriately identify the specific surface facing the imaging device 8 when the first captured image is generated.

[0097] FIG. 13 is a flowchart showing another example of the target surface identification process. In the target surface identification process shown in FIG. 13, the above-described steps s1, s2, s3, s4, s5, and s6 are executed. After step s6, step s21 is executed.

[0098] Here, a three-dimensional orthogonal coordinate system is set for the end effector 91 as a holding mechanism for holding the object 10. This three-dimensional orthogonal coordinate system is called the holding mechanism coordinate system. The holding mechanism coordinate system is a coordinate system that represents the real space as seen from the end effector 91 as a coordinate space based on the position of the tip of the arm 90. As described above, in step s4, due to the movement of the arm 90, the end effector 91 that holds the object 10 rotates by an angle +W around the rotation axis J, so that the object 10 rotates by an angle +W around the rotation axis J. It can also be said that when the end effector 91 rotates by an angle +W around the rotation axis J, the posture of the object 10 changes from the first posture to the second posture. The holding mechanism coordinate system of the end effector 91 when holding the object 10 in the second posture is obtained by rotating the holding mechanism coordinate system of the end effector 91 when holding the object 10 in the first posture by an angle +W around the rotation axis J. In other words, the holding mechanism coordinate system when the second captured image is generated is obtained by rotating the holding mechanism coordinate system when the first captured image is generated by an angle +W around the rotation axis J. Hereinafter, the holding mechanism coordinate system of the end effector 91 when holding the object 10 in the first posture is called the pre-rotation holding mechanism coordinate system, and the holding mechanism coordinate system of the end effector 91 when holding the object 10 in the second posture is called the post-rotation holding mechanism coordinate system.

[0099] In step S21, the specifying unit 21 converts a plurality of first estimated postures obtained in step S3 from the imaging device coordinate system to the pre-rotation holding mechanism coordinate system, and acquires a plurality of first converted estimated postures. In this example, in step S21, the specifying unit 21 converts the first estimated posture during the use of the front-side process from the imaging device coordinate system to the pre-rotation holding mechanism coordinate system, and acquires the first converted estimated posture during the use of the front-side process. Further, the specifying unit 21 converts the first estimated posture during the use of the back-side process from the imaging device coordinate system to the pre-rotation holding mechanism coordinate system, and acquires the first converted estimated posture during the use of the back-side process. Converting the first estimated posture from the imaging device coordinate system to the pre-rotation holding mechanism coordinate system can also be said to be converting the first estimated posture from the estimated posture of the object 10 when the object 10 in the first posture is viewed from the imaging device 8 to the estimated posture of the object 10 when the object 10 in the first posture is viewed from the end effector 91 that holds the object 10 in the first posture. Therefore, it can be said that the first converted estimated posture is the estimated posture of the object 10 when the object 10 in the first posture is viewed from the end effector 91 that holds the object 10 in the first posture.

[0100] After step S21, in step S22, the specifying unit 21 converts a plurality of second estimated postures obtained in step S6 from the imaging device coordinate system to the post-rotation holding mechanism coordinate system, and acquires a plurality of second converted estimated postures. In this example, in step S22, the specifying unit 21 converts the second estimated posture during the use of the front-side process from the imaging device coordinate system to the post-rotation holding mechanism coordinate system, and acquires the second converted estimated posture during the use of the front-side process. Further, the specifying unit 21 converts the second estimated posture during the use of the back-side process from the imaging device coordinate system to the post-rotation holding mechanism coordinate system, and acquires the second converted estimated posture during the use of the back-side process. Converting the second estimated posture from the imaging device coordinate system to the post-rotation holding mechanism coordinate system can also be said to be converting the second estimated posture from the estimated posture of the object 10 when the object 10 in the second posture is viewed from the imaging device 8 to the estimated posture of the object 10 when the object 10 in the second posture is viewed from the end effector 91 that holds the object 10 in the second posture. Therefore, it can be said that the second converted estimated posture is the estimated posture of the object 10 when the object 10 in the second posture is viewed from the end effector 91 that holds the object 10 in the second posture.

[0101] For example, consider the case where the object 10 in the front-facing first posture appears in the first captured image. In this case, as described above, the first estimated posture during the front-side processing use is a posture close to the actual posture of the object 10 as seen from the imaging device 8. Also, the second estimated posture during the front-side processing use is a posture close to the actual posture of the object 10 as seen from the imaging device 8. Since the end effector 91 rotates together with the object 10, the actual appearance of the object 10 in the first posture as seen from the end effector 91 holding the object 10 in the first posture is the same as the actual appearance of the object 10 in the second posture as seen from the end effector 91 holding the object 10 in the second posture. Therefore, the first conversion estimated posture during the front-side processing use obtained from the first estimated posture during the front-side processing use, which is close to the actual posture of the object 10, and the second conversion estimated posture during the front-side processing use obtained from the second estimated posture during the front-side processing use, which is close to the actual posture of the object 10, are postures close to each other.

[0102] On the other hand, when the object 10 in the front-facing first posture appears in the first captured image, the first estimated posture during the back-side processing use is a posture close to the posture of the object 10 when the object 10 in the back-facing first posture is seen from the imaging device 8. Since the end effector 91 rotates together with the object 10, if the second estimated posture during the back-side processing use is, for example, a posture close to the posture of the object 10 when the object 10 in the back-facing first posture is rotated by an angle +W and then seen from the imaging device 8, that is, a posture close to the back-facing second posture (the right side in FIG. 6), then, like the relationship between the first conversion estimated posture and the second conversion estimated posture during the front-side processing use, the first conversion estimated posture and the second conversion estimated posture during the back-side processing use are postures close to each other. However, as described above, the second estimated posture during the back-side processing use is a posture close to the back-facing third posture. Therefore, the first conversion estimated posture and the second conversion estimated posture during the back-side processing use are significantly different from each other.

[0103] When the object 10 in the reverse first posture is shown in the first captured image, it can be considered in the same way as above. When the object 10 in the reverse first posture is shown in the first captured image, the first conversion estimated posture when using the back-side processing and the second conversion estimated posture when using the back-side processing are in poses close to each other. On the other hand, the first conversion estimated posture when using the front-side processing and the second conversion estimated posture when using the front-side processing will be significantly different from each other.

[0104] After step s22, in step s23, the specifying unit 21 performs a second comparison process of comparing the plurality of first conversion estimated postures obtained in step s21 and the plurality of second conversion estimated postures obtained in step s22 for each corresponding surface. In this example, the specifying unit 21 performs a second comparison process of comparing the first conversion estimated posture when using the front-side processing and the first conversion estimated posture when using the back-side processing, and the second conversion estimated posture when using the front-side processing and the second conversion estimated posture when using the back-side processing for each corresponding surface. Since the first conversion estimated posture when using the front-side processing and the second conversion estimated posture when using the front-side processing correspond to the front surface 101, in the second comparison process, the first conversion estimated posture when using the front-side processing and the second conversion estimated posture when using the front-side processing are compared. Since the first conversion estimated posture when using the back-side processing and the second conversion estimated posture when using the back-side processing correspond to the back surface 102, in the second comparison process, the first conversion estimated posture when using the back-side processing and the second conversion estimated posture when using the back-side processing are compared.

[0105] In the second comparison process, the specifying unit 21, for example, obtains an evaluation value (also referred to as a second similarity evaluation value) indicating the similarity between each first conversion estimated posture and the second conversion estimated posture corresponding to the same specific surface as the specific surface corresponding to the first conversion estimated posture. As a result of the second comparison process, a plurality of second similarity evaluation values corresponding to a plurality of specific surfaces are obtained respectively. The second similarity evaluation value indicating the similarity between the first conversion estimated posture corresponding to a certain specific surface and the second conversion estimated posture corresponding to the certain specific surface is the second similarity evaluation value corresponding to the certain specific surface.

[0106] In the second comparison process, the specific part 21 obtains, for example, a second similarity evaluation value (also referred to as the second similarity evaluation value during front - side processing) indicating the similarity between the first estimated transformed posture during front - side processing and the second estimated transformed posture during front - side processing. Also, in the second comparison process, the specific part 21 obtains, for example, a second similarity evaluation value (also referred to as the second similarity evaluation value during back - side processing) indicating the similarity between the first estimated transformed posture during back - side processing and the second estimated transformed posture during back - side processing. The second similarity evaluation value during front - side processing and the second similarity evaluation value during back - side processing are the result of the second comparison process. The specific part 21 may calculate the second similarity evaluation value in the same way as when calculating the first similarity evaluation value, for example.

[0107] After step s23, in step s24, based on the result of the second comparison process, the specific part 21 specifies, as the target surface, the specific surface facing the imaging device 8 when the object 10 in the first posture is imaged by the imaging device 8. In step s23, the specific part 21 specifies, for example, the minimum second similarity evaluation value among a plurality of second similarity evaluation values. Then, the specific part 21 sets the specific surface corresponding to the specified minimum second similarity evaluation value as the target surface.

[0108] In this example, when the first captured image is generated by the imaging device 8, the specific part 21 specifies the surfaces facing the imaging device 8 on the front surface 101 and the back surface 102 based on the second similarity evaluation value during front - side processing and the second similarity evaluation value during back - side processing. For example, if the second similarity evaluation value during front - side processing is smaller than the second similarity evaluation value during back - side processing, the specific part 21 determines that the surface facing the imaging device 8 in step s2 is the front surface 101. On the other hand, for example, if the second similarity evaluation value during back - side processing is smaller than the second similarity evaluation value during front - side processing, the specific part 21 determines that the surface facing the imaging device 8 in step s2 is the back surface 102. Note that when the second similarity evaluation value during front - side processing is the same as the second similarity evaluation value during back - side processing, the specific part 21 may use the front surface 101 as the target surface or the back surface 102 as the target surface.

[0109] As described above, when the first captured image showing the object 10 in the front-facing first posture is generated, the first conversion estimated posture during the front-side processing use is a posture close to the second conversion estimated posture during the front-side processing use. On the other hand, the first conversion estimated posture during the back-side processing use is a posture significantly different from the second conversion estimated posture during the back-side processing use. Therefore, when the front surface 101 faces the imaging device 8 when the first captured image is generated, the second similarity evaluation value during the front-side processing use becomes small, and the second similarity evaluation value during the back-side processing use becomes large. Thus, when the front surface 101 faces the imaging device 8 when the first captured image is generated, there is a high possibility that the second similarity evaluation value during the front-side processing use is smaller than the second similarity evaluation value during the back-side processing use. Therefore, when the second similarity evaluation value during the front-side processing use is smaller than the second similarity evaluation value during the back-side processing use, by determining that the surface facing the imaging device 8 is the front surface 101, it is possible to appropriately identify the surface facing the imaging device 8 when the first captured image is generated.

[0110] Also, as described above, when the first captured image showing the object 10 in the back-facing first posture is generated, the first conversion estimated posture during the back-side processing use is a posture close to the second conversion estimated posture during the back-side processing use. On the other hand, the first conversion estimated posture during the front-side processing use is a posture significantly different from the second conversion estimated posture during the front-side processing use. Therefore, when the back surface 102 faces the imaging device 8 when the first captured image is generated, the second similarity evaluation value during the back-side processing use becomes small, and the second similarity evaluation value during the front-side processing use becomes large. Thus, when the back surface 102 faces the imaging device 8 when the first captured image is generated, there is a high possibility that the second similarity evaluation value during the back-side processing use is smaller than the second similarity evaluation value during the front-side processing use. Therefore, when the second similarity evaluation value during the back-side processing use is smaller than the second similarity evaluation value during the front-side processing use, by determining that the surface facing the imaging device 8 is the back surface 102, it is possible to appropriately identify the surface facing the imaging device 8 when the first captured image is generated.

[0111] FIG. 14 is a flowchart showing another example of the target surface identification process. In the target surface identification process shown in FIG. 14, the above-described steps s1, s2, s3, s11, s5, s6 are executed. After step s6, step s31 is executed.

[0112] Here, as described above, in step s11, the imaging device 8 rotates by an angle -W around the rotation axis J. Therefore, the imaging device coordinate system of the imaging device 8 when generating the second captured image is obtained by rotating the imaging device coordinate system of the imaging device 8 when generating the first captured image by an angle -W around the rotation axis J. The imaging device coordinate system of the imaging device 8 when generating the first captured image is referred to as the pre-rotation imaging device coordinate system, and the imaging device coordinate system of the imaging device 8 when generating the second captured image is referred to as the post-rotation imaging device coordinate system. The plurality of first estimated postures obtained in step s3 are represented in the pre-rotation holding mechanism coordinate system, and the plurality of second estimated postures obtained in step s6 are represented in the post-rotation holding mechanism coordinate system.

[0113] In step s31, the specifying unit 21 converts the plurality of first estimated postures obtained in step s3 from the pre-rotation imaging device coordinate system to the holding mechanism coordinate system to obtain a plurality of third converted estimated postures. In this example, in step s31, the specifying unit 21 converts the first estimated posture during the use of the front-side process from the pre-rotation imaging device coordinate system to the holding mechanism coordinate system to obtain the third converted estimated posture during the use of the front-side process. Further, the specifying unit 21 converts the first estimated posture during the use of the back-side process from the pre-rotation imaging device coordinate system to the holding mechanism coordinate system to obtain the third converted estimated posture during the use of the back-side process. If the holding posture of the object 10 by the robot 9 in the target surface identification process is the same, the third converted estimated posture during the use of the front-side process is the same as the above-described second converted estimated posture during the use of the front-side process, and the third converted estimated posture during the use of the back-side process is the same as the above-described second converted estimated posture during the use of the back-side process.

[0114] After step S31, in step S32, the specifying unit 21 converts the plurality of second estimated postures obtained in step S6 from the post-rotation imaging device coordinate system to the holding mechanism coordinate system, and acquires a plurality of fourth converted estimated postures. In this example, in step S32, the specifying unit 21 converts the second estimated posture during front-side processing from the post-rotation imaging device coordinate system to the holding mechanism coordinate system, and acquires the fourth converted estimated posture during front-side processing. Further, the specifying unit 21 converts the second estimated posture during back-side processing from the post-rotation imaging device coordinate system to the holding mechanism coordinate system, and acquires the fourth converted estimated posture during back-side processing.

[0115] For example, consider the case where the object 10 in the first posture facing the front is shown in the first captured image. In this case, as described above, the first estimated posture during front-side processing is a posture close to the actual posture of the object 10 as seen from the imaging device 8. Also, the second estimated posture during front-side processing is a posture close to the actual posture of the object 10 as seen from the imaging device 8. Similar to the case where the object 10 rotates together with the end effector 91 as in the example of FIG. 13, even when the imaging device 8 rotates, when the first captured image is generated, the actual state of seeing the object 10 from the end effector 91 is the same as the actual state of seeing the object 10 from the end effector 91 when the second captured image is generated. Therefore, the third converted estimated posture during front-side processing obtained from the first estimated posture during front-side processing close to the actual posture of the object 10 and the fourth converted estimated posture during front-side processing obtained from the second estimated posture during front-side processing close to the actual posture of the object 10 are close to each other.

[0116] On the one hand, when the object 10 in the front-facing first posture appears in the first captured image, the first estimated posture during the use of the back-side process is a posture close to the back-facing first posture. If the second estimated posture during the use of the back-side process is, hypothetically, a posture close to the back-facing second posture, then like the relationship between the third converted estimated posture and the fourth converted estimated posture during the use of the front-side process, the third converted estimated posture and the fourth converted estimated posture during the use of the back-side process are in postures close to each other. However, the second estimated posture during the use of the back-side process is a posture close to the back-facing third posture. Therefore, the third converted estimated posture and the fourth converted estimated posture during the use of the back-side process are significantly different from each other.

[0117] The same consideration as above can be made even when the object 10 in the back-facing first posture appears in the first captured image. When the object 10 in the back-facing first posture appears in the first captured image, the third converted estimated posture and the fourth converted estimated posture during the use of the back-side process are in postures close to each other. In contrast, the third converted estimated posture and the fourth converted estimated posture during the use of the front-side process are significantly different from each other.

[0118] After step s32, in step s33, the specifying unit 21 performs a third comparison process of comparing the plurality of third converted estimated postures obtained in step s31 and the plurality of fourth converted estimated postures obtained in step s32 for each corresponding surface. In this example, the specifying unit 21 performs a third comparison process of comparing the third converted estimated posture during the use of the front-side process and the third converted estimated posture during the use of the back-side process, and the fourth converted estimated posture during the use of the front-side process and the fourth converted estimated posture during the use of the back-side process for each corresponding surface. In the third comparison process, the third converted estimated posture and the fourth converted estimated posture during the use of the front-side process are compared, and the third converted estimated posture and the fourth converted estimated posture during the use of the back-side process are compared.

[0119] In the third comparison process, for example, with respect to each third transformation estimated posture, the specific unit 21 obtains an evaluation value (also referred to as the third similarity evaluation value) indicating the similarity between the third transformation estimated posture and the fourth transformation estimated posture corresponding to the same specific surface as the specific surface corresponding to the third transformation estimated posture. As a result of the third comparison process, a plurality of third similarity evaluation values corresponding to a plurality of specific surfaces are obtained respectively. The third similarity evaluation value indicating the similarity between the third transformation estimated posture corresponding to a certain specific surface and the fourth transformation estimated posture corresponding to the certain specific surface is the third similarity evaluation value corresponding to the certain specific surface.

[0120] In this example, in the third comparison process, the specific unit 21 obtains a third similarity evaluation value (also referred to as the third similarity evaluation value during front - side processing use) indicating the similarity between the third transformation estimated posture during front - side processing use and the fourth transformation estimated posture during front - side processing use. Further, in the third comparison process, for example, the specific unit 21 obtains a third similarity evaluation value (also referred to as the third similarity evaluation value during back - side processing use) indicating the similarity between the third transformation estimated posture during back - side processing use and the fourth transformation estimated posture during back - side processing use. The third similarity evaluation value during front - side processing use and the third similarity evaluation value during back - side processing use are the results of the third comparison process. The specific unit 21 may calculate the third similarity evaluation value in the same manner as when calculating the first similarity evaluation value, for example.

[0121] After step s33, in step s34, based on the result of the third comparison process, the specific unit 21 specifies, as the target surface, the specific surface facing the imaging device 8 when the object 10 in the first posture is imaged by the imaging device 8. In step s34, for example, the specific unit 21 specifies the minimum third similarity evaluation value among the plurality of third similarity evaluation values. Then, the specific unit 21 sets the specific surface corresponding to the specified minimum third similarity evaluation value as the target surface.

[0122] In this example, the specific part 21 identifies the surfaces facing the imaging device 8 on the front surface 101 and the back surface 102 when the first captured image is generated by the imaging device 8, based on the third similarity evaluation value during front - side processing and the third similarity evaluation value during back - side processing. For example, if the third similarity evaluation value during front - side processing is smaller than the third similarity evaluation value during back - side processing, the specific part 21 determines that the surface facing the imaging device 8 in step s2 is the front surface 101. On the other hand, for example, if the third similarity evaluation value during back - side processing is smaller than the third similarity evaluation value during front - side processing, the specific part 21 determines that the surface facing the imaging device 8 in step s2 is the back surface 102. Note that when the third similarity evaluation value during front - side processing is the same as the third similarity evaluation value during back - side processing, the specific part 21 may use the front surface 101 as the target surface or the back surface 102 as the target surface.

[0123] FIG. 15 is a flowchart showing another example of the target - surface identification process. In the target - surface identification process shown in FIG. 15, the above - mentioned steps s1, s2, s3, s4, s5, s7 are executed. After step s7, step s41 is executed.

[0124] In step s41, for each of the plurality of first - rotation estimated postures obtained in step s7, the specific part 21 identifies the corresponding template from the plurality of templates used in the posture - estimation process corresponding to the first - rotation estimated posture among the plurality of types of posture - estimation processes. As a result, a plurality of corresponding templates corresponding to the plurality of first - rotation estimated postures are identified. In this example, the specific part 21 identifies, as the corresponding front - side template, the front - side template in which the same posture as the first - rotation estimated posture during front - side processing is associated, from the plurality of front - side templates used in the front - side posture - estimation process corresponding to the first - rotation estimated posture during front - side processing. Also, the specific part 21 identifies, as the corresponding back - side template, the back - side template in which the same posture as the first - rotation estimated posture during back - side processing is associated, from the plurality of back - side templates used in the back - side posture - estimation process corresponding to the first - rotation estimated posture during back - side processing.

[0125] After step S41, in step S42, the specifying unit 21 performs template matching to compare the second captured image obtained in step S5 with each of the plurality of corresponding templates specified in step S41. The specifying unit 21 obtains a matching score between each corresponding template and the second captured image in the same manner as the template matching performed in the pose estimation process by the estimation unit 20. As a result of the template matching in step S42, a plurality of matching scores corresponding to the plurality of corresponding templates are obtained respectively. The matching score corresponding to a corresponding template is the matching score between the second captured image for the corresponding template. In this example, the specifying unit 21 obtains the matching score between the template for correspondence table and the second captured image, and obtains the matching score between the template for reverse side of correspondence and the second captured image. Hereinafter, the matching score between the template for correspondence table and the second captured image may be referred to as the matching score when using the template for correspondence table, and the matching score between the template for reverse side of correspondence and the second captured image may be referred to as the matching score when using the template for reverse side of correspondence.

[0126] After step S42, in step S43, the specifying unit 21 specifies, as a target surface, the specific surface facing the imaging device 8 side when the object 10 in the first pose is imaged by the imaging device 8, based on the result of the template matching in step S42. In step S43, the specifying unit 21 specifies, for example, the highest matching score among the plurality of matching scores as the result of the template matching. Then, the specifying unit 21 sets the specific surface corresponding to the corresponding template corresponding to the specified highest matching score as the target surface. The specific surface corresponding to a certain correspondence template is the specific surface corresponding to the pose estimation process in which the certain correspondence template is used. For example, the specific surface corresponding to the template for correspondence table is the front surface 101 corresponding to the front pose estimation process in which the template for correspondence table is used, and the specific surface corresponding to the template for reverse side of correspondence is the back surface 102 corresponding to the back pose estimation process in which the template for reverse side of correspondence is used.

[0127] In this example, the specific part 21 specifies the surfaces facing the imaging device 8 on the front surface 101 and the back surface 102 when the first captured image is generated by the imaging device 8, based on the matching score when using the template for the correspondence table and the matching score when using the template for the corresponding back. For example, if the matching score when using the template for the correspondence table is higher than the matching score when using the template for the corresponding back, the specific part 21 determines that the surface facing the imaging device 8 in step s2 is the front surface 101. On the other hand, for example, if the matching score when using the template for the corresponding back is higher than the matching score when using the template for the correspondence table, the specific part 21 determines that the surface facing the imaging device 8 in step s2 is the back surface 102. Note that when the matching score when using the template for the correspondence table is the same as the matching score when using the template for the corresponding back, the specific part 21 may use the front surface 101 as the target surface or the back surface 102 as the target surface.

[0128] For example, consider a case where the object 10 in the front-facing first posture is shown in the first captured image. In this case, as described above, the first estimated posture during the use of the front-side processing is a posture close to the front-facing first posture. Therefore, the first rotation estimated posture during the use of the front-side processing is a posture close to the front-facing second posture. Thus, the corresponding front-side template corresponding to the first rotation estimated posture during the use of the front-side processing is an image similar to the second captured image in which the object 10 in the front-facing second posture is shown. Therefore, the matching score between the corresponding front-side template and the second captured image becomes high. On the other hand, the first estimated posture during the use of the back-side processing is a posture close to the back-facing first posture. Therefore, the first rotation estimated posture during the use of the back-side processing is a posture close to the back-facing second posture. Thus, the corresponding back-side template corresponding to the first rotation estimated posture during the use of the back-side processing is similar to the captured image in which the back-facing second posture is shown and is not similar to the second captured image in which the object 10 in the front-facing second posture is shown. Therefore, the matching score between the corresponding back-side template and the second captured image becomes low. Therefore, when the front surface 101 faces the imaging device 8 when the first captured image is generated, it is highly likely that the matching score when using the corresponding front-side template is higher than the matching score when using the corresponding back-side template.

[0129] Also, consider the case where the object 10 in the reverse first posture is shown in the first captured image. In this case, as described above, the first estimated posture when using the back-side processing is a posture close to the reverse first posture. Therefore, the first rotation estimated posture when using the back-side processing is a posture close to the actual reverse second posture. Thus, the corresponding back-side template corresponding to the first rotation estimated posture when using the back-side processing is an image similar to the second captured image in which the object 10 in the reverse second posture is shown. Therefore, the matching score between the corresponding back-side template and the second captured image becomes high. On the other hand, the first estimated posture when using the front-side processing is a posture close to the front first posture. Therefore, the first rotation estimated posture when using the front-side processing is a posture close to the front second posture. Thus, the corresponding front-side template corresponding to the first rotation estimated posture when using the front-side processing is similar to the captured image in which the front second posture is shown and is not similar to the second captured image in which the object 10 in the reverse second posture is shown. Therefore, the matching score between the corresponding front-side template and the second captured image becomes low. Therefore, when the back surface 102 faces the imaging device 8 side when the first captured image is generated, it is highly likely that the matching score when using the corresponding back-side template is higher than the matching score when using the corresponding front-side template.

[0130] As described above, in this example, when the first captured image is generated, depending on whether the specific surface facing the imaging device 8 is the front surface 101 or the back surface 102, the appearance of the object 10 shown in the second captured image varies significantly. Therefore, when the front surface 101 faces the imaging device 8 when the first captured image is generated, the matching score when using the template for the front surface correspondence is likely to be higher than the matching score when using the template for the back surface correspondence. When the back surface 102 faces the imaging device 8 when the first captured image is generated, the matching score when using the template for the back surface correspondence is likely to be higher than the matching score when using the template for the front surface correspondence. Thus, when the matching score when using the template for the front surface correspondence is higher than the matching score when using the template for the back surface correspondence, it is determined that the surface facing the imaging device 8 is the front surface 101, and when the matching score when using the template for the back surface correspondence is higher than the matching score when using the template for the front surface correspondence, it is determined that the surface facing the imaging device 8 is the back surface 102. By doing so, it is possible to appropriately identify the surface facing the imaging device 8 when the first captured image is generated.

[0131] Note that, as shown in FIG. 16, instead of the step s4 of rotating the object 10 by an angle +W, a step s11 of rotating the imaging device 8 by an angle -W may be executed.

[0132] As described above, in the examples of FIGS. 15 and 16, template matching is performed to compare the correspondence templates specified for a plurality of first rotation estimated postures with the second captured image, and based on the result of the template matching, the specific surface facing the imaging device 8 when the first captured image is generated by the imaging device 8 is specified. Thereby, when the appearance of the object 10 shown in the second captured image varies significantly depending on the specific surface facing the imaging device 8 when the first captured image is generated, it is possible to appropriately identify the specific surface facing the imaging device 8 when the first captured image is generated.

[0133] In addition to the examples of FIGS. 15 and 16 described above, in the posture estimation process, the posture of the object 10 may be estimated by a method other than template matching. For example, the estimation unit 20 may estimate the posture of the object 10 by a method using a non-linear function such as a neural network. For example, when the estimation unit 20 estimates the posture of the object 10 using a neural network, in the front-side posture estimation process, a neural network learned based on a large number of captured images obtained by capturing the object 10 from the front surface 101 side at various angles is used, and in the back-side posture estimation process, a neural network learned based on a large number of captured images obtained by capturing the object 10 from the back surface 102 side at various angles may be used.

[0134] In the examples of FIGS. 4 and 11 described above, the first rotation estimated posture is obtained from the first estimated posture, but the first rotation estimated posture may be directly obtained from the first captured image without obtaining the first estimated posture. For example, an estimator (e.g., a neural network) learned to estimate the first rotation estimated posture when using the front-side process using machine learning may estimate the first rotation estimated posture when using the front-side process based on the first captured image and the angle +W around the rotation axis J. The first rotation estimated posture when using the front-side process estimated by the estimator corresponds to a posture obtained by rotating the estimated posture of the object 10 in the first posture by the angle +W around the rotation axis J assuming that the front surface 101 of the object 10 faces the imaging device 8 side. Also, an estimator learned to estimate the first rotation estimated posture when using the back-side process using machine learning may estimate the first rotation estimated posture when using the back-side process based on the first captured image and the angle +W around the rotation axis J. The first rotation estimated posture when using the back-side process estimated by the estimator corresponds to a posture obtained by rotating the estimated posture of the object 10 in the first posture by the angle +W around the rotation axis J assuming that the back surface 102 of the object 10 faces the imaging device 8 side.

[0135] In the examples of FIGS. 4 and 11 described above, a plurality of first estimated postures used were a plurality of first rotation estimated postures rotated in the same manner as when the posture of the object 10 rotates from the first posture to the second posture. However, a plurality of second estimated postures may be a plurality of second rotation estimated postures rotated in a direction opposite to that when the posture of the object 10 rotates from the first posture to the second posture. In this case, in step s7 described above, the specifying unit 21 obtains a plurality of second rotation estimated postures obtained by rotating the plurality of second estimated postures obtained in step s6 in a direction opposite to that when the posture of the object 10 rotates from the first posture to the second posture. For example, the specifying unit 21 obtains a second rotation estimated posture (also referred to as the second rotation estimated posture during front surface processing use) obtained by rotating the second estimated posture during front surface processing use in a direction opposite to that when the posture of the object 10 rotates from the first posture to the second posture. Further, the specifying unit 21 obtains a second rotation estimated posture (also referred to as the second rotation estimated posture during back surface processing use) obtained by rotating the second estimated posture during back surface processing use in a direction opposite to that when the posture of the object 10 rotates from the first posture to the second posture. The specifying unit 21 sets, in the imaging device coordinate system, the posture obtained by rotating the second estimated posture during front surface processing use by an angle -W around the rotation axis J as the second rotation estimated posture during front surface processing use. Further, the specifying unit 21 sets, in the imaging device coordinate system, the posture obtained by rotating the second estimated posture during back surface processing use by an angle -W around the rotation axis J as the second rotation estimated posture during back surface processing use.

[0136] As can be understood from the above description, when the object 10 in the front-facing first posture appears in the first captured image, the second rotation estimated posture during front surface processing use becomes a posture close to the first estimated posture during front surface processing use. On the other hand, the second rotation estimated posture during back surface processing use becomes a posture significantly different from the first estimated posture during back surface processing use. Therefore, when the object 10 in the front-facing first posture appears in the first captured image, that is, when the target surface is the front surface 101, the similarity between the second rotation estimated posture during front surface processing use and the first estimated posture during front surface processing use increases, and the similarity between the second rotation estimated posture during back surface processing use and the first estimated posture during back surface processing use decreases.

[0137] Also, when the object 10 in the reverse first posture is shown in the first captured image, the second rotation estimated posture during back-side processing use becomes a posture close to the first estimated posture during back-side processing use. On the other hand, the second rotation estimated posture during front-side processing use becomes a posture significantly different from the first estimated posture during front-side processing use. Therefore, when the object 10 in the reverse first posture is shown in the first captured image, that is, when the target surface is the back surface 102, the similarity between the second rotation estimated posture during back-side processing use and the first estimated posture during back-side processing use becomes high, and the similarity between the second rotation estimated posture during front-side processing use and the first estimated posture during front-side processing use becomes low.

[0138] When a plurality of second rotation estimated postures are obtained in the above step s7, in the above step s8, the specifying unit 21 performs a fourth comparison process of comparing the plurality of second rotation estimated postures and the plurality of first estimated postures for each corresponding specific surface in the same manner as the first comparison process. Then, in the above step s9, the specifying unit 21 specifies the target surface based on the result of the fourth comparison process in step s8. In step s9, when the similarity between the second rotation estimated posture during front-side processing use and the first estimated posture during front-side processing use is higher than the similarity between the second rotation estimated posture during back-side processing use and the first estimated posture during back-side processing use as a result of the fourth comparison process, the specifying unit 21 sets the front surface 101 as the target surface. On the other hand, when the similarity between the second rotation estimated posture during back-side processing use and the first estimated posture during back-side processing use is higher than the similarity between the second rotation estimated posture during front-side processing use and the first estimated posture during front-side processing use as a result of the fourth comparison process, the specifying unit 21 determines that the back surface 102 is the target surface.

[0139] Note that the second rotational estimated posture may be directly obtained from the second captured image without obtaining the second estimated posture. For example, an estimator (e.g., a neural network) that is trained to use machine learning to estimate the second rotational estimated posture during front-side processing may estimate the second rotational estimated posture during front-side processing based on the second captured image and the angle -W around the rotation axis J. The second rotational estimated posture during front-side processing estimated by the estimator corresponds to a rotational posture obtained by rotating the estimated posture, which is obtained by assuming that the front surface 101 of the object 10 in the second posture faces the imaging device 8 side, by the angle -W around the rotation axis J. Also, an estimator that is trained to use machine learning to estimate the second rotational estimated posture during back-side processing may estimate the second rotational estimated posture during back-side processing based on the second captured image and the angle -W around the rotation axis J. The second rotational estimated posture during back-side processing estimated by the estimator corresponds to a rotational posture obtained by rotating the estimated posture, which is obtained by assuming that the back surface 102 of the object 10 faces the imaging device 8 side, by the angle -W around the rotation axis J.

[0140] As described above, the processing device has been described in detail. However, the above description is illustrative in all aspects and the present disclosure is not limited thereto. Also, the various examples described above can be applied in combination as long as they do not conflict with each other. And it is understood that countless examples that are not illustrated can be assumed without departing from the scope of this disclosure.

[0141] For example, in this example, the appearance of the object 10 shown in the first captured image is the first appearance from the left in FIG. 3 on the front surface and the third appearance from the left in FIG. 3 on the back surface. However, the present disclosure is not limited to this. For example, the front surface may be the first appearance from the left in FIG. 7 and the back surface may be the second appearance from the left in FIG. 7, or the front surface may be the first appearance from the left in FIG. 8 and the back surface may be the second appearance from the left in FIG. 8. In these cases, W may be any minute angle.

[0142] The present disclosure includes the following content.

[0143] In one embodiment, (1) the processing device includes a control unit that acquires a first image of an object having a plurality of surfaces with similar appearances from an imaging device in a first posture of the object, and identifies a target surface facing the imaging device side among the plurality of surfaces. The control unit further acquires a second image of the object in a second posture different from the first posture, and identifies the target surface based on the first image and the second image.

[0144] (2) In the processing device of (1) above, the control unit acquires a plurality of first estimated postures obtained by estimating the posture of the object when each of the plurality of surfaces is assumed to be the target surface based on the first image, and acquires a plurality of second estimated postures obtained by estimating the posture of the object when each of the plurality of surfaces is assumed to be the target surface based on the second image, and identifies the target surface based on the plurality of first estimated postures and the plurality of second estimated postures.

[0145] (3) In the processing device of (2) above, the second posture is a posture in which the object is rotated relative to the imaging device so that the posture of the object is different from the first posture. The control unit acquires a plurality of first rotation estimated postures obtained by rotating the plurality of first estimated postures, performs a comparison process of comparing the plurality of first rotation estimated postures and the plurality of second estimated postures for each corresponding surface, and identifies the target surface based on the result of the comparison process.

[0146] (4) In the processing device of (2) above, the control unit acquires a plurality of first transformation estimated postures indicating the posture of the object shown in the first image of the object in the coordinate system of a holding mechanism based on the plurality of first estimated postures, acquires a plurality of second transformation estimated postures indicating the posture of the object shown in the second image of the object in the coordinate system of the holding mechanism based on the plurality of second estimated postures, and identifies the target surface based on the plurality of first transformation estimated postures and the plurality of second transformation estimated postures.

[0147] (5) In the processing apparatus of (1) above, the second posture is a posture in which the object is rotated relative to the imaging apparatus such that the posture of the object is different from the first posture. The control unit acquires a plurality of first estimated postures obtained by estimating the posture of the object when each of the plurality of surfaces is the target surface based on the first image, acquires a plurality of first rotation estimated postures obtained by rotating the plurality of first estimated postures, and identifies the target surface based on the degree of similarity between the plurality of first rotation estimated postures and the second image.

[0148] (6) In the processing apparatus of (5) above, the control unit is capable of performing template matching for comparing a plurality of templates and images respectively associated with the postures of the object, identifies corresponding templates corresponding to each of the plurality of first rotation estimated postures, performs template matching for comparing the corresponding templates and the second image, and identifies the target surface based on the result of the template matching.

[0149] (7) In any one of the processing apparatuses of (1) to (6) above, the control unit acquires the second image of the second posture changed by the movement of the holding mechanism that holds the object.

[0150] (8) In any one of the processing apparatuses of (1) to (6) above, the control unit acquires the second image of the second posture changed by the movement of the imaging apparatus.

[0151] (9) In any one of the processing apparatuses of (1) to (8) above, the control unit acquires a plurality of first estimated postures obtained by estimating the posture of the object when each of the plurality of surfaces is the target surface based on the first image, and sets the first estimated posture corresponding to the identified target surface among the plurality of first estimated postures as the final estimated result of the posture of the object when the first image is generated by the imaging apparatus.

[0152] (10) In any one of the processing apparatuses (2) to (9) above, the control unit sets, as the final estimated result of the posture of the object when the second image is generated by the imaging device, the second estimated posture or the first rotational estimated posture corresponding to the specified target surface among the plurality of second estimated postures or the plurality of first rotational estimated postures.

[0153] (11) In the processing apparatus (1) above, the control unit acquires a plurality of second estimated postures obtained by estimating the posture of the object assuming that each of the plurality of surfaces is the target surface based on the second image, and specifies the target surface based on the plurality of second estimated postures and the first image.

[0154] (12) In the processing apparatus (11) above, the second posture is a posture in which the object is rotated relative to the imaging device so that the posture of the object is different from the first posture. The control unit acquires a plurality of second rotational estimated postures obtained by rotating the plurality of second estimated postures, and specifies the target surface based on the plurality of second rotational estimated postures and the first image.

[0155] (13) In the processing apparatus (12) above, the control unit acquires a plurality of first estimated postures obtained by estimating the posture of the object assuming that each of the plurality of surfaces is the target surface based on the first image, and specifies the target surface based on the plurality of second rotational estimated postures and the plurality of first estimated postures.

[0156] (14) The program is a program for causing a computer device to function as any one of the processing apparatuses (1) to (13) above.

Explanation of Reference Numerals

[0157] 1 Processing apparatus 2 Control unit 4, 5 Interface 8 Imaging device 10 Object 30 Program 91 End effector (holding mechanism) 101 Front surface 102 Back surface

Claims

1. A control unit is provided that acquires a first captured image of an object having a plurality of surfaces with similar appearances from an imaging device, and identifies a target surface facing the imaging device side that appears in the first captured image among the plurality of surfaces. The control unit acquires a second captured image of the object in a second posture different from the first posture, and a processing device that identifies which surface among the plurality of surfaces the target surface is based on the first captured image and the second captured image.

2. The processing device according to claim 1, wherein the control unit acquires a plurality of first estimated postures obtained by estimating the posture of the object when each of the plurality of surfaces is assumed to be the target surface based on the first captured image, acquires a plurality of second estimated postures obtained by estimating the posture of the object when each of the plurality of surfaces is assumed to be the target surface based on the second captured image, and a processing device that identifies the target surface based on the plurality of first estimated postures and the plurality of second estimated postures.

3. The processing device according to claim 2, wherein the second posture is a posture in which the object is relatively rotated with respect to the imaging device so that the posture of the object is different from the first posture, and the control unit acquires a plurality of first rotation estimated postures obtained by rotating the plurality of first estimated postures, performs a comparison process of comparing the plurality of first rotation estimated postures and the plurality of second estimated postures for each corresponding surface, and identifies the target surface based on the result of the comparison process.

4. The processing device according to claim 2, wherein the control unit acquires a plurality of first transformation estimated postures indicating the posture of the object appearing in the first captured image of the object in the coordinate system of a holding mechanism based on the plurality of first estimated postures, acquires a plurality of second transformation estimated postures indicating the posture of the object appearing in the second captured image of the object in the coordinate system of the holding mechanism based on the plurality of second estimated postures, and a processing device that identifies the target surface based on the plurality of first transformation estimated postures and the plurality of second transformation estimated postures.

5. The processing device according to claim 1, wherein the second posture is a posture in which the object is relatively rotated with respect to the imaging device so that the posture of the object is different from the first posture, and the control unit acquires a plurality of first estimated postures obtained by estimating the posture of the object when each of the plurality of surfaces is assumed to be the target surface based on the first captured image, Obtain a plurality of first rotation estimated postures obtained by rotating the plurality of first estimated postures, A processing device that identifies the target surface based on the degree of similarity between the plurality of first rotation estimated postures and the second captured image.

6. The processing device according to claim 5, wherein The control unit, Is capable of performing template matching for comparing a plurality of templates and images with which the postures of the object are respectively associated, Identify the corresponding template corresponding to each of the plurality of first rotation estimated postures, Perform template matching for comparing the corresponding template and the second captured image, and identify the target surface based on the result of the template matching. A processing device.

7. The processing device according to any one of claims 1 to 6, wherein The control unit acquires the second captured image of the second posture changed by the movement of the holding mechanism that holds the object. A processing device.

8. The processing device according to any one of claims 1 to 6, wherein The control unit acquires the second captured image of the second posture changed by the movement of the imaging device. A processing device.

9. The processing device according to any one of claims 1 to 6, wherein The control unit, Based on the first captured image, obtain a plurality of first estimated postures obtained by estimating the posture of the object when each of the plurality of surfaces is the target surface, Among the plurality of first estimated postures, the first estimated posture corresponding to the identified target surface is used as the final estimated result of the posture of the object when the first captured image is generated by the imaging device. A processing device.

10. The processing device according to any one of claims 2 to 4, wherein Among the plurality of second estimated postures, the control unit uses the second estimated posture corresponding to the identified target surface as the final estimated result of the posture of the object when the second captured image is generated by the imaging device. A processing device.

11. The processing device according to any one of claims 3, 5, and 6, wherein Among the plurality of first rotation estimated postures, the control unit uses the first rotation estimated posture corresponding to the identified target surface as the final estimated result of the posture of the object when the second captured image is generated by the imaging device. A processing device.

12. The processing device according to claim 1, wherein The control unit, Based on the second captured image, obtain a plurality of second estimated postures obtained by estimating the posture of the object when each of the plurality of surfaces is the target surface. Identify the target surface based on the plurality of second estimated postures and the first captured image. Processing device.

13. The processing device according to claim 12, wherein the second posture is a posture in which the object is rotated relative to the imaging device such that the posture of the object is different from the first posture. The control unit, obtains a plurality of second rotation estimated postures obtained by rotating the plurality of second estimated postures. Identify the target surface based on the plurality of second rotation estimated postures and the first captured image. Processing device.

14. The processing device according to claim 13, wherein the control unit, based on the first captured image, obtains a plurality of first estimated postures obtained by estimating the posture of the object when each of the plurality of surfaces is the target surface. Identify the target surface based on the plurality of second rotation estimated postures and the plurality of first estimated postures. Processing device.

15. A program for causing a computer device to function as the processing device according to any one of claims 1 to 6 and claims 12 to 14.

Images