Three-dimensional information calculation device, three-dimensional information calculation method, three-dimensional information calculation program, and three-dimensional information calculation system

Abstract

To easily calculate three-dimensional data of a surface of an object with a simple configuration from a photographed image photographed by a monocular camera.SOLUTION: A photographed image acquisition unit 40A of a three-dimensional information calculation device 10 acquires a photographed image 50 of an object 20 with the variable shape and a structure 22 with the fixed shape and position in which the object 20 is arranged in a contact manner in at least a partial region in a real space from an imaging unit 12 being a monocular camera whose imaging position is fixed. A contour extraction unit 40B extracts a contour 52 of the object 20 included in the photographed image 50. A structure data acquisition unit 40C acquires three-dimensional data 54 of the structure 22. A contour three-dimensional data calculation unit 40D calculates three-dimensional data 58 of the contour 52 on the basis of the three-dimensional data 54 of the structure 22 and the contour 52. A three-dimensional data generation unit 40E generates three-dimensional data 60 of a surface of the object 20 from the three-dimensional data 58 of the contour 52.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] Embodiments of the present invention relate to a three-dimensional information calculation device, a three-dimensional information calculation method, a three-dimensional information calculation program, and a three-dimensional information calculation system. [Background technology]

[0002] Three-dimensional data representing the three-dimensional position coordinates of an object in real space is calculated by using images captured by an imaging unit such as a monocular camera to determine the distance to the object. For example, a method for estimating the distance between an imaging unit and an object by triangulation using multiple imaging units has been disclosed (see, for example, Patent Document 1). Methods for estimating the distance to an object from images of patterns projected onto the object by a projector, etc., and methods for installing distance measurement filters on the imaging unit have also been disclosed (see, for example, Patent Document 2). Furthermore, a method for estimating the distance between an imaging unit and an object has been disclosed using a learning model that has learned the relationship between the apparent size of the object projected onto the imaging unit and the distance between the imaging unit and the object (see, for example, Patent Document 3).

[0003] However, the technologies described in Patent Documents 1 and 2 required the separate preparation of multiple imaging units, projectors, or filters for distance measurement, which sometimes led to problems with system size and complexity. Furthermore, methods utilizing learning models, such as those described in Patent Document 3, required the preparation of a large amount of training data in advance, making them difficult to apply to environments where sufficient training data could not be prepared. In other words, with conventional technologies, it was difficult to easily calculate three-dimensional data of an object from images captured by a monocular camera using a simple configuration. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Patent No. 3324821 [Patent Document 2] Japanese Patent Publication No. 2002-13918 [Patent Document 3] Patent No. 5963353 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] The present invention has been made in view of the above, and aims to provide a three-dimensional information calculation device, a three-dimensional information calculation method, a three-dimensional information calculation program, and a three-dimensional information calculation system that can easily calculate three-dimensional data of the surface of an object from an image captured by a monocular camera with a simple configuration. [Means for solving the problem]

[0006] The three-dimensional information calculation device of this embodiment includes an image acquisition unit and a contour extraction unit, Recognition unit, second correction unit, The system comprises a structural data acquisition unit, a contour three-dimensional data calculation unit, and a three-dimensional data generation unit. The image acquisition unit acquires images of a shape-variable object and a shape- and position-fixed structure in which the object is in contact with at least a portion of the area in real space, from an imaging unit which is a monocular camera with a fixed imaging position. The contour extraction unit extracts the contours of the object included in the captured images. The recognition unit recognizes whether the object region of the object identified from the contour included in the captured image overlaps with a non-existent region in real space where the object cannot exist. The second correction unit corrects the parameters included in the position and orientation information of the camera unit so that, if the recognition result of the recognition unit indicates that the object region overlaps with the non-existent region, three-dimensional data of the contour in which the object region and the non-existent region do not overlap is calculated. The structural data acquisition unit is, Based on the depth image of the structure and the position and orientation information of the imaging unit, The three-dimensional data of the structure is acquired. The contour three-dimensional data calculation unit calculates the contour three-dimensional data based on the contour and the three-dimensional data of the structure. The three-dimensional data generation unit generates the surface three-dimensional data of the object from the contour three-dimensional data. The structure is a storage tank for storing the object, which is garbage, at its inner bottom, and has an entrance on its wall for bringing the object into the structure, a ramp provided below the entrance for guiding the object to the bottom of the structure, and a prevention member for preventing the object, which flows down the ramp from the entrance towards the bottom, from scattering in a direction intersecting the vertical direction from the ramp, and the recognition unit identifies the area in the captured image in which the prevention member is visible as the non-existent area. [Brief explanation of the drawing]

[0007] [Figure 1] A schematic diagram of the overall configuration of a three-dimensional information calculation system. [Figure 2] A block diagram of the functional configuration of a three-dimensional information calculation system. [Figure 3] A schematic diagram of the captured image. [Figure 4] Diagram illustrating contour extraction. [Figure 5] Schematic diagram of depth image. [Figure 6] Explanatory diagram of the position and orientation information of the imaging unit. [Figure 7] Explanatory diagram of the calculation of three-dimensional data of the contour. [Figure 8] Explanatory diagram of the calculation of three-dimensional data of the contour. [Figure 9A] Explanatory diagram of the generation of three-dimensional data of the surface of the object. [Figure 9B] Explanatory diagram of the generation of three-dimensional data of the surface of the object. [Figure 9C] Explanatory diagram of the generation of three-dimensional data of the surface of the object. [Figure 10A] Explanatory diagram of the control of the operation unit. [Figure 10B] Explanatory diagram of the control of the operation unit. [Figure 11A] Explanatory diagram of the correction of three-dimensional data of the surface of the object. [Figure 11B] Explanatory diagram of the correction of three-dimensional data of the surface of the object. [Figure 12A] Explanatory diagram of the state where the object region and the non-existent region overlap. [Figure 12B] Schematic diagram of three-dimensional data of the contour. [Figure 13] Flowchart of the information processing flow. [Figure 14] Hardware configuration diagram.

Embodiments for Carrying Out the Invention

[0008] Hereinafter, with reference to the accompanying drawings, a three-dimensional information calculation device, a three-dimensional information calculation method, a three-dimensional information calculation program, and a three-dimensional information calculation system will be described in detail.

[0009] FIG. 1 is a schematic diagram showing an example of the overall configuration of the three-dimensional information calculation system 1 of the present embodiment. In the present embodiment, an example in which the three-dimensional information calculation system 1 is applied to a facility for storing garbage will be described as an example. Note that the application target of the three-dimensional information calculation system 1 is not limited to a facility for storing garbage.

[0010] The three-dimensional information calculation system 1 comprises a three-dimensional information calculation device 10, an imaging unit 12, and an operation unit 14. The three-dimensional information calculation device 10, the imaging unit 12, and the operation unit 14 are connected in a communication manner.

[0011] The three-dimensional information calculation device 10 is an information processing device that generates three-dimensional data of the surface of the object 20.

[0012] Object 20 is the object from which three-dimensional data is generated. Object 20 is a variable-shape object. Variable shape means that the surface shape changes. Object 20 may be an object whose surface shape changes due to external force, an object whose surface shape changes due to internal force, or an object whose shape changes autonomously. Examples of object 20 include garbage, mud, soil, gravel, minerals, wood, pruned branches, liquids, etc. In this embodiment, the form in which object 20 is garbage will be described as an example. Garbage is an aggregate of multiple substances, and its surface shape changes when external force is applied.

[0013] The object 20 is positioned in contact with the structure 22.

[0014] The structure 22 is an object in which the object 20 is in contact with at least a portion of its area in real space. The structure 22 is fixed in shape and position. However, the shape of a portion of the structure 22 may be variable. However, the variable-shape area of ​​the structure 22 is a variable-shape area according to predetermined rules.

[0015] The structure 22 is, for example, a box-shaped member that holds or supports the object 20. In this embodiment, the structure 22 is described as a storage tank that stores the object 20, which is garbage, inside. A storage tank that stores garbage is sometimes called a garbage pit. In this embodiment, the structure 22 is described as a garbage pit installed in an incineration facility or the like for burning garbage. In this embodiment, the shape of the structure 22 is described as a substantially rectangular parallelepiped as an example. However, the shape of the structure 22 is not limited to a rectangular parallelepiped.

[0016] The structure 22, which is a waste pit, stores the waste 20 at its inner bottom. The walls of the structure 22 are provided with, for example, an entrance 22A for bringing in the waste 20, an exit 22D for discharging the waste 20 to an incinerator or the like, and a scale 22E for measuring the approximate height of the waste 20 stored in the structure 22. The entrance 22A and the exit 22D correspond to areas whose shape is variable according to predetermined rules. Below the entrance 22A on the inner wall of the structure 22, a ramp 22B is provided to guide the waste 20 to the bottom of the structure 22. The waste 20 brought into the structure 22 through the entrance 22A is stored at the bottom of the structure 22 via the ramp 22B. The ramp 22B is provided with, for example, a protective member 22C to prevent the waste 20 from scattering and to guide it to the bottom. The object 20 flows along the slope of the ramp 22B to the bottom of the structure 22, while its scattering is suppressed by the prevention member 22C, and is stored at the bottom.

[0017] The imaging unit 12 is a monocular camera with a fixed shooting position. The imaging unit 12 can be, for example, a video camera, a network camera, a single-lens reflex camera, or a camera mounted on a smartphone. The imaging unit 12 is positioned to capture the object 20 and the structure 22. The imaging unit 12 acquires image data including the object 20 and the structure 22 through shooting. Hereafter, the image data will simply be referred to as the captured image.

[0018] The operating unit 14 is a mechanism that performs operations on an object 20 in real space. The operation on the object 20 may be an operation on the entire object 20 or an operation on a part of the object 20. The operating unit 14 may be a mechanism that performs operations by physically contacting the object 20, or a mechanism that performs operations on the object 20 without physical contact using light, liquid, etc. Mechanisms that perform operations by physically contacting the object 20 include, for example, a crane that grips, stirs, moves, releases from gripping, hooks, etc., a drill that drills holes in the object 20, a shovel that excavates, etc., the object 20, etc. Mechanisms that perform operations on the object 20 without contact include, for example, a mechanism that irradiates light, a mechanism that sprays liquid, etc. If the object 20 is magnetic, the operating unit 14 may be a mechanism that holds or releases the object 20 by magnetic force.

[0019] In this embodiment, one example described is a configuration in which the operating unit 14 is a crane that performs actions such as gripping, stirring, moving, releasing from gripping, and hooking of the object 20.

[0020] The operating unit 14 is supported by a support part 16 fixed to the structure 22. The support part 16 supports the operating unit 14 so that it can move in the vertical direction (arrow Y direction) and in directions perpendicular to the vertical direction (arrow X direction, arrow Z direction). The arrow Y direction is the direction that coincides with the vertical direction. The arrow Y direction, arrow X direction, and arrow Z direction are perpendicular to each other. The vertical direction, arrow Y direction, may be described as the Y-axis or Y-axis direction, the arrow X direction as the X-axis or X-axis direction, and the arrow X direction as the Z-axis or Z-axis direction.

[0021] The operating part 14 is configured to be operable at various positions within the space of the structure 22 by being supported so as to be movable in the X, Y, and Z axes. Furthermore, the tip of the operating part 14 is configured, for example, as a drivable claw, and operations such as gripping and releasing the object 20 are performed by driving the claw.

[0022] The three-dimensional information calculation device 10 generates three-dimensional data of the surface of the object 20 using the captured image including the object 20 and structure 22 captured by the imaging unit 12. The three-dimensional information calculation device 10 also controls the operation unit 14 based on the generated three-dimensional data of the surface of the object 20.

[0023] Figure 2 is a block diagram showing an example of the functional configuration of the three-dimensional information calculation system 1. For explanatory purposes, Figure 2 also shows a portion of the object 20 and the structure 22 together.

[0024] The three-dimensional information calculation device 10 comprises a storage unit 30, a UI (user interface) unit 32, a communication unit 34, and a control unit 40. The storage unit 30, UI unit 32, communication unit 34, and control unit 40 are connected to each other via a bus 36 or the like.

[0025] The storage unit 30 stores various types of data. The storage unit 30 may be, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, a hard disk, or an optical disc. The storage unit 30 may also be a storage device located outside the three-dimensional information calculation device 10.

[0026] The communication unit 34 is a communication interface that communicates with the imaging unit 12 and the operation unit 14. The communication unit 34 may also communicate with an external information processing device via a network or the like.

[0027] The UI unit 32 has a reception function for receiving user input and a display function for displaying various information. The reception function is implemented, for example, by a pointing device such as a mouse or a keyboard. The display function is implemented, for example, by a display. The UI unit 32 may be a touch panel that integrates the reception function and the display function.

[0028] The control unit 40 performs various information processing operations in the three-dimensional information calculation device 10.

[0029] The control unit 40 includes an image acquisition unit 40A, a contour extraction unit 40B, a structure data acquisition unit 40C, a contour three-dimensional data calculation unit 40D, a three-dimensional data generation unit 40E, a position control unit 40F, an operation result acquisition unit 40G, a first correction unit 40H, a recognition unit 40I, and a second correction unit 40J.

[0030] The image acquisition unit 40A, contour extraction unit 40B, structure data acquisition unit 40C, contour three-dimensional data calculation unit 40D, three-dimensional data generation unit 40E, position control unit 40F, operation result acquisition unit 40G, first correction unit 40H, recognition unit 40I, and second correction unit 40J4 are implemented by, for example, one or more processors. For example, each of the above units may be implemented by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may be implemented by a dedicated IC or other processor, i.e., by hardware. Each of the above units may be implemented by using both software and hardware. When multiple processors are used, each processor may implement one of the above units, or two or more of the above units. Furthermore, at least one of the above units may be mounted on an external information processing device connected to the control unit 40 via a communication unit 34 and a network, etc.

[0031] The image acquisition unit 40A acquires the captured image taken by the shooting unit 12.

[0032] Figure 3 is a schematic diagram of an example of a captured image 50A. Captured image 50A is an example of a captured image 50 taken by the imaging unit 12. Captured image 50 includes the object 20 and the structure 22. More specifically, one captured image 50 includes the object 20 and the structure 22.

[0033] The captured image 50 may be either a color image or a black and white image. The shooting unit 12, for example, takes images sequentially in a time series and outputs the captured images 50 obtained by the shooting sequentially to the control unit 40. In this case, the captured image acquisition unit 40A acquires the captured images 50 sequentially from the shooting unit 12 in a time series. Alternatively, the shooting unit 12 may store the captured images 50 taken sequentially in a time series in the storage unit 30. In this case, the captured image acquisition unit 40A only needs to acquire the captured images 50 from the shooting unit 12 or the storage unit 30. That is, the captured image acquisition unit 40A may acquire the captured images 50 sequentially in real time, or it may acquire the captured images 50 from the storage unit 30 at a predetermined timing.

[0034] Returning to Figure 2, the explanation continues. The contour extraction unit 40B extracts the contours of the object 20 contained in the captured image 50.

[0035] Figure 4 is an explanatory diagram of an example of contour extraction 52. The contour extraction unit 40B identifies the object region of the object 20 included in the captured image 50 and extracts the contour of the object region as the contour 52 of the object 20.

[0036] For example, the contour extraction unit 40B extracts the object region of the object 20 contained in the captured image 50A using semantic segmentation or the like. Semantic segmentation is a technique disclosed, for example, in "O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in MICCAI. Springer, 2015, pp. 234-241."

[0037] Furthermore, the color of the object 20 and the area other than the object 20 in the captured image 50 may differ. In this case, the contour extraction unit 40B may extract the object area of ​​the object 20 using the difference in color values ​​represented by the pixel values ​​of each pixel in the captured image 50. Also, the texture of the object 20 in the captured image 50 may be complex, while the texture of the area other than the object 20 is simple. In this case, the contour extraction unit 40B may calculate the degree of texture complexity from the density of edges in the captured image 50 and extract the object area of ​​the object 20 using the difference in the degree of complexity.

[0038] The contour extraction unit 40B then extracts the contour 52 of the extracted object region. The imaging unit 12 may contain lens distortion. In this case, the contour extraction unit 40B may further perform a process to estimate the lens distortion parameter using a calibration board or the like and remove the effect of distortion from the extracted contour 52.

[0039] Returning to Figure 2, let's continue the explanation. The structural data acquisition unit 40C acquires three-dimensional data of the structure 22. The three-dimensional data of the structure 22 consists of a group of three-dimensional position coordinate data of the surface of the structure 22 in real space.

[0040] The structure data acquisition unit 40C acquires three-dimensional data of the structure 22, defining the three-dimensional position coordinate data of the surface of the structure 22 in real space for each pixel, based on the depth image of the structure 22 and the position and orientation information of the imaging unit 12.

[0041] Figure 5 is a schematic diagram of an example of a depth image 55. The depth image 55 is a depth image that defines depth information to the surface of the structure 22 for each pixel in the captured image 50 of the structure 22, which is captured by the imaging unit 12 when the object 20 is not visible. In other words, the depth image 55 is a depth image that defines depth information, which is the distance between the imaging unit 12 and the surface of the structure 22, for each pixel.

[0042] The structural data acquisition unit 40C calculates three-dimensional position coordinate data in real space for each pixel of the surface of the structure 22, based on the depth information for each pixel defined in the depth image 55 and the position and orientation information of the imaging unit 12.

[0043] Figure 6 is an explanatory diagram illustrating an example of the position and orientation information of the imaging unit 12.

[0044] The position and orientation information of the imaging unit 12 is represented by the position coordinates (x, y, z) of the imaging unit 12 in real space and the orientation θc of the imaging unit 12. The orientation θc represents, for example, the orientation of the optical axis of the lens provided on the imaging unit 12. Specifically, the orientation θc is represented by the roll, pitch, and yaw angles of the imaging unit 12. The storage unit 30 has the position and orientation information of the imaging unit 12 stored in advance. The structure data acquisition unit 40C can read the position and orientation information from the storage unit 30 and use it to calculate the three-dimensional position coordinate data of the surface of the structure 22 in real space.

[0045] Figure 6 shows, as an example, the positional relationship between a point i on a structure 22 in real space and the imaging unit 12. In this case, the three-dimensional position coordinate data of point i is calculated by the following equation (1).

[0046]

number

[0047] In equation (1), P represents the three-dimensional position coordinate data of point i in real space that is reflected in a pixel in the captured image 50. u and v represent the pixel positions of point i in the camera image. The pixel in which point i is reflected in the captured image 50 corresponds to the pixel at an angle αi from the center of the captured image 50. T represents the three-dimensional position coordinate data of the installation position of the imaging unit 12. R is the orientation θc of the imaging unit 12 expressed as a rotation matrix. d represents the distance between the imaging unit 12 and point i, i.e., depth information. c=(cx,cy) represents the position coordinate data of the optical center of the imaging unit 12 in real space. f=(fx,fy) represents the focal length of the imaging unit 12.

[0048] The structure data acquisition unit 40C calculates the three-dimensional position coordinate data of the surface of the structure 22 in real space for each pixel, using the depth information for each pixel defined in the depth image 55, the position and orientation information of the imaging unit 12, and the above equation (1). Through this calculation, the structure data acquisition unit 40C acquires the three-dimensional data 54 of the structure 22.

[0049] The structural data acquisition unit 40C may acquire three-dimensional data 54 of the structure 22 using a captured image of the structure 22 on which an AR (Augmented Reality) marker has been placed. Alternatively, the structural data acquisition unit 40C may create three-dimensional data 54 of the structure 22 from the CAD (Computer-Aided Design) data of the structure 22. Furthermore, the structural data acquisition unit 40C may acquire three-dimensional data 54 of the structure 22 by reading the three-dimensional data 54 of the structure 22 measured using a measuring tape or the like from the storage unit 30.

[0050] Returning to Figure 2, the explanation continues. The contour 3D data calculation unit 40D calculates the 3D data of the contour 52 based on the contour 52 and the 3D data 54 of the structure 22.

[0051] Figures 7 and 8 are explanatory diagrams illustrating an example of the calculation of three-dimensional data 58 of the contour 52.

[0052] As described above, in real space, the object 20 is in contact with the structure 22. In other words, in real space, the object 20 is in contact with at least a portion of the area of ​​the structure 22. Therefore, the contour three-dimensional data calculation unit 40D identifies the three-dimensional position coordinate data of the surface of the structure 22 that corresponds to the pixel in which the contact area between the object 20 and the structure 22 in real space is captured, from among the multiple pixels constituting the contour 52 included in the captured image 50, as the three-dimensional position coordinate data of the pixel constituting the contour 52.

[0053] For example, the contour three-dimensional data calculation unit 40D identifies all pixels that constitute the contour 52 included in the captured image 50 as pixels that constitute the contact area with the structure 22. Then, the contour three-dimensional data calculation unit 40D identifies the three-dimensional position coordinate data of the surface of the structure 22, which is associated with the pixels at the same pixel positions as each of all pixels that constitute the contour 52, included in the depth image 55, as the three-dimensional position coordinate data of each pixel that constitutes the contour 52.

[0054] In detail, as shown in Figure 7, the contour 3D data calculation unit 40D superimposes the contour 52 onto the depth image 55 by placing each pixel constituting the contour 52 on the same pixel position in the depth image 55. The contour 3D data calculation unit 40D then defines the three-dimensional position coordinate data of the surface of the structure 22 corresponding to the pixel at the same pixel position in the depth image 55 for each pixel constituting the contour 52. Through these processes, as shown in Figure 8, the contour 3D data calculation unit 40D calculates three-dimensional data 58 of the contour 52, with the three-dimensional position coordinate data defined for each pixel.

[0055] The conversion process from depth information to three-dimensional position coordinate data may be performed by the contour three-dimensional data calculation unit 40D. In this case, the structure data acquisition unit 40C acquires the depth image 55 of the structure 22 as three-dimensional data 54 of the structure 22. The contour three-dimensional data calculation unit 40D may then calculate the three-dimensional data 58 of the contour 52 by calculating the three-dimensional position coordinate data using the above formula (1) for the pixels at the same pixel positions as the contour 52 in the depth image 55.

[0056] Returning to Figure 2, the explanation continues. The three-dimensional data generation unit 40E generates three-dimensional data of the surface of the object 20 from the three-dimensional data 58 of the contour 52.

[0057] The three-dimensional data generation unit 40E generates three-dimensional data of the surface of the object 20 using the three-dimensional position coordinate data of each pixel that constitutes the contour 52.

[0058] Figures 9A to 9C are explanatory diagrams illustrating an example of generating three-dimensional data 60 of the surface of the object 20. Figure 9A shows an example of a captured image 50A including the object 20 and the structure 22. Figure 9B is a schematic diagram of the object 20 in real space viewed from the vertical direction, the Y-axis. Figure 9C is a schematic diagram of the object 20 in real space viewed from the Z-axis direction, which is perpendicular to the vertical direction.

[0059] The three-dimensional data generation unit 40E identifies specific points, which are five points on the contour 52 represented by the three-dimensional data 58 that satisfy predetermined conditions.

[0060] In detail, the three-dimensional data generation unit 40E identifies two points, a first point P1 and a second point P2, which lie on a first straight line L1 along a two-dimensional plane (ZX plane) that intersects the vertical direction (arrow Y direction) in real space. The three-dimensional data generation unit 40E can identify the first point P1 and the second point P2 using the three-dimensional position coordinate data of each point represented by the three-dimensional data 58 of the contour 52. Since the first point P1 and the second point P2 are two points that lie on the first straight line L1 along the two-dimensional plane which is the ZX plane, they are points with the same position in the Z-axis direction or in the X-axis direction. In this embodiment, it will be explained that the positions of the first point P1 and the second point P2 in the Z-axis direction are the same.

[0061] The first point P1 and the second point P2 may be two points that lie on the first straight line L1 along a two-dimensional plane (ZX plane) that intersects the vertical direction (arrow Y direction) in real space, but it is preferable that they also satisfy the following conditions. In particular, it is preferable that the first point P1 and the second point P2 are points located at one end and the other end of the first straight line L1 in the extension direction of the contour 52.

[0062] The three-dimensional data generation unit 40E then identifies a third point P3 that lies between the first point P1 and the second point P2 on the first line L1, among the multiple points that constitute the contour 52. The three-dimensional data generation unit 40E can identify the third point P3 using the three-dimensional position coordinate data of each point represented by the three-dimensional data 58 of the contour 52. Therefore, the positions of the first point P1, the second point P2, and the third point P3 in the Z-axis direction are the same.

[0063] Furthermore, the three-dimensional data generation unit 40E identifies a fourth point P4 located on a second line L2 that passes through the first point P1 and intersects the first line L1 on the two-dimensional plane, the ZX plane. The three-dimensional data generation unit 40E can identify the fourth point P4 using the three-dimensional position coordinate data of each point represented by the three-dimensional data 58 of the contour 52. Since the first point P1 and the fourth point P4 are two points located on the second line L2 along the two-dimensional plane, which is the ZX plane, their positions in the Z-axis direction or the X-axis direction are the same. Since the second line L2 is a line that intersects the first line L1, in this embodiment, it is explained that the positions of the first point P1 and the fourth point P4 in the X-axis direction are the same.

[0064] Furthermore, the three-dimensional data generation unit 40E identifies the fifth point P5, which lies on the third line L3 that passes through the second point P2 and intersects the first line L1 on the two-dimensional plane, the ZX plane. The three-dimensional data generation unit 40E can identify P5 using the three-dimensional position coordinate data of each point represented by the three-dimensional data 58 of the contour 52. Since the second point P2 and the fifth point P5 are two points that lie on the third line L3 along the two-dimensional plane, the ZX plane, their positions in the Z-axis direction or the X-axis direction are the same. Since the third line L3 is a line that intersects the second line L2, in this embodiment, it is explained that the positions of the second point P2 and the fifth point P5 in the X-axis direction are the same.

[0065] Through these processes, the three-dimensional data generation unit 40E identifies points P1 to P5 as specific points.

[0066] Then, the three-dimensional data generation unit 40E calculates three-dimensional position coordinate data corresponding to point Px that lies on the line L4 passing through the fourth point P4 and the fifth point P5 on the two-dimensional plane (ZX plane) within the object region of the object 20 included in the captured image 50, based on the three-dimensional position coordinate data of each specific point.

[0067] Here, the points P1 to P5, which are points along contour 52, satisfy the above conditions. Therefore, the relationship between the ratio of the straight-line distances of these points P1 to P5 on the ZX plane is expressed by the following equation (A). That is, the ratio of the distance from the third point P3 to the first point P1 on the ZY plane to the distance from the second point P2 to the third point P3 is the same as the ratio of the distance from point Px to the fourth point P4 to the distance from the fifth point P5 to point Px.

[0068] |P3-P1|:|P2-P3|=|Px-P4|:|P5-Px| Formula (A)

[0069] Therefore, the three-dimensional data generation unit 40E calculates the three-dimensional position coordinate data of point Px within the object region of the object 20 included in the captured image 50, for example, using equation (2) below.

[0070]

number

[0071] In equation (2), (X1, Y1, Z1) represents the three-dimensional position coordinate data of the first point P1. (X2, Y2, Z2) represents the three-dimensional position coordinate data of the second point P2. (X3, Y3, Z3) represents the three-dimensional position coordinate data of the third point P3. (X4, Y4, Z4) represents the three-dimensional position coordinate data of the fourth point P4. (X5, Y5, Z5) represents the three-dimensional position coordinate data of the fifth point P5.

[0072] In formula (2), v 14 This represents the distance in the Y-axis direction between the first point P1 and the fourth point P4. 25 This represents the distance in the Y-axis direction between the second point P2 and the fifth point P5. 3xrepresents the distance in the Y-axis direction between the third point P3 and the point Px. The distance in the Y-axis direction between these points represents the difference in height of the three-dimensional positions corresponding to these points in real space, that is, the distance between the three-dimensional positions in the vertical direction.

[0073] As shown in Equation (2), v, which is the distance in the Y-axis direction between the third point P3 and the point Px 3x is v 14 and v 25 is calculated by. Therefore, the three-dimensional position coordinate data corresponding to the point Px whose position in the X-axis direction is the same as that of the third point P3 and whose position in the Z-axis direction is the same as that of the fourth point P4 and the fifth point P5 can be calculated by the above Equation (2).

[0074] The three-dimensional data generation unit 40E fixes the first point P1 and the second point P2, and varies the points specified as the third point P3, the fourth point P4, and the fifth point P5 within the range that satisfies the above conditions. Then, each time the three-dimensional data generation unit 40E varies the points to be the third point P3, the fourth point P4, and the fifth point P5, it calculates the three-dimensional position coordinate data corresponding to the point Px using the above Equation (2). Through these calculation processes, the three-dimensional data generation unit 40E generates the three-dimensional data 60 of the surface of the object 20, which consists of a group of the three-dimensional position coordinate data of each point constituting the contour 52 and a group of the three-dimensional position coordinate data of each point Px in the object region, which is the region within the contour 52.

[0075] Note that the three-dimensional data generation unit 40E may calculate the three-dimensional position coordinate data corresponding to the point Px using an equation other than the above Equation (2). For example, the three-dimensional data generation unit 40E may estimate the distance v 23 in the Y-axis direction between the second point P2 and the third point P3, the distance v 21 in the Y-axis direction between the second point P2 and the first point P1, and the distance v 54 in the Y-axis direction between the fifth point P5 and the fourth point P4, and then calculate the three-dimensional position coordinate data of the point Px by estimating the distance v 5x in the Y-axis direction between the fourth point P4 and the point Px.

[0076] Specifically, the three-dimensional data generation unit 40E may calculate the three-dimensional position coordinate data corresponding to point Px using the following equation (3) instead of the above equation (2).

[0077]

number

[0078] In equation (3), (X1, Y1, Z1) represents the three-dimensional position coordinate data of the first point P1. (X2, Y2, Z2) represents the three-dimensional position coordinate data of the second point P2. (X3, Y3, Z3) represents the three-dimensional position coordinate data of the third point P3. (X4, Y4, Z4) represents the three-dimensional position coordinate data of the fourth point P4. (X5, Y5, Z5) represents the three-dimensional position coordinate data of the fifth point P5.

[0079] In formula (3), v 23 This represents the distance in the Y-axis direction between the second point P2 and the third point P3. 21 This represents the distance in the Y-axis direction between the second point P2 and the first point P1. 54 This represents the distance in the Y-axis direction between the 5th point P5 and the 4th point P4. 5x This represents the distance in the Y-axis direction between the fifth point P5 and point Px.

[0080] As shown in equation (3), v is the distance in the Y-axis direction between the fifth point P5 and point Px. 5x is, v 23 , v 21 , and v 54 It is calculated by the above equation (3). Therefore, the three-dimensional position coordinate data corresponding to point Px, whose position in the X-axis direction is the same as the third point P3 and whose position in the Z-axis direction is the same as the fourth point P4 and the fifth point P5, can be calculated by equation (3) above.

[0081] Returning to Figure 2, the explanation continues. The three-dimensional data generation unit 40E stores the three-dimensional data 60 of the surface of the generated object 20 in the storage unit 30. The three-dimensional data generation unit 40E also outputs the three-dimensional data 60 of the surface of the object 20 to the position control unit 40F. The three-dimensional data generation unit 40E may also output the three-dimensional data 60 of the surface of the generated object 20 to the UI unit 32. In this case, the three-dimensional data 60 of the surface of the object 20 will be displayed on the display. The three-dimensional data generation unit 40E may also transmit the three-dimensional data 60 of the surface of the object 20 to an external information processing device via the communication unit 34 and the network.

[0082] The position control unit 40F controls the position of the operating unit 14 in real space based on the three-dimensional data 60 of the surface of the object 20 generated by the three-dimensional data generation unit 40E. For example, the position control unit 40F controls the operating unit 14 to move to a target position based on the three-dimensional data 60 of the surface of the object 20, and to perform a predetermined operation at that target position. The predetermined operation may be, for example, gripping, stirring, moving, or releasing the object 20.

[0083] Figure 10A is an explanatory diagram illustrating an example of control of the operation unit 14 by the position control unit 40F.

[0084] In Figure 10A, the horizontal axis represents time. For example, consider a scenario where the three-dimensional data generation unit 40E generates three-dimensional data 60 of the surface of the object 20 based on the captured image 50 taken at time T1. Then, consider a scenario where the position control unit 40F sets the position represented by the three-dimensional position coordinate data of point Px2 included in the three-dimensional data 60 of the surface of the object 20 as the target position. In this case, the position control unit 40F controls the operation unit 14 to move to the target position represented by the three-dimensional position coordinate data of point Px2 and to perform a predetermined operation. For example, consider a scenario where the position control unit 40F controls the operation unit 14 to grasp a part 20A of the object 20 at the target position and move it to another predetermined position. In this case, the surface shape of the object 20 at time T2 will be different from the three-dimensional data 60 of the surface of the object 20 because a part 20A of the object 20 at the target position is grasped by the operation unit 14.

[0085] Furthermore, let's consider a scenario where the position control unit 40F sets the target position to a position represented by the three-dimensional position coordinate data of point Px3 included in the three-dimensional data 60 of the surface of the object 20. In this case, the position control unit 40F controls the operation unit 14 to move to the target position represented by the three-dimensional position coordinate data of point Px3 and to perform a predetermined operation. For example, let's consider a scenario where the position control unit 40F controls the operation unit 14 to grasp a part 20A of the object 20 at the target position and move it to another predetermined position. In this case, the surface shape of the object 20 at time T3 will be different from the three-dimensional data 60 of the surface of the object 20 because a part 20A of the object 20 at the target position is grasped by the operation unit 14.

[0086] Here, we assume a scenario in which the three-dimensional data generation unit 40E generates three-dimensional data 60 of the surface of the object 20 based on the captured image 50 taken at time T3. In this case, the three-dimensional data 60 of the surface of the object 20 generated from the captured image 50 at time T3 will be three-dimensional data that includes points P2' and Px3', which are defined as three-dimensional position coordinate data along the surface of the object 20. In this way, the three-dimensional data generation unit 40E generates three-dimensional data 60 of the surface of the object 20 based on captured images 50 acquired in a time series.

[0087] Therefore, even if the surface shape of the object 20 changes due to the operation of the operation unit 14, the three-dimensional data generation unit 40E can generate three-dimensional data 60 of the surface of the object 20 with high accuracy. Furthermore, even if the surface shape of the object 20 changes due to the operation of the operation unit 14, the position control unit 40F can control the operation unit 14 to perform operations according to the surface shape of the object 20 in real space.

[0088] Figure 10B is an explanatory diagram illustrating an example of control of the operation unit 14 by the position control unit 40F.

[0089] In Figure 10B, the horizontal axis represents time. For example, consider a scenario where the three-dimensional data generation unit 40E generates three-dimensional data 60 of the surface of the object 20 based on the captured image 50 taken at time T1. Then, consider a scenario where the position control unit 40F sets the position represented by the three-dimensional position coordinate data of point Px2 included in the three-dimensional data 60 of the surface of the object 20 as the target position. In this case, the position control unit 40F controls the operation unit 14 to move to the target position represented by the three-dimensional position coordinate data of point Px2 and to perform a predetermined operation. For example, consider a scenario where the position control unit 40F controls the operation unit 40F to grasp a part 20A of the object 20 at the target position and move it to another predetermined position. In this case, the surface shape of the object 20 at time T2 will be different from the three-dimensional data 60 of the surface of the object 20 because a part 20A of the object 20 at the target position is grasped by the operation unit 14.

[0090] Furthermore, we consider a scenario in which the operation unit 14 is controlled to supplement a portion 20B of the object 20 to the position corresponding to point Px2 in real space, so that it matches the position represented by the three-dimensional position coordinate data of point Px2 included in the three-dimensional data 60 of the surface of the object 20. In this case, the position control unit 40F controls the operation unit 14 to supplement the position corresponding to point Px2 in real space with a portion 20B of the object 20 that has been moved from another location, so that the position matches the position represented by the three-dimensional position coordinate data of point Px2 included in the three-dimensional data 60 of the surface of the object 20 (see times T3 and T4).

[0091] Thus, the position control unit 40F may control the operation unit 14 to perform various operations on the object 20 in real space so that it matches the three-dimensional data 60 of the object 20's surface generated from the captured image 50 taken at a certain time (for example, time T1).

[0092] Returning to Figure 2, the explanation continues. The operation result acquisition unit 40G acquires operation result information at the target position of the operation unit 14, which has been positioned at the target position based on the three-dimensional data 60 of the surface of the object 20.

[0093] The operation result information represents the operation result of the operating unit 14, which has been controlled to the target position, at that target position. For example, the operation result information represents the position of the operating unit 14 when it has been operated to the target position and reached the surface of the object 20, the weight of the object 20 gripped by the operating unit 14, the opening and closing status of the claw portion of the crane which is the operating unit 14, and the type of object 20 that is operated by the operating unit 14. The operation result acquisition unit 40G acquires the operation result information by acquiring the detection results of various sensors provided on the operating unit 14.

[0094] The first correction unit 40H determines whether the operation result information of the operation unit 14 represents information indicating a positional anomaly of the operation unit 14. For example, the first correction unit 40H determines that the information represents a positional anomaly if the weight of the object 20 gripped by the operation unit 14, as included in the operation result information, is less than the weight obtained when the operation unit 14 performs a gripping operation on the object 20 at the target position. Also, for example, the first correction unit 40H determines that the information represents a positional anomaly if the position of the operation unit 14, which is the position of the tip of the crane when it is operated to the target position and reaches the surface of the object 20, as included in the operation result information, is different from the three-dimensional position coordinate data represented by the target position.

[0095] If the operation result information of the operation unit 14 is information representing a positional anomaly of the operation unit 14, the first correction unit 40H corrects the parameters included in the position and orientation information of the imaging unit 12 and at least one of the three-dimensional position coordinate data of the object 20 surface, based on the operation result information, so that the target position in real space matches the three-dimensional position coordinate data of the point corresponding to the target position included in the three-dimensional data 60 of the surface of the object 20.

[0096] For example, the first correction unit 40H corrects the three-dimensional data 60 of the surface of the object 20.

[0097] Figure 11A is an explanatory diagram illustrating an example of correcting the three-dimensional data 60 of the surface of the object 20.

[0098] In Figure 11A, the horizontal axis represents time. For example, consider a scenario where the three-dimensional data generation unit 40E generates three-dimensional data 60 of the surface of object 20 based on a captured image 50 taken at time T1. The three-dimensional position coordinate data of point Px2 in this three-dimensional data 60 of the surface of object 20 may differ from the actual three-dimensional position coordinate data of the surface of object 20 in real space.

[0099] In this state, as shown at time T2, we assume that the position control unit 40F controls the operation unit 14 to move to the target position represented by the three-dimensional position coordinate data of point Px2 and to grasp a part of the object 20. In the example shown in Figure 11A, the three-dimensional position coordinate data of point Px2 and the three-dimensional position coordinate data of the actual surface of the object 20 in real space show different values, so the operation unit 14 cannot perform the operation of grasping a part of the object 20, and no weight change occurs. For this reason, for example, the operation result acquisition unit 40G acquires operation result information that includes information representing zero (0) as information representing the weight of the object 20 grasped by the operation unit 14. In this case, for example, the first correction unit 40H determines that the operation result information represents a position anomaly of the operation unit 14, and corrects and updates the three-dimensional data 60 of the surface of the object 20 so that the position anomaly is resolved.

[0100] For example, consider a scenario where the three-dimensional data 60 of the surface of object 20 is corrected at time T3. In this case, as shown in the state at time T3 in Figure 11A, the three-dimensional position coordinate data of point Px2 is corrected to a position that aligns with the actual surface shape of object 20, and the three-dimensional data 60 of the surface of object 20 is corrected to a value that aligns with the actual surface shape of object 20.

[0101] Furthermore, for example, the first correction unit 40H takes at least one of the orientation θc of the imaging unit 12 and the three-dimensional position coordinate data T of the installation position of the imaging unit 12, which are included in the position orientation information, as parameters and corrects these parameters. The structure data acquisition unit 40C then updates the parameters by storing the corrected parameters in the storage unit 30.

[0102] Therefore, when the structural data acquisition unit 40C performs the above processing using the updated parameters, the three-dimensional data generation unit 40E can generate three-dimensional data 60 of the surface of the object 20 that has been corrected based on the operation result information of the operation unit 14.

[0103] Figure 11B is an explanatory diagram illustrating another example of correcting the three-dimensional data 60 of the surface of object 20.

[0104] In Figure 11B, the horizontal axis represents time. For example, consider a scenario where the three-dimensional data generation unit 40E generates three-dimensional data 60 of the surface of object 20 based on a captured image 50 taken at time T1. The three-dimensional position coordinate data, including points Px1, Px2, and Px3, in this three-dimensional data 60 of the surface of object 20 may differ from the actual three-dimensional position coordinate data of the surface of object 20 in real space.

[0105] In this state, as shown at time T2, we assume that the position control unit 40F controls the operation unit 14 to move to the target position represented by the three-dimensional position coordinate data of point Px3 and to grasp a part of the object 20. In the example shown in Figure 11B, the three-dimensional position coordinate data of point Px3 and the three-dimensional position coordinate data of the actual surface of the object 20 in real space show different values, so the operation unit 14 is unable to grasp a part of the object 20 and no weight change occurs. In this case, the operation result acquisition unit 40G acquires operation result information that includes information representing zero (0) as information representing the weight of the object 20 grasped by the operation unit 14. In this case, for example, the first correction unit 40H determines that the operation result information represents a position anomaly of the operation unit 14 and corrects and updates the parameters so that the position anomaly is resolved.

[0106] Therefore, when the structural data acquisition unit 40C performs the above processing using the updated parameters, the three-dimensional data generation unit 40E can generate three-dimensional data 60 of the surface of the object 20 that has been corrected based on the operation result information of the operation unit 14. For example, let's assume a scenario in which the three-dimensional data 60 of the surface of the object 20 using the updated parameters is generated at time T3. In this case, as shown in the state at time T3 in Figure 11B, the three-dimensional data generation unit 40E can be corrected based on the operation result information of the operation unit 14 and generate three-dimensional data 60 of the surface of the object 20 that conforms to the actual surface shape of the object 20. That is, by correcting the parameters, the three-dimensional position coordinate data of points including points Px1, Px2, and Px3 is corrected to a position that conforms to the actual surface shape of the object 20, and the three-dimensional data 60 of the surface of the object 20 is corrected to a value that conforms to the actual surface shape of the object 20.

[0107] Returning to Figure 2, we continue the explanation.

[0108] The recognition unit 40I recognizes whether the object region of the object 20 identified from the contour 52 included in the captured image 50 overlaps with a non-existent region in real space where the object 20 cannot exist. A non-existent region is a region in real space where the object 20 cannot exist. A region where the object 20 cannot exist is a region where the probability of the object 20 existing is below a threshold due to the structure, shape, and type of the structure 22.

[0109] The structure 22 may contain areas where objects are not present. For example, as shown in Figure 1, the protective member 22C included in the structure 22 is a member that prevents objects 20 flowing on the slope 22B from scattering in a direction intersecting the vertical direction. The likelihood of objects 20 being present on this protective member 22C is low. In this case, the recognition unit 40I identifies the area in the captured image 50 where the protective member 22C is visible as an area where objects are not present.

[0110] Furthermore, as shown in Figure 1, the inner wall of the structure 22 includes a scale 22E. For example, the height in the Y-axis direction of the contour 52 identified from the three-dimensional data 58 of the contour 52 may exceed the height indicated by the scale 22E, but the contour 52 included in the captured image 50 may not reach the position of the scale 22E. In this case, the object 20 is unlikely to exist at the position of the scale 22E, and the position of the scale 22E becomes a non-existent region. In this case, the recognition unit 40I identifies the region in the captured image 50 in which the scale 22E is reflected as a non-existent region.

[0111] The recognition unit 40I then recognizes whether the object region of the object 20 identified from the contour 52 included in the captured image 50 overlaps with the identified non-existent region.

[0112] Figure 12A is an explanatory diagram illustrating an example of a state in which the object region and the non-existent region overlap. Figure 12A and Figure 12B, which will be described later, show an example of a configuration in which the non-existent region is the region of the prevention member 22C.

[0113] As shown in Figure 12A, the object region identified from the contour 52 extracted by the contour extraction unit 40B may include an overlapping region D that overlaps with the non-existent region of the prevention member 22C.

[0114] The second correction unit 40J corrects the parameters included in the position and orientation information of the imaging unit 12 so that, if the recognition result of the recognition unit 40I represents an overlap with a non-existent area of ​​the object region (for example, the prevention member 22C), three-dimensional data 58 of the contour 52 in which the object region and the non-existent area do not overlap is calculated.

[0115] The second correction unit 40J corrects and updates at least one of the parameters, attitude θc and the three-dimensional position coordinate data T of the installation position of the imaging unit 12, similar to the first correction unit 40H. That is, the second correction unit 40J corrects and updates at least one of the parameters, attitude θc and the three-dimensional position coordinate data T of the installation position of the imaging unit 12, so that three-dimensional data 58 of the contour 52 in which the object region and the non-existent region do not overlap is calculated.

[0116] Therefore, when the structural data acquisition unit 40C performs the above processing using the updated parameters, the contour three-dimensional data calculation unit 40D can calculate three-dimensional data 58 of the contour 52 in which the object region and the non-existent region do not overlap.

[0117] Figure 12B is a schematic diagram showing an example of three-dimensional data 58 of the contour 52 calculated using the updated parameters by the second correction unit 40J.

[0118] As shown in Figure 12B, when the structural data acquisition unit 40C performs the above processing using the updated parameters, the contour three-dimensional data calculation unit 40D can calculate three-dimensional data 58 of the contour 52 in which the object region and the non-existent region do not overlap. Furthermore, by using the three-dimensional data 58 of the contour 52 calculated using the updated parameters, the three-dimensional data generation unit 40E can improve the accuracy of the three-dimensional data 60 of the surface of the object 20 to be generated.

[0119] Returning to Figure 2, let's continue the explanation. The image acquisition unit 40A may acquire all of the multiple images 50 that are sequentially taken in chronological order as images 50 used to generate the three-dimensional data 60 of the surface of the object 20. Alternatively, the image acquisition unit 40A may acquire some of the multiple images 50 that are sequentially taken in chronological order as images 50 used to generate the three-dimensional data 60 of the surface of the object 20.

[0120] For example, the image acquisition unit 40A may acquire images 50 from among the images 50 captured by the imaging unit 12 that satisfy the processing target conditions, and use them as images 50 to generate three-dimensional data 60 of the surface of the object 20.

[0121] Specifically, depending on the timing of the shooting by the shooting unit 12, the operating unit 14 may be largely visible within the shooting angle of the shooting unit 12, or a part of the object 20 may be significantly lifted into the shooting angle due to operations such as gripping by the operating unit 14 and appear in the shooting angle. In addition, due to operations by the operating unit 14, a part of the object 20 that has been released from being gripped and is falling to the bottom may be visible between the shooting unit 12 and the object 20 stored at the bottom of the structure 22. If three-dimensional data 60 of the surface of the object 20 is generated using such a captured image 50, the accuracy of generating the three-dimensional data 60 of the surface of the object 20 may decrease.

[0122] Therefore, the image acquisition unit 40A may acquire images 50 from among the images 50 captured by the imaging unit 12 that satisfy the processing target conditions, and use them as images 50 to generate three-dimensional data 60 of the surface of the object 20. The processing target conditions can be predetermined to ensure that no decrease in the accuracy of the three-dimensional data 60 of the surface of the object 20 occurs. Examples of processing target conditions include: the operating unit 14 is not captured in the field of view at a size greater than a predetermined size; the operating unit 14 is not captured in the field of view; and no objects 20 other than the object 20 stored at the bottom of the structure 22 are captured in the field of view.

[0123] The image acquisition unit 40A analyzes the captured image 50 taken by the imaging unit 12 using known image processing techniques to determine whether the captured image 50 satisfies the processing conditions. For example, the image acquisition unit 40A uses known object detection techniques to identify the size of the area in the captured image 50 in which the operating unit 14 is reflected, and determines that the processing conditions are met if the identified size is less than a threshold.

[0124] Furthermore, the image acquisition unit 40A may use the operation information of the operation unit 14 at the time the image 50 was taken to determine whether the image 50 taken at that time satisfies the processing target conditions. For example, the image acquisition unit 40A can acquire operation information from the operation unit 14 that includes at least one of the opening / closing status of the claws and the weight change, and determine that an image 50 whose shooting time is the timing when the claws change from a state of gripping the object 20 to a state of releasing it, or the timing when a weight change exceeding a threshold occurs, does not satisfy the processing target conditions.

[0125] Furthermore, the image acquisition unit 40A may use both the captured image 50 and the operation information from the operation unit 14 to determine whether or not the captured image 50 satisfies the processing target conditions.

[0126] The image acquisition unit 40A acquires images 50 from among the images 50 captured by the imaging unit 12 that satisfy the processing target conditions, and uses these images 50 to generate three-dimensional data 60 of the surface of the object 20. This improves the accuracy of the three-dimensional data 60 of the surface of the object 20.

[0127] Next, an example of the information processing flow performed by the three-dimensional information calculation device 10 of this embodiment will be described.

[0128] Figure 13 is a flowchart showing an example of the information processing flow performed by the three-dimensional information calculation device 10 of this embodiment.

[0129] The image acquisition unit 40A receives the captured images 50 taken by the shooting unit 12 (step S100). For example, the image acquisition unit 40A receives the captured images 50 from the shooting unit 12. Alternatively, the image acquisition unit 40A may receive the captured images 50 by reading them from the storage unit 30. Step S100 will be explained assuming that the captured images 50 taken in chronological order are received sequentially from the shooting unit 12.

[0130] The image acquisition unit 40A determines whether the captured image 50 received in step S100 satisfies the processing conditions (step S102). If the determination in step S102 is negative (step S102: No), the process returns to step S100. If the determination in step S102 is positive (step S102: Yes), the process proceeds to step S104.

[0131] In step S104, the captured image acquisition unit 40A acquires the captured image 50 that was determined to be positive in step S102 (step S104).

[0132] The contour extraction unit 40B extracts the contour 52 of the object 20 included in the captured image 50 acquired in step S104 (step S106).

[0133] The recognition unit 40I recognizes the object region within the contour 52 extracted from the captured image 50 in step S106 (step S108). The recognition unit 40I then determines whether the object region of the object 20 identified from the contour 52 included in the captured image 50 overlaps with the non-existent region (step S110). If the determination in step S110 is negative (step S110: No), the process proceeds to step S116, which will be described later. If the determination in step S110 is positive (step S110: Yes), the process proceeds to step S112.

[0134] In step S112, the second correction unit 40J corrects the parameters included in the position and orientation information of the imaging unit 12 so that three-dimensional data 58 of the contour 52 is calculated such that the object region recognized in step S108 and the non-existent region used for the determination in step S110 do not overlap (step S112). For example, the second correction unit 40J corrects the parameters. Then, the second correction unit 40J updates the parameters stored in the storage unit 30 by storing the corrected parameters in the storage unit 30 (step S114).

[0135] Next, the structure data acquisition unit 40C acquires three-dimensional data of the structure 22 (step S116). Based on the depth image 55 of the structure 22 and the position and orientation information of the imaging unit 12 stored in the storage unit 30, the structure data acquisition unit 40C calculates three-dimensional data 54 of the structure 22, defining the three-dimensional position coordinate data of the surface of the structure 22 in real space for each pixel. Through this calculation process, the structure data acquisition unit 40C acquires three-dimensional data 54 of the structure 22.

[0136] The contour 3D data calculation unit 40D calculates 3D data 58 of the contour 52 based on the contour 52 extracted in step S106 and the 3D data 54 of the structure 22 acquired in step S116 (step S118).

[0137] The three-dimensional data generation unit 40E generates three-dimensional data 60 of the surface of the object 20 from the three-dimensional data 58 of the contour 52 calculated in step S118 (step S120).

[0138] The position control unit 40F controls the position of the operating unit 14 in real space based on the three-dimensional data 60 of the surface of the object 20 generated in step S120 (step S122).

[0139] The operation result acquisition unit 40G acquires operation result information at the target position of the operation unit 14, which has been positioned at the target position based on the three-dimensional data 60 of the surface of the object 20 as processed in step S122 (step S124).

[0140] The first correction unit 40H determines whether the operation result information obtained in step S124 represents a positional abnormality of the operation unit 14 (step S126). If the determination in step S126 is negative (step S126: No), the process proceeds to step S132, which will be described later. If the determination in step S126 is positive (step S126: Yes), the process proceeds to step S128.

[0141] In step S128, the first correction unit 40H corrects the parameters included in the position and orientation information of the imaging unit 12 so that the target position in real space matches the three-dimensional position coordinate data of the point corresponding to the target position included in the three-dimensional data 60 of the surface of the object 20, based on the operation result information acquired in step S124 (step S128). For example, the first correction unit 40H corrects the parameters. Then, the first correction unit 40H updates the parameters stored in the storage unit 30 by storing the corrected parameters in the storage unit 30 (step S130). The first correction unit 40H may also correct the three-dimensional data 60 of the surface of the object 20 so that the target position in real space matches the three-dimensional position coordinate data of the point corresponding to the target position included in the three-dimensional data 60 of the surface of the object 20.

[0142] Next, the position control unit 40F determines whether or not to terminate the control processing of the operation unit 14 using the captured image 50 that was used to generate the three-dimensional data 60 of the surface of the object 20 currently in use (step S132). If the determination in step S132 is negative (step S132: No), the process returns to step S116. If the determination in step S132 is positive (step S132: Yes), the process proceeds to step S134.

[0143] In step S134, the three-dimensional information calculation device 10 determines whether or not to terminate the process (step S134). If the determination in step S134 is negative (step S134: No), the process returns to step S100. If the determination in step S134 is positive (step S134: Yes), this routine terminates.

[0144] As described above, the three-dimensional information calculation device 10 of this embodiment comprises an image acquisition unit 40A, a contour extraction unit 40B, a structure data acquisition unit 40C, a contour three-dimensional data calculation unit 40D, and a three-dimensional data generation unit 40E. The image acquisition unit 40A acquires images 50 of a shape-variable object 20 and a structure 22 with a fixed shape and position in which the object 20 is in contact with at least a portion of the area in real space, from an image acquisition unit 12, which is a monocular camera with a fixed shooting position. The contour extraction unit 40B extracts the contour 52 of the object 20 included in the image acquisition image 50. The structure data acquisition unit 40C acquires three-dimensional data 54 of the structure 22. The contour three-dimensional data calculation unit 40D calculates three-dimensional data 58 of the contour 52 based on the contour 52 and the three-dimensional data 54 of the structure 22. The three-dimensional data generation unit 40E generates three-dimensional data 60 of the surface of the object 20 from the three-dimensional data 58 of the contour 52.

[0145] In conventional technology, calculating the three-dimensional data 60 of the surface of the object 20 requires the separate preparation of multiple imaging units, projectors, or filters for distance measurement, which can lead to problems with system size and complexity. Furthermore, conventional methods using learning models require the preparation of a large amount of training data in advance, making them difficult to apply to environments where training data cannot be prepared. In other words, conventional technology makes it difficult to easily calculate the three-dimensional data 60 of the surface of the object 20 from captured images 50 taken with a monocular camera using a simple configuration.

[0146] On the other hand, the three-dimensional information calculation device 10 of this embodiment generates three-dimensional data 60 of the surface of the object 20 from the captured images 50 of the object 20 and structure 22 taken by the imaging unit 12, which is a monocular camera.

[0147] Therefore, the three-dimensional information calculation device 10 of this embodiment does not require the separate preparation of multiple imaging units, projectors, or filters for distance measurement, and can generate three-dimensional data 60 of the surface of the object 20 from the captured image 50 without using a learning model.

[0148] Therefore, the three-dimensional information calculation device 10 of this embodiment can easily calculate three-dimensional data 60 of the surface of an object 20 from an image 50 captured by a monocular camera with a simple configuration.

[0149] Next, the hardware configuration of the three-dimensional information calculation device 10 of this embodiment will be described.

[0150] Figure 14 is a hardware configuration diagram of an example of the three-dimensional information calculation device 10 of this embodiment.

[0151] The three-dimensional information calculation device 10 of this embodiment includes a control device such as a CPU 91, a storage device such as a ROM (Read Only Memory) 92 and a RAM (Random Access Memory) 93, a communication I / F 94 that connects to a network for communication, and a bus 95 that connects each part.

[0152] The program executed by the three-dimensional information calculation device 10 of this embodiment is provided pre-installed in a ROM 92 or the like.

[0153] The program executed by the three-dimensional information calculation device 10 of this embodiment may be configured to be provided as a computer program product by recording it as an installable or executable file on a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0154] Furthermore, the program executed by the three-dimensional information calculation device 10 of this embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. Alternatively, the program executed by the three-dimensional information calculation device 10 of this embodiment may be provided or distributed via a network such as the Internet.

[0155] The program executed in the three-dimensional information calculation device 10 of this embodiment can cause the computer to function as a component of the three-dimensional information calculation device 10 of this embodiment. This computer can read the program from a computer-readable storage medium onto its main memory and execute it using the CPU 91.

[0156] Although embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. The novel embodiments described above can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0157] 1. Three-dimensional information calculation system 10 Three-dimensional information calculation device 12 Photography Department 14 Control section 20 Objects 22 Structures 40A Image acquisition unit 40B Contour extraction section 40C Structural Data Acquisition Unit 40D Contour 3D Data Calculation Unit 40E Three-dimensional data generation unit 40F Position Control Unit 40G Operation result acquisition part 40H 1st correction section 40I recognition part 40J 2nd correction section

Claims

1. An image acquisition unit acquires images of a shape-variable object and a shape- and position-fixed structure in which the object is in contact with at least a portion of the area in real space, from an imaging unit which is a monocular camera with a fixed shooting position. A contour extraction unit that extracts the contour of the object included in the captured image, A recognition unit that recognizes whether the object region of the object identified from the contour included in the captured image overlaps with a non-existent region in real space where the object cannot exist, If the recognition result of the recognition unit represents an overlap between the object region and the non-existent region, a second correction unit corrects the parameters included in the position and orientation information of the imaging unit so that three-dimensional data of the contour where the object region and the non-existent region do not overlap is calculated. A structure data acquisition unit acquires three-dimensional data of the structure based on the depth image of the structure and the position and orientation information of the imaging unit. A contour 3D data calculation unit calculates the contour 3D data based on the contour and the three-dimensional data of the structure, A three-dimensional data generation unit generates three-dimensional data of the surface of the object from the three-dimensional data of the contour, Equipped with, The aforementioned structure is A storage tank for storing the aforementioned object, which is waste, at its inner bottom, comprising: an entrance on the wall for bringing the object into the structure; a ramp provided below the entrance for guiding the object to the bottom of the structure; and a prevention member for preventing the object, which flows along the ramp from the entrance towards the bottom, from scattering in a direction intersecting the vertical direction from the ramp. The aforementioned recognition unit, The region in the captured image in which the prevention member is visible is identified as the region where the member is absent. Three-dimensional information calculation device.

2. The aforementioned structural data acquisition unit is: Based on a depth image that defines depth information to the surface of the structure for each pixel in the captured image, and the position and orientation information of the camera unit, three-dimensional data of the structure is obtained by defining the three-dimensional position coordinate data of the surface of the structure in real space for each pixel. The contour three-dimensional data calculation unit is: Based on the three-dimensional position coordinate data of the surface of the structure corresponding to the pixel in which the contact area between the object and the structure in real space is reflected, among the plurality of pixels constituting the contour included in the captured image, three-dimensional data of the contour is calculated, consisting of a group of three-dimensional position coordinate data of each pixel constituting the contour. The three-dimensional information calculation device according to claim 1.

3. The three-dimensional data generation unit is On the contour of the object, The following points are identified as specific points: a first point and a second point, which are two points that lie on a first straight line along a two-dimensional plane intersecting the vertical direction in real space; a third point that lies between the first point and the second point on the first straight line; a fourth point that lies on a second straight line that passes through the first point and intersects the first straight line on the two-dimensional plane; and a fifth point that lies on a third straight line that passes through the second point and intersects the first straight line on the two-dimensional plane. Based on the three-dimensional position coordinate data of each of the specified points, the three-dimensional position coordinate data corresponding to a point located on a straight line passing through the fourth and fifth points on the two-dimensional plane within the object region of the object included in the captured image is calculated. The three-dimensional information calculation device according to claim 2.

4. A position control unit controls the position in real space of an operating unit that performs operations on the object, based on three-dimensional data of the object's surface. A three-dimensional information calculation device according to any one of claims 1 to 3, comprising:

5. An operation result acquisition unit acquires operation result information at the target position of the operation unit, which is positioned at the target position based on three-dimensional data of the surface of the object; If the operation result information indicates an abnormality in which the weight of the object grasped by the operation unit is less than the weight obtained when the operation unit performs a gripping operation on the object at the target position, or the position of the operation unit, which is the position of the tip of the operation unit when it is operated to the target position and reaches the surface of the object, differs from the three-dimensional position coordinate data represented by the target position, then a first correction unit corrects at least one of the parameters included in the position and orientation information of the imaging unit and the three-dimensional data of the object's surface, so that the three-dimensional position coordinate data of the point corresponding to the target position in the three-dimensional data of the object's surface included in the three-dimensional data of the object's surface matches the target position in real space. It also has, The three-dimensional information calculation device according to claim 4.

6. The aforementioned image acquisition unit, From the images captured by the aforementioned imaging unit, the images that satisfy the processing conditions are acquired. A three-dimensional information calculation device according to any one of claims 1 to 5.

7. Image acquisition step: Acquires images of a shape-variable object and a shape- and position-fixed structure in which the object is in contact with at least a portion of the area in real space, from an imaging unit which is a monocular camera with a fixed shooting position. Extraction step of extracting the contour of the object included in the captured image, A recognition step of recognizing whether the object region of the object identified from the contour included in the captured image overlaps with a non-existent region in real space where the object cannot exist, If the recognition result of the recognition step represents an overlap between the object region and the non-existent region, a correction step is performed to correct the parameters included in the position and orientation information of the imaging unit so that three-dimensional data of the contour where the object region and the non-existent region do not overlap is calculated. A structural data acquisition step in which three-dimensional data of the structure is acquired based on the depth image of the structure and the position and orientation information of the imaging unit, A calculation step of calculating the three-dimensional data of the contour based on the contour and the three-dimensional data of the structure, A generation step of generating three-dimensional data of the surface of the object from the three-dimensional data of the contour, Includes, The aforementioned structure is A storage tank for storing the aforementioned object, which is waste, at its inner bottom, comprising: an entrance on the wall for bringing the object into the structure; a ramp provided below the entrance for guiding the object to the bottom of the structure; and a prevention member for preventing the object, which flows along the ramp from the entrance towards the bottom, from scattering in a direction intersecting the vertical direction from the ramp. The aforementioned recognition step is, The region in the captured image in which the prevention member is visible is identified as the region where the member is absent. Three-dimensional information calculation method.

8. A three-dimensional information calculation program to be executed by a computer, Image acquisition step: Acquires images of a shape-variable object and a shape- and position-fixed structure in which the object is in contact with at least a portion of the area in real space, from an imaging unit which is a monocular camera with a fixed shooting position. Extraction step of extracting the contour of the object included in the captured image, A recognition step of recognizing whether the object region of the object identified from the contour included in the captured image overlaps with a non-existent region in real space where the object cannot exist, If the recognition result of the recognition step represents an overlap between the object region and the non-existent region, a correction step is performed to correct the parameters included in the position and orientation information of the imaging unit so that three-dimensional data of the contour where the object region and the non-existent region do not overlap is calculated. A structural data acquisition step in which three-dimensional data of the structure is acquired based on the depth image of the structure and the position and orientation information of the imaging unit, A calculation step of calculating the three-dimensional data of the contour based on the contour and the three-dimensional data of the structure, A generation step of generating three-dimensional data of the surface of the object from the three-dimensional data of the contour, This is a three-dimensional information calculation program for causing the aforementioned computer to execute the following: The aforementioned structure is A storage tank for storing the aforementioned object, which is waste, at its inner bottom, comprising: an entrance on the wall for bringing the object into the structure; a ramp provided below the entrance for guiding the object to the bottom of the structure; and a prevention member for preventing the object, which flows along the ramp from the entrance towards the bottom, from scattering in a direction intersecting the vertical direction from the ramp. The aforementioned recognition step is, The region in the captured image in which the prevention member is visible is identified as the region where the member is absent. A program for calculating three-dimensional information.

9. The shooting unit is a monocular camera with a fixed shooting position, The image acquisition unit acquires images from the aforementioned imaging unit of an object with a variable shape, and a structure with a fixed shape and position in which the object is in contact with at least a portion of the area in real space. A contour extraction unit that extracts the contour of the object included in the captured image, A recognition unit that recognizes whether the object region of the object identified from the contour included in the captured image overlaps with a non-existent region in real space where the object cannot exist, If the recognition result of the recognition unit represents an overlap between the object region and the non-existent region, a second correction unit corrects the parameters included in the position and orientation information of the imaging unit so that three-dimensional data of the contour where the object region and the non-existent region do not overlap is calculated. A structure data acquisition unit acquires three-dimensional data of the structure based on the depth image of the structure and the position and orientation information of the imaging unit. A contour 3D data calculation unit calculates the contour 3D data based on the contour and the three-dimensional data of the structure, A three-dimensional data generation unit generates three-dimensional data of the surface of the object from the three-dimensional data of the contour, Equipped with, The aforementioned structure is A storage tank for storing the aforementioned object, which is waste, at its inner bottom, comprising: an entrance on the wall for bringing the object into the structure; a ramp provided below the entrance for guiding the object to the bottom of the structure; and a prevention member for preventing the object, which flows along the ramp from the entrance towards the bottom, from scattering in a direction intersecting the vertical direction from the ramp. The aforementioned recognition unit, The region in the captured image in which the prevention member is visible is identified as the region where the member is absent. Three-dimensional information calculation system.

Images