3D object area estimation device and program

Abstract

To provide a three-dimensional subject region estimation device which can estimate a three-dimensional subject region of a subject from a photographed image.SOLUTION: A three-dimensional subject region estimation device 1 comprises: two-dimensional subject region estimation means 10 which estimates a two-dimensional subject region from a photographed image; projection line calculation means 110 which calculates a straight line passing through the gravity center of world coordinates of the two-dimensional subject region and the optical center of a camera as a projection line; nearest point perpendicular foot calculation means 111 which calculates a position of a perpendicular foot to each projection line from the nearest point of the plurality of projection lines; subject region apex calculation means 112 which projects an apex of the two-dimensional subject region to a projection surface projecting the gravity center of the two-dimensional subject region to the position of the perpendicular foot to calculate the world coordinates of the apex; and three-dimensional shape approximation means 113 which approximates all the apexes with the three-dimensional shape and forms the three-dimensional subject region.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a three-dimensional object region estimation device and a program therefor. [Background technology]

[0002] Conventionally, there are various methods for estimating a subject area from an image captured by a camera. For example, there is a method of recognizing a subject based on color information from the prior information of a captured image and estimating the subject area in three-dimensional space using the principle of triangulation with two cameras (see Patent Document 1), and a method of generating a three-dimensional model by combining a volume intersection method and a stereo matching method using a silhouette image extracted by chromakey processing or the like (see Non-Patent Document 1). Recently, there has been a technology that uses deep learning to estimate a two-dimensional object region in a captured image without using prior information such as the color or shape of the object (see Non-Patent Document 2).There is also a technology that uses deep learning to estimate the position of an object in three-dimensional space from a single image (see Non-Patent Document 3). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 9-322179 [Non-patent literature]

[0004] [Non-Patent Document 1] Tomiyama, H., Katayama, M., Iwadate, Y., and Imaizumi, H., "3D Dynamic Object Generation from Multi-View Images Using Volume Intersection and Stereo Matching," Journal of the Institute of Image Information and Television Engineers, no. 58, vol. 6, pp. 797-806, 2004. [Non-patent document 2] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi, “You Only Look Once: Unified, Real-Time Object Detection”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779-788, 2016. [Non-patent document 3] Adel Ahmadyan, Liangkai Zhang, Artsiom Ablavatski, Jianing Wei, Matthias Grundmann, “Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7822-7831, 2021. Summary of the Invention [Problem to be solved by the invention]

[0005] The method described in Patent Document 1 requires prior information such as color information to estimate the subject region, and therefore this method requires a separate device for acquiring the prior information. The method described in Non-Patent Document 1 generates the shape of a three-dimensional model with high accuracy, which results in high calculation costs. The method described in Non-Patent Document 2 uses deep learning to estimate a two-dimensional object region in a captured image. Applying this method to estimate the object region increases costs in proportion to the number of cameras and resolution characteristics. The method described in Non-Patent Document 3 estimates a three-dimensional object region from a single captured image, so length information such as the distance between the stereo cameras is undefined. Therefore, this method does not determine the scale of the object region. Furthermore, because this method uses only a single captured image, it is impossible to avoid erroneous estimation due to occlusion.

[0006] The present invention has been made in consideration of such problems, and aims to provide a three-dimensional object area estimation device and a program therefor that are capable of estimating a three-dimensional object area from images captured by multiple cameras, even if the number of cameras is small or the cameras have low resolution. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems, the three-dimensional subject area estimation device of the present invention is a three-dimensional subject area estimation device that estimates the three-dimensional subject area of ​​a subject from a captured image, and is equipped with a two-dimensional subject area estimation means and a three-dimensional subject area reconstruction means, and the three-dimensional subject area reconstruction means is configured to include a projection line calculation means, a nearest point droop calculation means, a subject area vertex calculation means, and a three-dimensional shape approximation means.

[0008] In this configuration, the 3D object region estimation device estimates a 2D object region for each image captured by multiple cameras using a 2D object region estimation means. The 2D object region can be estimated using a machine learning model such as a neural network that has been trained in advance to detect the object region from the captured image.

[0009] Then, the three-dimensional object region estimation device reconstructs the three-dimensional object region of the object from the two-dimensional object regions of each camera by the three-dimensional object region reconstruction means. That is, the three-dimensional object region estimation device converts the center of gravity of the two-dimensional object region into world coordinates for each camera using a projection line calculation means, and calculates a straight line passing through the center of gravity in the world coordinates and the optical center of the camera as a projection line. Then, the three-dimensional object region estimation device calculates the position of the vertical foot from the nearest point of the projection line to each projection line for each camera by the nearest point vertical foot calculation means.

[0010] Then, the 3D object region estimation device uses the object region vertex calculation means to project the vertices of the 2D object region onto a projection plane that projects the center of gravity of the 2D object region onto the position of the foot drop, and calculates the world coordinates of the vertices after projection, thereby specifying the positions of the vertices of the 2D object region on the world coordinates for each camera. Then, the three-dimensional object region estimation device uses the three-dimensional shape approximation means to approximate the positions of the vertices of the world coordinates after projection with a three-dimensional shape, and calculates shape information of the three-dimensional shape as the three-dimensional object region.

[0011] In this way, the three-dimensional object region estimation device estimates a three-dimensional object region in a three-dimensional shape from a plurality of captured images. The three-dimensional object region estimation device can be operated by a three-dimensional object region estimation program that causes a computer to function as each of the above-mentioned means. [Effects of the Invention]

[0012] According to the present invention, a three-dimensional object region can be estimated quickly and accurately from images captured by a plurality of cameras. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is an overall configuration diagram showing the configuration of a subject region estimation system including a three-dimensional subject region estimation device according to a first embodiment of the present invention. [Figure 2] 1 is a block diagram showing the configuration of a three-dimensional object region estimation device according to a first embodiment of the present invention. [Figure 3] FIG. 2 is an explanatory diagram for explaining an example of a two-dimensional subject region estimated by a subject region estimation means. [Figure 4] FIG. 10 is an explanatory diagram for explaining a projection line in a projection line calculation means. [Figure 5] FIG. 10 is an explanatory diagram for explaining the position of the vertical foot in the nearest neighbor vertical foot calculation means. [Figure 6] 10 is an explanatory diagram for explaining projection of a two-dimensional subject region onto a projection plane in which the position of the dropping foot in the subject region vertex calculation means coincides with the center of gravity of the two-dimensional subject region. FIG. [Figure 7] 10 is an explanatory diagram for explaining the projection of vertices of a two-dimensional object region onto a projection plane in object region vertex calculation means. FIG. [Figure 8] FIG. 10 is an explanatory diagram for explaining that a subject exists in a two-dimensional subject area after projection. [Figure 9] FIG. 10 is an explanatory diagram for explaining that the subject exists within the vertices of the two-dimensional subject area after projection. [Figure 10] 10 is an explanatory diagram for explaining approximation of the vertices of a two-dimensional subject region to a spherical shape in a three-dimensional shape approximation means. FIG. [Figure 11] 4 is a flowchart showing the operation of the three-dimensional object region estimation device according to the first embodiment of the present invention. [Figure 12] FIG. 10 is an overall configuration diagram showing the configuration of a subject region estimation system including a three-dimensional subject region estimation device according to a second embodiment of the present invention. [Figure 13] FIG. 10 is a block diagram showing the configuration of a three-dimensional object region estimation device according to a second embodiment of the present invention. [Figure 14] 10 is a flowchart showing the operation of the three-dimensional object region estimation device according to the second embodiment of the present invention. [Figure 15] FIG. 10 is a block diagram showing the configuration of a three-dimensional object region estimation device according to a modified example. [Figure 16] 10A and 10B are explanatory diagrams for explaining an example of visualization of a subject region by a subject region visualization means. [Figure 17] 10A and 10B are diagrams showing modified examples of a two-dimensional subject area, in which (a) shows the entire subject area, (b) shows a bust-up area of ​​the subject, and (c) shows the face area of ​​the subject. [Figure 18] FIG. 10 is an explanatory diagram illustrating an example in which multiple target regions are set for one subject. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. [Subject Area Estimation System: First Embodiment] The configuration of a subject region estimation system including a three-dimensional subject region estimation device according to a first embodiment of the present invention will be described with reference to FIG.

[0015] The subject region estimation system 1000 captures an image of a subject S using two or more cameras 2, and estimates the region of the subject S in a three-dimensional space from the captured images. As shown in FIG. 1, the subject region estimation system 1000 includes a three-dimensional subject region estimation device 1 and two or more cameras 2 (21, 22, . . . ).

[0016] The three-dimensional object region estimation device 1 estimates the region of an object S in three-dimensional space (three-dimensional object region) from a plurality of captured images taken by a plurality of cameras 2 (21, 22, . . . ).

[0017] The camera 2 captures an image of the subject S. The camera 2 is a fixed camera whose image capturing position is determined in advance. This camera 2 can be a general video camera that captures the subject S. Note that, in order to estimate the three-dimensional subject area, it is necessary to capture images of the subject S from multiple directions, so at least two cameras 2 are required. The camera 2 is connected to the three-dimensional object region estimation device 1 by wire or wirelessly, and outputs captured images (frame images) in time series to the three-dimensional object region estimation device 1. The configuration and operation of the three-dimensional object region estimation device 1 will be described below.

[0018] [Configuration of 3D object area estimation device] The configuration of a three-dimensional object region estimation device 1 according to a first embodiment of the present invention will be described with reference to FIG. 2 (and also with reference to FIG. 1 as needed). Here, the three-dimensional object region estimation device 1 comprises two-dimensional object region estimation means 10 and three-dimensional object region reconstruction means 11.

[0019] The two-dimensional subject area estimation means 10 estimates a two-dimensional subject area (two-dimensional subject area) of an input photographed image for each camera 2 (21, 22, . . . ). The two-dimensional object region estimation means 10 includes a camera calibration means 100 and an object region estimation means 101 .

[0020] The camera calibration means 100 performs camera calibration of the camera 2 (21, 22, . . . ). Here, camera calibration is a process of estimating the camera parameters of the camera 2. The camera parameters include the camera position, camera attitude, focal length, principal point position, etc. of camera 2. The camera position is the world coordinate of the optical center of camera 2. The camera attitude is the three-axis attitude angles (tilt, pan, roll) and rotation matrix that define the attitude of the imaging surface relative to the world coordinate. Note that camera calibration can be performed using general techniques described in, for example, the following reference documents, JP 2018-189580 A, etc., and therefore detailed explanations will be omitted here. (Reference) Zhengyou Zhang, “A flexible new technique for camera calibration”, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.22, no. 11, pp. 1330-1334, 2000.

[0021] The camera calibration means 100 performs camera calibration as a preprocessing before the three-dimensional object region estimation device 1 estimates the three-dimensional object region. Camera calibration means 100 outputs the estimated camera parameters to three-dimensional object region reconstruction means 11. Note that camera calibration means 100 may be configured separately as a camera calibration device outside three-dimensional object region estimation device 1.

[0022] The subject area estimation means 101 estimates a two-dimensional subject area for each image captured by a plurality of cameras 2 (21, 22, . . . ). The subject area estimation means 101 can estimate the subject area from the captured image by general object detection. For example, the subject area estimation means 101 estimates a two-dimensional subject area from the captured image captured by the camera 2 using a machine learning model such as a neural network that has been trained in advance to detect the subject area from the captured image. For this region estimation by object detection, a Single Shot Multibox Detector (SSD), You Only Look Once (YOLO; Non-Patent Document 2) v5, etc. can be used. Note that, although an example of the subject S will be described here as a person, it does not necessarily have to be a person. That is, the subject region estimation means 101 can set any object as the subject by learning in advance something other than a person as a machine learning model.

[0023] 3, the subject region estimation means 101 determines a rectangular region (bounding box) surrounding the subject S in the captured image I as a two-dimensional subject region A. Note that the two-dimensional subject region may be a triangle or a polygon with pentagons or more sides. The subject area estimation means 101 outputs the estimated two-dimensional subject area for each camera 2, specifically the positions of the vertices of the two-dimensional subject area on the image plane of the captured image, to the three-dimensional subject area reconstruction means 11.

[0024] The three-dimensional object region reconstructing means 11 reconstructs the three-dimensional object region of the object S from the two-dimensional object region for each camera estimated by the two-dimensional object region estimating means 10 based on the camera parameters. The three-dimensional object region reconstruction means 11 includes a projection line calculation means 110 , a nearest point vertical foot calculation means 111 , an object region vertex calculation means 112 , and a three-dimensional shape approximation means 113 .

[0025] The projection line calculation means 110 converts the center of gravity (center of the bounding box) of the two-dimensional subject area for each camera 2 into world coordinates, and calculates a straight line passing through the center of gravity in the world coordinates and the optical center of the camera as a projection line. The projection line calculation means 110 outputs information specifying the projection line for each camera 2 (here, the camera origin (optical center) and the inclination of the projection line) to the three-dimensional object region reconstruction means 11.

[0026] Here, the method of calculating the projection line L in the projection line calculation means 110 will be specifically described with reference to FIG. Figure 4 shows that in a pinhole model where the camera position (optical center) is the camera origin C, a two-dimensional object area A exists on an image plane IP separated from the camera origin C by the focal length. The coordinates (u, v) of a pixel position on the image plane IP and the coordinates (x, y, z) in the world coordinate system have the relationship of the following equation (1).

[0027]

number

[0028] where (f x ,f y ) is the focal length of camera 2, (c x ,c y ) indicates the principal point position of camera 2. R indicates the orientation (rotation matrix) of the camera coordinate system in the world coordinate system, and t indicates the position of the origin of the camera coordinate system in the world coordinate system. Note that s is the scale that matches the third row of the matrix on the left side with the fourth row of the matrix on the right side. Here, the coordinates of the center of gravity G of the two-dimensional object area A on the image plane IP are (u G ,v G ), the coordinates of the center of gravity G in the world coordinate system (x G ,y G ,z G ) can be calculated using the following formula (2).

[0029]

number

[0030] In addition, [R t] -1 can be solved as a pseudo-inverse matrix. That is, the projection line calculation means 110 calculates the coordinates (x C ,y C ,z C ) and the slope of which is given by the following equation (3) is determined as the projection line L.

[0031]

number

[0032] Returning to FIG. 2, the description of the configuration of the three-dimensional object region estimation device 1 will be continued. The nearest point vertical foot calculation means 111 calculates the position of the vertical foot from the nearest point on the projection line for each camera 2 to each projection line. Here, the vertical foot is the point where the foot of a vertical line dropped on a straight line (here, the projection line) intersects with the straight line. The nearest neighbor point vertical foot calculation means 111 calculates the vertical feet P1, P2, . . . , P N The nearest point Q is calculated so that the sum of the distances (N is the number of cameras 2) is the smallest. That is, the nearest point vertical foot calculation means 111 calculates the nearest point Q by solving the least squares problem shown in the following equation (4). Note that, n Q|| is the vector P n Indicates the length of Q.

[0033]

number

[0034] The method for solving equation (4) is a common method, and therefore the explanation will be omitted. Then, the nearest point vertical foot calculation means 111 calculates the vertical foot P n Camera 2 with (n=1,2,…,N) on the projection line n Camera origin C n Vector C in n P nand the vector QP n Since and are perpendicular, the vertical foot P n Calculate.

[0035]

number

[0036] The nearest neighbor point vertical foot calculation means 111 outputs the calculated position of the vertical foot of the projection line for each camera 2 to the object area vertex calculation means 112 . Here, the nearest point vertical foot calculation means 111 calculates the nearest point Q and then calculates each vertical foot P n However, when there are two cameras 2, the nearest point vertical foot calculation means 111 does not necessarily need to calculate the nearest point Q. This is because when there are two cameras 2, the straight line that is the shortest distance between the two projection lines is always perpendicular to the projection lines.

[0037] Here, with reference to FIG. 5, a method in which the nearest point vertical foot calculation means 111 calculates the vertical foot from the nearest point to the projection line when there are two cameras 2 will be described. Figure 5 shows two projection lines L1 and L2 at skewed positions passing through the centers of gravity G1 and G2 of two-dimensional subject regions A1 and A2 on image planes IP1 and IP2, which are spaced apart from the two camera origins C1 and C2 by the focal length. The vertical feet of the projection lines L1 and L2 from the nearest point Q are shown as P1 and P2. For convenience, the nearest point Q is shown in Figure 5, but is not used in this method. In this case, the projection lines L1 (vector C1G1) and L2 (vector C2G2) and the line (vector P1P2) connecting P1 and P2 are perpendicular to each other, and therefore satisfy the relationship of the following formula (6).

[0038]

number

[0039] Here, with m1 and m2 as variables, the following equation (7) holds.

[0040]

number

[0041] From this equation (7), the vector P1P2 can be transformed into the following equation (8).

[0042]

number

[0043] Substituting this vector P1P2 into equation (6) and rearranging it, m1 and m2 become the following equation (9).

[0044]

number

[0045] By substituting these m1 and m2 into equation (7), the vertical feet P1 and P2 can be found from the known world coordinates of the camera origins C1 and C2 and the centers of gravity G1 and G2. That is, the nearest point vertical foot calculation means 111 calculates vertical feet P1 and P2 by substituting m1 and m2 into equation (7). Returning to FIG. 2, the description of the configuration of the three-dimensional object region estimation device 1 will be continued.

[0046] The object area vertex calculation means 112 projects the vertices of the two-dimensional object area onto a projection plane that projects the center of gravity of the two-dimensional object area onto the position of the foot drop for each camera 2, and calculates the world coordinates of the vertices after projection. The subject area vertex calculation means 112 outputs the world coordinates (three-dimensional coordinates) corresponding to the calculated vertices to the three-dimensional shape approximation means 113.

[0047] Here, with reference to FIGS. 6 and 7, a method for calculating three-dimensional coordinates corresponding to the vertices of the two-dimensional subject area in the subject area vertex calculation means 112 will be specifically described. Figure 6 shows a state in which, in a pinhole model in which the camera position (optical center) is the camera origin C, a two-dimensional subject area A exists on an image plane IP that is separated by the focal length from the camera origin C. Figure 6 also shows a two-dimensional subject area A' obtained by projecting the two-dimensional subject area A onto a projection plane PP parallel to the image plane IP, where G', the projection of the center of gravity G of the two-dimensional subject area A, coincides with the position of the foot P. FIG. 7 shows only the two-dimensional subject area A and the two-dimensional subject area A' after projection extracted from FIG. Here, the camera origin C, center of gravity G, and vertical foot P satisfy the relationship of the following equation (10) based on the same relationship as that of the above equation (7).

[0048]

number

[0049] Here, m1 is the value calculated by the above formula (9). Similarly, in a pinhole model in which the camera position (optical center) is the camera origin C, the relationship between the three-dimensional coordinates of vertices V0, V1, V2, and V3 of the two-dimensional subject area A and the three-dimensional coordinates of vertices V0′, V1′, V2′, and V3′ after projection satisfies the following equation (11).

[0050]

number

[0051] The subject area vertex calculation means 112 calculates the world coordinates (three-dimensional coordinates) of the vertices V0, V1, V2, and V3 of the two-dimensional subject area A on the image plane IP by inputting the image coordinates of the vertices V0, V1, V2, and V3 of the two-dimensional subject area A in place of the coordinates of the center of gravity in equation (2). Then, the subject area vertex calculation means 112 can calculate the three-dimensional coordinates of the vertices V0', V1', V2', V3' of the projected two-dimensional subject area A' using the world coordinates of the calculated vertices V0, V1, V2, V3, the three-dimensional coordinates of the known camera origin C, and the value m1, according to the above equation (11).

[0052] The center of gravity G' of the two-dimensional subject area A' after this projection is located at the position of the vertical foot P from the nearest point of the projection line L of the multiple cameras 2 connecting the camera origin C and the center of gravity G of the two-dimensional subject area A. Therefore, the subject S is present in the area specified by the projected two-dimensional subject area A′ calculated for each camera 2.

[0053] For example, as shown in Figures 8 and 9, when there are two cameras 2, the centers of gravity G1' and G2' of the two-dimensional subject areas A1' and A2' after projection are located at the position of the vertical foot from the nearest point of multiple projection lines L1 and L2 connecting the camera origins C1 and C2 and the centers of gravity of the two-dimensional subject areas before projection. Therefore, the object S is located at the vertex V 10 ′,V 11 ′,V 12 ′,V 13 ' and exists in the two-dimensional object area A1' of vertex V 20 ′,V 21 ′,V 22 ′,V 23 ' will be present in the two-dimensional object region A2'. In this way, the subject S exists in a three-dimensional spatial region specified by the vertices of the two-dimensional subject regions A1' and A2' after projection. Returning to FIG. 2, the description of the configuration of the three-dimensional object region estimation device 1 will be continued.

[0054] The three-dimensional shape approximation means 113 approximates the positions of the vertices in the world coordinates after projection calculated by the subject area vertex calculation means 112 with a three-dimensional shape, and calculates shape information of the three-dimensional shape as the three-dimensional subject area. In this way, the three-dimensional shape approximation means 113 can express the area in three-dimensional space where the subject exists as a single three-dimensional shape from the point group of multiple vertices. Here, the three-dimensional shape approximation means 113 approximates all the vertices calculated by the subject area vertex calculation means 112 with a spherical shape, and calculates shape information of the approximated sphere as the three-dimensional subject area. That is, as shown in FIG. 10, the three-dimensional shape approximation means 113 calculates all the vertices calculated by the object region vertex calculation means 112, here, V 10 ′,V 11 ′,V 12 ′,V 13 ′,V 20 ′,V 21 ′,V 22 ′,V 23 ' is approximated by the shape of a sphere SP. This allows the three-dimensional shape approximation means 113 to identify the three-dimensional subject area of ​​the subject S by the position and size of the sphere.

[0055] Here, the three-dimensional shape approximation means 113 calculates the shape of the sphere (center position and radius) that minimizes the sum of the differences between the distance from each vertex to the center position of the sphere and the radius of the sphere. That is, the three-dimensional shape approximation means 113 calculates the shape of the sphere (center position and radius) by solving the least squares problem shown in the following equation (12). Note that V n ' indicates the world coordinates of the vertices calculated by the object area vertex calculation means 112, and n takes a value from 1 to M (M is the total number of vertices). s is the center position of the sphere (3D coordinates), and r is the radius of the sphere. n ′-X s || is V n ′ and X s Indicates the distance to.

[0056]

number

[0057] The three-dimensional shape approximation means 113 outputs the calculated shape of the sphere (center position and radius) as the three-dimensional subject area. The solution to equation (12) will be explained below using mathematical formulas. In order for the above-mentioned equation (12) to take the minimum value, the following equations (13) and (14) must be satisfied.

[0058]

number

[0059] From the above formula (13), the radius r is given by the following formula (15).

[0060]

number

[0061] Moreover, the above formula (14) can be transformed into the following formula (16).

[0062]

number

[0063] The inner product has the relationship of the following equation (17) using transpose T. Note that a, b, and c are vectors of the same dimension.

[0064]

number

[0065] Therefore, part of the right side of the above equation (16) can be transformed into the following equations (18) and (19).

[0066]

number

[0067] By substituting the transformed expressions of the expressions (18) and (19) into the expression (16) and transforming it, the following expression (20) can be derived.

[0068]

number

[0069] That is, the three-dimensional shape approximation means 113 estimates the object region in three-dimensional space using equation (20) by estimating the center position Xs Calculate. The three-dimensional shape approximation means 113 also calculates the center position X s is substituted into the above equation (15) to calculate the radius of the sphere.

[0070] As described above, the three-dimensional object region estimation device 1 can accurately estimate a three-dimensional object region from a plurality of captured images even with a small number of cameras. Furthermore, the three-dimensional object region estimation device 1 estimates the three-dimensional object region by a simple process of projecting the two-dimensional object region for each camera 2, and therefore can perform estimation at high speed. The three-dimensional object region estimation device 1 can be operated by a program (three-dimensional object region estimation program) that causes a computer to function as each of the above-mentioned units.

[0071] [Operation of 3D object area estimation device] Next, the operation of the three-dimensional object region estimation device 1 according to the first embodiment of the present invention will be described with reference to FIG. 11 (and for the configuration, refer to FIG. 2 as appropriate). It is assumed here that the camera calibration means 100 has previously performed camera calibration for the cameras 2 (21, 22, . . . ) and acquired the camera parameters.

[0072] In step S1, the subject area estimation means 101 uses a trained machine learning model to estimate a two-dimensional subject area for each camera 2 from the images captured by the cameras 2 (21, 22, ...). Here, the two-dimensional subject area is defined as a bounding box (see FIG. 3). In step S2, projection line calculation means 110 converts the center of gravity of the two-dimensional object region (center of the bounding box) into world coordinates for each camera, and calculates a straight line from the camera origin (optical center) through the center of gravity as a projection line (see FIG. 4). The center of gravity of the two-dimensional object region is projected onto this projection line for each camera 2.

[0073] In step S3, the nearest point vertical foot calculation means 111 calculates the position of the vertical foot perpendicularly extending from the nearest point of the projection lines of the multiple cameras 2 calculated in step S2 to each projection line (see FIG. 5). Based on this vertical foot position, the nearest point vertical foot calculation means 111 can identify the position where the center of gravity of the two-dimensional subject area is projected. In step S4, the subject area vertex calculation means 112 projects the vertices of the two-dimensional subject area onto a projection plane that projects the center of gravity of the two-dimensional subject area onto the position of the foot drop calculated in step S3, and calculates the world coordinates of the vertices after projection (see Figures 6 to 9). The subject is located inside the vertices of the bounding box calculated for each camera 2.

[0074] In step S5, the three-dimensional shape approximation means 113 approximates all the vertices (three-dimensional coordinates) calculated in step S4 to the surface of a sphere, and calculates the shape of the approximated sphere (center position and radius) as the three-dimensional subject area (see FIG. 10). In step S6, the three-dimensional object region estimation device 1 determines whether or not an instruction to end estimation has been given by an external instruction or the like. If the instruction to end the estimation is not given (No in step S6), the three-dimensional object region estimation device 1 returns to step S1 and continues the operation of estimating the three-dimensional object region. On the other hand, if an instruction to end estimation is given (Yes in step S6), the three-dimensional object region estimation device 1 ends the operation.

[0075] Through the above operations, the three-dimensional object region estimation device 1 can quickly estimate a three-dimensional object region from a plurality of captured images by simple processing of projecting a two-dimensional object region. In this way, the three-dimensional subject area estimation device 1 can estimate the three-dimensional subject area at high speed, making it possible, for example, to estimate the three-dimensional subject area of ​​a performer on stage in real time and, in conjunction with this, control the lighting in accordance with the performer's movements.

[0076] [Subject area estimation system: second embodiment] Next, with reference to FIG. 12, a configuration of a subject region estimation system including a three-dimensional subject region estimation device according to a second embodiment of the present invention will be described.

[0077] The subject area estimation system 1000B photographs the subject S with two or more cameras 2, estimates the area of ​​the subject S in three-dimensional space from the photographed images, and tracks and photographs the subject S with one or more photographing cameras 3. Here, as shown in FIG. 12, subject region estimation system 1000B is made up of three-dimensional subject region estimation device 1B, two or more cameras 2 (21, 22, . . . ), and one or more photographing cameras 3 (31, 32, . . . ).

[0078] The three-dimensional subject area estimation device 1B estimates the area of ​​the subject S in three-dimensional space (three-dimensional subject area) from multiple captured images taken by multiple cameras 2 (21, 22, ...), and controls the capturing cameras 3 (31, 32, ...) to follow the subject S. The camera 2 is the same as the camera described in FIG. 1, and therefore a description thereof will be omitted.

[0079] The photographing camera 3 photographs the subject S under the control of the three-dimensional subject region estimation device 1B. The photographic camera 3 may be a robot camera whose pan, tilt, zoom or translation can be controlled, or a camera mounted on a drone whose position, attitude and zoom can be controlled. The photographing camera 3 is connected to the three-dimensional object region estimation device 1B by wire or wirelessly, and operates under the control of the three-dimensional object region estimation device 1B. The configuration and operation of the three-dimensional object region estimation device 1 will be described below.

[0080] [Configuration of 3D object area estimation device] The configuration of a three-dimensional object region estimation device 1B according to a second embodiment of the present invention will be described with reference to FIG. 13 (and also with reference to FIG. 12 as needed). Here, the three-dimensional object region estimation device 1B comprises two-dimensional object region estimation means 10, three-dimensional object region reconstruction means 11, and camera control means 12.

[0081] The two-dimensional object region estimation means 10 and the three-dimensional object region reconstruction means 11 have the same configuration as the three-dimensional object region estimation device 1 explained in FIG. 2, and therefore explanation thereof will be omitted. However, in the two-dimensional object region estimation means 10 of the three-dimensional object region estimation device 1B, in addition to the cameras 2 (21, 22, ...), the camera calibration means 100 performs camera calibration for the photographing cameras 3 (31, 32, ...) at predetermined reference positions. The camera calibration means 100 outputs the camera parameters of the photographing cameras 3 (31, 32, ...) to the camera control means 12 (not shown).

[0082] The camera control means 12 controls the photographing camera 3 so that the three-dimensional object region of the object S reconstructed by the three-dimensional object region reconstructing means 11 is contained within the angle of view. The camera control means 12 includes a driving amount calculation means 120 and a photographing camera control means 121 .

[0083] The drive amount calculation means 120 calculates the drive amount of the photographing camera 3 for fitting the three-dimensional subject area of ​​the subject S within the angle of view. For example, if the photographing camera 3 is a robot camera, the drive amount calculation means 120 calculates the pan, tilt, and zoom drive amounts or the translation amount (movement amount) of the camera position from the current pan, tilt, and zoom values ​​or camera position of the photographing camera 3 to fit the three-dimensional subject area of ​​the subject S within the angle of view. The current pan, tilt, and zoom values ​​and camera position of the photographing camera 3 may be stored sequentially in a storage means (not shown), or may be acquired from the photographing camera 3.

[0084] In addition, for example, if the photographing camera 3 is a camera mounted on a drone, the drive amount calculation means 120 calculates the drive amounts of the position, attitude and zoom of the photographing camera 3 from the current position, attitude and zoom values ​​of the photographing camera 3 in order to fit the three-dimensional subject area of ​​the subject S within the angle of view. The amount of driving of the photographing camera 3 to fit the three-dimensional subject area of ​​the subject S within the angle of view can be calculated using a general method, such as the method described in Japanese Patent Application Laid-Open No. 2007-1343845. The drive amount calculation means 120 outputs the calculated drive amount to the photographing camera control means 121 . The photographing camera control means 121 controls the driving of the photographing camera 3 using the driving amount calculated by the driving amount calculation means 120 .

[0085] As described above, in addition to the effects of the three-dimensional object region estimation device 1, the three-dimensional object region estimation device 1B can also capture an image of a subject by using the photographing camera 3 to follow the subject. The three-dimensional object region estimation device 1B can be operated by a program (three-dimensional object region estimation program) that causes a computer to function as each of the above-mentioned units.

[0086] [Operation of 3D object area estimation device] Next, the operation of the three-dimensional object region estimation device 1B according to the second embodiment of the present invention will be described with reference to FIG. 14 (for the configuration, refer to FIG. 13 as appropriate). In FIG. 14, the operations of steps S1 to S5 and S6 are the same as those of the three-dimensional object region estimation device 1 described in FIG. 11, and therefore the description thereof will be omitted. After step S5, in step S5B, the camera control means 12 controls the photographing camera 3 so that the area specified by the three-dimensional subject area estimated in step S5 is included in the angle of view.

[0087] Here, first, the drive amount calculation means 120 calculates the drive amount of the photographing camera 3 for fitting the three-dimensional subject area of ​​the subject S within the angle of view. Then, the photographing camera control means 121 controls the driving of the photographing camera 3 with the driving amount calculated by the driving amount calculation means 120. This allows three-dimensional object region estimation device 1B to photograph the object with photographing camera 3 in real time in conjunction with the movement of the object. Although the embodiments of the present invention have been described above, the present invention is not limited to these embodiments.

[0088] (Variation 1) For example, the three-dimensional object region estimation device 1B may further include a display means 13 as shown in FIG. 15, the display means 13 displays images captured by the photographic cameras 3 on a display device (monitor) not shown. Note that a plurality of display devices (not shown) may be connected in accordance with the number of photographic cameras 3, or a single display device may be configured to display images from a plurality of photographic cameras 3. Here, the display means 13 includes a subject region visualization means 130 .

[0089] The object region visualization means 130 visualizes the three-dimensional object region calculated by the three-dimensional shape approximation means 113 of the three-dimensional object region reconstruction means 11 in the image captured by the imaging camera 3 . The object region visualization means 130 visualizes the image captured by the photographing camera 3 by adding a sphere, which is the three-dimensional object region calculated by the three-dimensional shape approximation means 113 .

[0090] For example, as shown in FIG. 16, for each image I captured by the camera 3, the surface of a sphere that approximates the three-dimensional object region calculated by the three-dimensional shape approximation means 113 is divided into discrete points P SP By displaying it in this way, the three-dimensional subject area can be visualized. In addition, point P SP is the center position X in the world coordinate system in Figure 10. s In a sphere SP specified by and radius r, for example, the center position X s The coordinates of the sphere surface with the azimuth angle and elevation angle each at 30 degree intervals are projected onto the captured image I. Point PSP The points may be displayed larger the closer they are to the captured image I in world coordinates.

[0091] (Variation 2) Also, here, the three-dimensional shape approximation means 113 approximated all vertices of the two-dimensional subject region with a spherical shape. That is, the three-dimensional shape approximation means 113 approximated all vertices with an ellipsoid formed by rotating an ellipse whose major axis and minor axis are equal around the major axis (= minor axis) as an axis. However, the approximate three-dimensional shape may be an ellipsoid formed by rotating an ellipse whose major and minor axes are different. This allows the three-dimensional shape approximation means 113 to more accurately identify the shape of the subject area when the subject is a person.

[0092] (Variation 3) Also, here, the subject region estimation means 101 estimates the region of the entire subject as the two-dimensional subject region. However, the subject area estimation means 101 does not necessarily need to estimate the area of ​​the entire subject, and may estimate a plurality of two-dimensional subject areas for each predetermined target area. For example, the subject area estimation means 101 may estimate the area of ​​the entire subject (subject area A A ), the area of ​​the subject's bust (subject area A B ), the subject's face area (subject area A C ) etc. may be the target area. In this case, the subject area estimation means 101 may estimate the two-dimensional subject area using a machine learning model such as a neural network that has been trained in advance according to the target area to be estimated (subject area, bust-up area, face area, etc.).

[0093] (Variation 4) Here, the two-dimensional subject area estimation means 10 estimates one two-dimensional subject area from the captured image for each camera 2, and the three-dimensional subject area reconstruction means 11 reconstructs the multiple two-dimensional subject areas into three dimensions, thereby estimating the three-dimensional subject area. However, the two-dimensional subject region estimation means 10 may estimate a plurality of different subject regions from the captured image, and the three-dimensional subject region reconstruction means 11 may reconstruct only the designated regions into three dimensions.

[0094] For example, as shown in FIG. 18, the subject area estimation means 101 estimates the entire subject area (subject area A A ) and the subject's bust-up area (subject area A B ), the subject's face area (subject area A C ) and estimate multiple two-dimensional object regions for a single object. In this case, the subject region estimation means 101 includes a plurality of machine learning models such as pre-trained neural networks, and estimates each region in parallel.

[0095] Then, the subject region estimation means 101 outputs the positions of the vertices of the two-dimensional subject region to the three-dimensional subject region reconstruction means 11 together with an identifier for dividing the two-dimensional subject region into the entire subject, a bust-up, a face, etc. Then, when an identifier is externally designated, the projection line calculation means 110 calculates a projection line only for the two-dimensional object region corresponding to the identifier, thereby enabling the three-dimensional object region reconstruction means 11 to estimate the three-dimensional object region only for the region specified by the identifier.

[0096] At this time, by inputting a control script from outside that associates identifiers with time information, the three-dimensional object area reconstruction means 11 can estimate the three-dimensional object area for the target area corresponding to the identifier along the time information of the control script. As a result, the three-dimensional object region estimation devices 1B and 1C can control the photographing camera 3 in accordance with the control script and follow and photograph the object. [Explanation of symbols]

[0097] 1,1B,1C 3D object area estimation device 10 2D object area estimation means 100 Camera Calibration Method 101 Subject area estimation means 11 3D object area reconstruction means 110 Projection line calculation means 111 Nearest point drop foot calculation method 112 Subject area vertex calculation means 113 Three-dimensional shape approximation means 12, 12B Camera control means 120 Drive amount calculation means 121 Camera control means 13 Display means 130 Object area visualization means 2 cameras (fixed cameras) 3. Camera 1000,1000B Subject Area Estimation System

Claims

1. A three-dimensional object region estimation device that estimates a three-dimensional object region of a subject from a captured image, comprising: a two-dimensional object region estimation means for estimating a two-dimensional object region for each of the images captured by the plurality of cameras; a three-dimensional object region reconstructing means for reconstructing a three-dimensional object region of the object from the two-dimensional object region of each camera, The three-dimensional object area reconstruction means includes: a projection line calculation means for converting the center of gravity of the two-dimensional object region into world coordinates for each of the cameras, and calculating a straight line passing through the center of gravity in the world coordinates and the optical center of the camera as a projection line; a nearest point vertical foot calculation means for calculating the position of a vertical foot from the nearest point of the projection line to each projection line for each of the cameras; an object area vertex calculation means for projecting, for each camera, vertices of the two-dimensional object area onto a projection plane that projects the center of gravity of the two-dimensional object area onto the position of the foot drop, and calculating world coordinates of the vertices after projection; a three-dimensional shape approximation means for approximating the positions of the vertices in the world coordinate system after the projection with a three-dimensional shape and calculating shape information of the three-dimensional shape as a three-dimensional subject area; A three-dimensional object region estimation device comprising:

2. a driving amount calculation means for calculating a driving amount of a photographing camera for fitting the three-dimensional object region within an angle of view; a photographing camera control means for driving the photographing camera based on the driving amount; The three-dimensional object region estimation device according to claim 1 , further comprising:

3. 3. The three-dimensional subject area estimation device according to claim 2, wherein the two-dimensional subject area estimation means estimates a plurality of two-dimensional subject areas for each predetermined target area of ​​the subject, and the three-dimensional subject area reconstruction means reconstructs the three-dimensional subject area of ​​the target area from the two-dimensional subject area of ​​the specified target area.

4. The three-dimensional object region estimation device according to claim 3, characterized in that the three-dimensional object region reconstructing means switches the target region for reconstructing the three-dimensional object region based on a control script in which an identifier specifying the target region is associated with time information.

5. 3. The three-dimensional object region estimation device according to claim 2, further comprising an object region visualization means for projecting the three-dimensional shape onto an image captured by the camera and visualizing the three-dimensional object region.

6. 3. The three-dimensional object region estimation device according to claim 2, wherein the three-dimensional shape is a sphere, and the three-dimensional shape approximation means calculates a center position and a radius of the sphere as the three-dimensional object region.

7. A three-dimensional object region estimation program for causing a computer to function as the three-dimensional object region estimation device according to any one of claims 1 to 6.

Images