Generation device, generation method, and program

Abstract

To restrain processing loads and generate a highly-accurate three-dimensional model.SOLUTION: A generation device: obtains a three-dimensional model of an object generated based on images captured with a plurality of photographing devices; obtains posture information of the object generated based on the images captured with the plurality of photographing devices; identifies an area, which is a partial area of the obtained three-dimensional model, on which compensation processing is performed, based on the obtained posture information; and performs the compensation processing on the identified area.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to image processing techniques. [Background technology]

[0002] Recently, a technology that generates an image (virtual viewpoint image) from an arbitrary virtual camera (virtual viewpoint) using the multiple images obtained by installing multiple cameras in different positions and synchronously shooting from multiple viewpoints has been attracting attention. A virtual viewpoint image can be generated by collecting images captured by multiple cameras in an image processing unit such as a server, and performing processing such as generating a 3D model and rendering in the image processing unit.

[0003] In the volume intersection method, which is one of the methods used in generating 3D models, the model is generated based on a mask image that represents the 2D silhouette of the target object, so the accuracy of generating a 3D model of a recess is reduced. Non-Patent Document 1 describes that correction processing is performed using color information for all voxels that make up the 3D model obtained by the volume intersection method. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] S. Seitz and C. Dyer, “Photorealistic Scene Reconstruction by Voxel Coloring”, International Journal of Computer Vision, Vol.35, No.2, pp.151-173, 1999 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technique of Non-Patent Document 1, since correction processing is performed on all voxels that configure a three-dimensional model, the processing load increases as the number of voxels and the number of target objects increase.

[0006] The present disclosure has been made in consideration of the above problems, and has an object to reduce the processing load and generate a highly accurate 3D model. [Means for solving the problem]

[0007] As one means for achieving the above object, the generating device of the present disclosure has the following configuration: a first acquiring means for acquiring a 3D model of an object generated based on images acquired by imaging with a plurality of imaging devices; A second acquisition means for acquiring orientation information of the object generated based on images acquired by imaging with a plurality of imaging devices; The above Object Posture information information on a portion of the object where a recess is likely to occur; A specification means for specifying a region to be subjected to correction processing, which is a part of a region of the three-dimensional model acquired by the first acquisition means, based on the above-mentioned. a correction unit that performs a correction process on the area identified by the identification unit; has. Effect of the Invention

[0008] According to the present disclosure, it is possible to reduce the processing load and generate a highly accurate three-dimensional model. [Brief description of the drawings]

[0009] [Figure 1] 2 shows an example of the configuration of a virtual viewpoint image processing system. [Diagram 2] 2 shows an example of camera arrangement in a virtual viewpoint image processing system. [Diagram 3] 3A to 3C are diagrams illustrating examples of a captured image, a foreground mask image, and a foreground texture image. [Figure 4] 1 is a diagram illustrating an example of a functional configuration of a model generating device according to a first embodiment. [Diagram 5]FIG. 13 is a diagram showing an example of an estimated pose. [Figure 6] 4 is a diagram illustrating an example of a data configuration of posture information. [Figure 7] 3 shows an example of a precision improvement target region according to the first embodiment. [Figure 8] FIG. 2 is a diagram illustrating an example of a hardware configuration of a model generating device according to the first embodiment. [Figure 9] 4 is a flowchart of a process executed by the model generation device according to the first embodiment. [Figure 10] FIG. 11 is a diagram illustrating an example of a functional configuration of a model generating device according to a second embodiment. [Figure 11] 4 is a data configuration example of posture pattern information. [Figure 12] 13 shows an example of a precision improvement target region according to the second embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of a functional configuration of a model generating device according to a third embodiment. [Figure 14] 13 shows an example of a precision improvement target region according to the third embodiment. [Figure 15] FIG. 1 is a diagram illustrating the basic principle of volume intersection. [Figure 16] FIG. 2 is a diagram for explaining a voxel representation of a target object. [Figure 17] FIG. 13 is a diagram showing an example of generating a 3D model for a target object including a recess. [Figure 18] 11A to 11C are diagrams illustrating an example of generating a 3D model for a target object including a recess using color information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [3D shape data using volume intersection method] The generation of three-dimensional shape data (three-dimensional model) by the conventional shape from silhouette method will be described with reference to Figs. 15 and 16. Fig. 15 is a diagram showing the basic principle of the shape from silhouette method. As shown in Fig. 15(a), a mask image 1503 representing a two-dimensional silhouette of the target object 1504 is obtained by binarizing the image based on the difference in color and brightness between the foreground and background on an imaging plane 1502 obtained by a camera 1501 capturing an image of the target object 1504. Here, the foreground means the target object for which a three-dimensional model is generated, and the background means an area other than the foreground. In addition, a frustum extending into three-dimensional space from the projection center of the camera 1501 through each point on the contour of the mask image 1503 is called the "view volume" of the target by the camera 1501 (i.e., the corresponding camera) (view volume 1505 in Fig. 15(a)).

[0012] Fig. 15(b) shows that a 3D model of a target object 1504 is obtained by determining a common area of ​​the view volumes of multiple cameras (cameras 1501a to 1501c in Fig. 15(b)), that is, a view volume intersection 1506. Here, the 3D model is represented by a collection of tiny cubes called voxels.

[0013] Fig. 16 shows a diagram for explaining the voxel representation of a target object (foreground). Fig. 16(a) shows a voxel 1601, Fig. 16(b) shows a voxel set 1602 representing a three-dimensional region (target space) for generating a three-dimensional model, Fig. 16(c) shows a target object 1603, and Fig. 16(d) shows a voxel set 1604 of the target object 1603 generated by the visual volume intersection method. For example, a color corresponding to each voxel is assigned from a captured image to each voxel constituting the voxel set of the target object (foreground) obtained in this way, and a virtual viewpoint image can be generated by projecting the voxels to an arbitrary viewpoint.

[0014] In the conventional 3D model using the volume intersection method, a model is generated based on a mask image that is a 2D silhouette of a target object, and therefore it is theoretically impossible to generate a 3D model of a recess. FIG. 17 shows an example of 3D model generation by the volume intersection method for a target object 1702 including a recess, using cameras 1701a to 1701g. As shown in FIG. 17, a recess 1703 is included in the volume of the mask image generated by any of cameras 1701a to 1701g. Therefore, the recess 1703 cannot be cut, and the shape of the target object 1702 cannot be correctly restored (generated).

[0015] Non-Patent Document 1 discloses a method for performing high-precision processing using color information on a 3D model obtained by a volume intersection method using a mask image in order to improve the accuracy of the model shape of a recess. In this method, it is assumed that when a voxel on the surface of a target object is projected onto multiple cameras that can see the voxel, the pixel values ​​of the multiple cameras at the projection destination are the same, and each voxel is projected onto a camera that can see the voxel, and multiple pixel values ​​of the projection destination are obtained. Then, voxels with small variance in pixel values ​​are left and voxels with large variance are deleted to generate a 3D model.

[0016] FIG. 18 shows an example of 3D model generation by the volume intersection method for the target object 1702 including the recess 1703 using color information by the cameras 1701c to 1701e shown in FIG. 17. For the voxel V1 on the surface of the target object 1702 obtained by the volume intersection method, the pixels of the voxel V1 seen from the cameras 1701c to 1701e are the same foreground surface for all cameras, so the variance is small. Therefore, the voxel V1 is not deleted. On the other hand, for the voxel V2 of the recess 1703, the pixels of the voxel V2 seen from the cameras 1701c to 1701e are in different positions from the surfaces T1, T2, and T3 of the target object 1702 in the line of sight of each camera, so the variance of the pixel value is large. Therefore, the voxel V2 is deleted. In this way, a high-quality 3D model is generated by not considering voxels seen from multiple cameras as being in the foreground and deleting (cutting) them, which have a large variance in pixel values.

[0017] It is preferable to generate such high-quality, highly accurate 3D models for all target objects, but if there are many target objects, the processing load may become enormous. In the following embodiment, a procedure for improving the accuracy of a 3D model without increasing the processing load will be described.

[0018] [First embodiment: High-precision processing based on body part] In the first embodiment, a procedure for generating a high-precision 3D model with reduced processing load is described by performing high-precision processing (correction processing) only on an area of ​​a simplified 3D model that is determined based on the part of a target object.

[0019] (System Configuration) 1 is a block diagram showing an example of the configuration of an image processing system 1 according to this embodiment. The image processing system 1 includes a camera array 10 consisting of a plurality of cameras (imaging devices) 10a-10r, a foreground extraction device group 11 consisting of a plurality of foreground extraction devices 11a-11r, a control device 12, a model generation device 13, and a rendering device 14.

[0020] The camera array 10 is composed of a plurality of cameras 10a to 10r, which capture images of a subject from various angles to obtain captured images and output them to the foreground extraction device group 11. FIG. 2 shows an example of the arrangement of cameras in this embodiment. As shown in FIG. 2, in this embodiment, the camera array 10 is arranged around a shooting area 20 shown as a rectangular parallelepiped, and all the cameras 10a to 10r synchronize the time of the shooting area from various angles to capture, for example, people 30 and 31 who are subjects. In this embodiment, the people 30 and 31 existing inside the shooting area 20 are three-dimensional objects (target objects) to be displayed in a virtual viewpoint image, and are the foreground. The background refers to the area other than the people 30 and 31.

[0021] The foreground extraction device group 11 is composed of foreground extraction devices 11a to 11r that extract foreground regions from captured images output from multiple cameras 10a to 10r, and extract (generate) a foreground mask image and a foreground texture image from the captured images. The foreground mask image and the foreground texture image will be described later with reference to FIG. 3.

[0022] The control device 12 calculates camera parameters indicating the positions and orientations of the cameras 10a to 10r from the captured images obtained by the camera array 10 capturing images in a time-synchronized manner, and outputs the camera parameters to the model generating device 13 and the rendering device 14. The camera parameters are parameters that are composed of external parameters and internal parameters. The external parameters are parameters that are composed of a rotation matrix and a translation matrix and indicate the position and orientation of the camera, and the internal parameters are parameters that include the focal length and optical center of the camera and indicate the angle of view of the camera and the size of the image sensor. The process of calculating the camera parameters is called calibration. The calibration is realized, for example, by using the correspondence between points in a three-dimensional world coordinate system acquired using multiple images capturing a specific pattern such as a checkerboard and points on a two-dimensional system that correspond to the points.

[0023] The model generating device 13 acquires camera parameters from the control device 12, and acquires a foreground mask image and a foreground texture image from the foreground extraction device group 11. Then, the model generating device 13 generates a 3D model of the foreground (simple 3D model) represented by a voxel set by a volume intersection method using the foreground mask image. Furthermore, the model generating device 13 uses the foreground texture image to remove unnecessary voxel sets (concave) from the simple 3D model based on color information, and generates a corrected 3D model of the foreground (high-precision 3D model). In this specification, the process of correcting a predetermined area in the 3D model (removing voxel sets) based on the color information is referred to as a high-precision process. The model generating device 13 outputs the high-precision 3D model together with the foreground texture image to the rendering device 14.

[0024] The rendering device 14 obtains the high-precision three-dimensional model and the foreground texture image from the model generating device 13, and obtains the camera parameters from the control device 12. The rendering device 14 determines the positional relationship between the foreground texture image and the high-precision three-dimensional model based on the camera parameters, and colors the three-dimensional model based on the color of the pixel of the foreground texture image corresponding to each voxel of the three-dimensional model. By such processing, the three-dimensional space is reconstructed, and an image viewed from an arbitrary viewpoint (virtual viewpoint image) is generated. The virtual viewpoint image may be a moving image or a still image.

[0025] In this embodiment, the foreground extraction devices 11a to 11r and the model generation device 13 are connected in a star topology, but they may be connected in a ring topology using a daisy chain connection, a bus topology, or the like.

[0026] (Example of foreground mask image and foreground texture image) FIG. 3 shows examples of a captured image, a foreground mask image, and a foreground texture image. Specifically, FIG. 3(a) shows a captured image 301 including a person 31 (i.e., foreground) captured by the camera 10g arranged as shown in FIG. 2. FIG. 3(b) shows a foreground mask image 302 generated by the foreground extraction device 11g connected to the camera 10g. The foreground mask image 302 is a silhouette image generated by, for example, determining an area of ​​a difference in brightness or color between a background image previously stored in the foreground extraction device 11g and the captured image 301 as a foreground area, and indicating the foreground area with a binary value of 1 (white) and the background area with a binary value of 0 (black). FIG. 3(c) shows a foreground texture image (foreground image) 303 generated by the foreground extraction device 11g, which is a color image obtained by cutting out the foreground area from the captured image 301.

[0027] (Configuration of the model generation device) FIG. 4 shows an example of the functional configuration of the model generating device 13 according to this embodiment. The receiving unit 101 receives and acquires the camera parameters of all the cameras (cameras 10a to 10r) output from the control device 12, and the foreground texture image and the foreground mask image output from the foreground extraction device group 11, and outputs them to the simple model generating unit 102. The simple model setting unit 103 sets a three-dimensional area and a voxel size for generating a three-dimensional model to be generated, and sets (inputs) them to the simple model generating unit 102. The three-dimensional area and the voxel size may be set by a user via the input / output unit 815 (FIG. 8), or may be determined in advance in the system. The simple model generating unit 102 generates a three-dimensional model of the foreground as a simple three-dimensional (3D) model from a plurality of foreground mask images and camera parameters by a volume intersection method with the set three-dimensional area and voxel size. The simple model generating unit 102 outputs the generated simple 3D model and the foreground texture image (and the foreground mask image) to the orientation estimating unit 104 and the precision improving unit 106.

[0028] The posture estimation unit 104 estimates the posture of the person in the foreground based on the foreground texture image (and the simple 3D model and the foreground mask image) to derive posture information. An example of the processing of the posture estimation unit 104 will be described. First, the posture estimation unit 104 estimates the two-dimensional positions of the body parts of the person in the foreground texture image acquired from the simple model generation unit 102 using a posture estimation model that has previously learned the body parts of the person from a large number of still images. Next, the posture estimation unit 104 estimates (calculates) the three-dimensional positions of the body parts of the person (i.e., the posture of the person) by triangulation based on the camera parameters and the two-dimensional positions of the body parts of the person in the captured images of the two cameras. The posture estimation unit 104 derives posture information based on the estimated three-dimensional positions. Note that the method of estimating the two-dimensional positions of the body parts of the person is not limited to the above-mentioned method, and the positions of the body parts may be estimated (specified) from the captured images by attaching specific markers to each body part. Furthermore, the method of estimating the three-dimensional positions of human parts is not limited to the above-mentioned method, and each human part may be estimated (identified) based on distance measurement using a laser or the like.

[0029] An example of posture information will be described with reference to Fig. 5 and Fig. 6. Fig. 5 shows an example of an image of a posture estimated for a person 31, and shows exemplary parts constituting the posture information in Fig. 6. Fig. 6 shows an example of a data configuration of posture information, where the posture information is composed of a part ID 601, a part name 602, and an X coordinate 603, a Y coordinate 604, and a Z coordinate 605 in three-dimensional coordinates (spatial coordinates) with a predetermined position as the origin. As shown in Fig. 5, in this embodiment, the parts constituting the posture information are 14 parts: head (B1), neck (B2), right shoulder (B3), left shoulder (B4), right arm (B5), left arm (B6), right hand (B7), left hand (B8), chest (B9), abdomen (B10), right thigh (B11), left thigh (B12), right foot (B13), and left foot (B14). In this embodiment, as shown in Fig. 6, the posture information is composed of information expressing the center position of each part as a representative point of each part in three-dimensional coordinates, but is not limited to this. Furthermore, the posture information is not limited to information only about the position of each part, and may take into account information such as the rotation of each part and its connection with other parts (for example, such information may be included additionally).

[0030] The target area determination unit 105 determines (specifies) an area to be improved in accuracy in the simple 3D model generated by the simple model generation unit 102 (high accuracy area / area where correction processing is performed) based on the posture information derived by the posture estimation unit 104. In this embodiment, the target area determination unit 105 determines an area that is of low quality in the simple 3D model generated by the volume intersection method, i.e., an area that is likely to become a recess, as the high accuracy area based on the body part. The target area determination unit 105 acquires (sets) in advance information on parts of the skeleton of a human or animal that are likely to become a recess (area where a recess is likely to occur), and determines the high accuracy area based on the information.

[0031] FIG. 7 shows an example of a high-precision region according to this embodiment. As shown in FIG. 7, for example, the neck 701 and two armpits 702 and 703 are likely to be concave in terms of the skeleton of a human or animal, and it is difficult to generate a high-precision model using the volume intersection method. It is assumed that the target region determination unit 105 recognizes in advance that the neck and armpits are likely to be concave. The target region determination unit 105 specifies the neck 701 and two armpits 702 and 703 in the simple 3D model as parts that are likely to be concave. Then, the target region determination unit 105 determines (specifies) a region (voxel group) in the simple 3D model that includes the neck 701 and the two armpits 702 and 703 and whose distance from the part in the posture information (FIG. 6) is equal to or less than a predetermined threshold value, as a high-precision region. That is, the target area determination unit 105 determines, in the simplified 3D model, areas (voxel groups) whose distance from the positions of the neck (B2), both shoulders (right shoulder (B3) and left shoulder (B4)), and both arms (right arm (B5), left arm (B6)) is less than a predetermined threshold as high-precision areas.

[0032] In this embodiment, the voxel group included in a predetermined distance from the part including the part that is likely to become a recess is determined as the high-precision region, but instead of / in addition to this, another high-precision region may be determined. For example, to target the armpits 702, 703 that are likely to become recesses, the region located in the downward direction (negative direction of the Z axis) of both shoulders (right shoulder (B3) and left shoulder (B4)) and both arms (right arm (B5), left arm (B6)) including the armpits 702, 703 may be determined as an alternative / additional high-precision region.

[0033] The high accuracy processing unit 106 refines the model based on color information by limiting the area based on the simple 3D model output from the simple model generating unit 102 and the refinement area (voxel group) output from the target area determining unit 105. That is, the high accuracy processing unit 106 refines the model based on color information as described above for the refinement area in the simple 3D model to generate a high accuracy 3D model. The high accuracy processing unit 106 outputs the high accuracy 3D model generated by the refinement process and the foreground texture image to the rendering device 14.

[0034] 8 shows an example of a hardware configuration of the model generating device 13 according to this embodiment. The CPU 811 is a central processing unit, and controls the entire model generating device 13 by executing a program stored in a ROM (Read Only Memory) 812 or a RAM (Random Access Memory) 813. For example, the CPU 811 loads a predetermined program stored in a storage device 814 into the RAM 813, and executes the program loaded into the RAM 813 to realize, for example, each functional unit of the model generating device 13. The ROM 812 is a read-only non-volatile memory. The RAM 813 is a memory that can be read and written at any time. A DRAM (Dynamic Random Access Memory) can be used as the RAM 813.

[0035] The storage device 814 is a large-capacity storage device constituted by, for example, a hard disk. The storage device 814 can store foreground images obtained from a plurality of cameras. The input / output unit 815 accepts various user inputs to the model generating device 13. The communication unit 816 controls communication between the model generating device 13 and an external device. In this embodiment, the communication unit 816 is connected to the foreground extraction device group 11 and the rendering device 14 as external devices. The external device and the model generating device 13 can be connected via a network such as a LAN (Local Area Network).

[0036] In addition, the processing of each functional unit in the model generating device 13 (for example, the processing flow described below) is realized by the CPU 811 executing a predetermined program, but this is not limited to this. For example, hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array) for accelerating calculations may be used. That is, each functional unit of the three-dimensional model generating device 13 may be realized by cooperation between software and hardware such as a dedicated IC (Integrated Circuit), or some or all of the functions may be realized only by hardware. Also, a configuration in which a plurality of model generating devices 13 are used to distribute and execute the processing of each functional unit may be used.

[0037] (Processing flow) 9 is a flowchart of processing executed by the model generating device 13 according to this embodiment. In S901, the receiving unit 101 receives camera parameters indicating the positions and orientations of all the cameras 10a to 10r from the control device 12. In S902, the receiving unit 101 receives a foreground mask image representing the foreground region of the images captured by all the cameras 10a to 10r and a foreground texture image (foreground image) obtained by cutting out the foreground region from the foreground extraction device group 11. In S903, the simple model generating unit 102 generates a three-dimensional model of the foreground (simple three-dimensional model) using the three-dimensional region and voxel size set by the simple model setting unit 103 from the camera parameters of all the cameras 10a to 10r and the foreground mask image by a volume intersection method.

[0038] In S904, the posture estimation unit 104 estimates the posture of the person in the foreground using the foreground texture image (and the simple 3D model generated by the simple model generation unit 102, etc.), and derives posture information. In S905, the target area determination unit 105 determines, as the high-precision area, an area (voxel group) whose distance from a part including a part that is likely to become a recess from the posture information derived by the posture estimation unit 104 is equal to or less than a predetermined threshold. In this embodiment, as described with reference to FIG. 7, the neck area 701 and two armpits 702 and 703 are parts that are likely to become recesses, and are targeted for the high-precision processing. The target area determination unit 105 determines, as the high-precision area, an area (voxel group) in the simple 3D model whose distance from the neck (B2) and both shoulders (right shoulder (B3) and left shoulder (B4)), which are body parts in the posture information, including the part, is equal to or less than a predetermined threshold.

[0039] In S906, the high accuracy processing unit 106 uses color information of the foreground texture image to perform high accuracy processing on a group of voxels in the simple 3D model determined by the target area determination unit 105. The high accuracy processing unit 106 outputs the 3D model obtained by the high accuracy processing (high accuracy 3D model) and the foreground texture image to the rendering device 14.

[0040] In this embodiment, the improvement of the accuracy of a three-dimensional model for a person (for example, persons 30 and 31 shown in FIG. 2) has been described, but the target object is not limited to a person. For example, the model generating device 13 may be configured to estimate the positions and postures of parts of an object such as an animal or a car, limit the area in which the accuracy of the three-dimensional model is to be improved based on the estimation result, and perform the accuracy improvement process on the limited area.

[0041] As described above, by determining areas that are likely to result in low-quality 3D models using the volume intersection method based on the parts obtained from the posture information, and then limiting the model to these areas and improving its accuracy, it is possible to generate a high-quality 3D model without increasing the processing load.

[0042] [Second embodiment: High-precision processing based on posture] In the second embodiment, a procedure for generating a high-quality 3D model with reduced processing load by performing high-precision processing only on an area in a simplified 3D model that is determined based on the posture of a target object will be described.

[0043] (Configuration of the model generation device) Fig. 10 shows an example of the functional configuration of a model generating device 13 according to this embodiment. The same components as those in Fig. 4 described in the first embodiment are given the same reference numerals, and the description thereof will be omitted.

[0044] The target area determination unit 1001 determines an area to be improved in accuracy (high accuracy area) in the simple 3D model generated by the simple model generation unit 102, based on the posture information derived by the posture estimation unit 104 and the simple 3D model. In this embodiment, the target area determination unit 1001 estimates the posture of the person based on the posture information, and determines (specifies) the high accuracy area based on the estimated posture of the person. In this embodiment, the posture of the person represents the appearance state of the person, such as standing upright, bending forward, or lying face down.

[0045] In this embodiment, the target area determining unit 1001 is assumed to hold (set) in advance posture pattern information, which is information on various human postures. Fig. 11 shows an example of the data configuration of posture pattern information, where the posture pattern information is composed of a posture ID 1101, a posture name 1102, posture information 1103, a target area 1104, and a target part 1105. The target area 1104 is an area that is previously associated with a corresponding posture and is likely to cause a depression, and a serious illness part is a part that includes the corresponding target area 1104. Note that in Fig. 11, the posture information 1103 shows only an image of the posture, but is assumed to include information that represents a representative point (for example, a center position) of each part in three-dimensional coordinates as in Fig. 6.

[0046] The target region determination unit 1001 matches the foreground posture information derived by the posture estimation unit 104 with posture information 1103 of each posture pattern, and determines (specifies) the posture pattern in which both posture information are closest. For example, the target region determination unit 1001 determines a posture pattern corresponding to the posture information 1103 in which the sum of the distances between each part in the derived posture information and the posture information 1103 is the smallest. Then, the target region determination unit 1001 determines (specifies) a region (voxel group) in the simplified 3D model included in the target region 1104 corresponding to the determined posture pattern as a high-precision region. Specifically, the target region determination unit 1001 determines a region (voxel group) in the simplified 3D model whose distance from the target part 1105 corresponding to the determined posture pattern is equal to or less than a predetermined threshold value as a high-precision region.

[0047] Fig. 12 shows an example of a high-precision region according to this embodiment. Fig. 12(a) shows a case where a person is in a leaning posture. In this case, the target region determination unit 1001 determines a posture pattern of ID1101=P1 from the posture pattern information shown in Fig. 11 based on posture information obtained by the posture estimation unit 104 for the person in that posture. Then, the target region determination unit 1001 determines a region 1201 in the simplified 3D model whose distance from the target part 1105 (chest, abdomen, right shoulder, left shoulder) of ID1101=P1 is equal to or less than a predetermined threshold as a high-precision region.

[0048] 12(b) shows a case where the person is lying face down. In this case, the target region determining unit 1001 determines a posture pattern of ID1101=P3 from the posture pattern information shown in FIG. 11 based on posture information obtained by the posture estimation unit 104 for the person in that posture. Then, the target region determining unit 1001 determines a region 1202 in the simplified 3D model whose distance from the target part 1105 (all parts) of ID1101=P3 is equal to or less than a predetermined threshold as a high-precision region.

[0049] The process flow of the model generating device 13 according to this embodiment is the same as that of the first embodiment shown in Fig. 9, and therefore will not be described. However, in the process of determining the high-precision region in S905, the process according to this embodiment is performed.

[0050] As described above, by using the posture determined from the posture information to determine areas that are likely to result in low-quality 3D models using the volume intersection method, and then by limiting the model to these areas and improving their accuracy, it is possible to generate a high-quality 3D model without increasing the processing load.

[0051] [Third embodiment: High accuracy based on positional relationship / arrangement of target objects] In the third embodiment, a procedure for generating a high-quality 3D model with reduced processing load is described by applying high-precision processing to a limited area determined based on the positional relationship / arrangement of target objects (or objects included in a single target object) in a simplified 3D model.

[0052] (Configuration of the model generation device) Fig. 13 shows an example of the functional configuration of a model generating device 13 according to this embodiment. The same components as those in Fig. 4 described in the first embodiment are given the same reference numerals, and the description thereof will be omitted.

[0053] The target area determination unit 1301 determines an area to be improved in the simple 3D model generated by the simple model generation unit 102 (high precision area) based on the posture information derived by the posture estimation unit 104 and the simple 3D model. In this embodiment, the target area determination unit 1301 determines (specifies) the high precision area based on the positional relationship of multiple target objects. As an example, the target area determination unit 1301 determines a voxel group included in an area where the distance between a part of a person (foreground) included in the posture information and other foreground is equal to or less than a threshold as a concave portion is likely to occur in the voxel group, as the target for improvement in precision.

[0054] Fig. 14 shows an example of a high-precision region according to this embodiment. Fig. 14(a) shows a case where a person 1401 is a first target object, a person 1402 is a second target object, and the left hand of the person 1402 is placed on the head of the person 1401. In this case, the posture estimation unit 104 derives posture information of each of the people 1401 and 1402. Then, the target region determination unit 1301 determines that the distance between the left hand of the person 1402 and the head of the person 1401 is equal to or less than a predetermined threshold (within a predetermined distance) based on the posture information of each of the people 1401 and 1402. Next, the target region determination unit 1301 determines a region (voxel group) 1403 around the left hand of the person 1401 and around the head of the person 1402 as a high-precision target. For example, the target region determining unit 1301 determines a region (voxel group) 1403 in the simplified 3D model whose distance from the left hand of the person 1402 and the distance from the head of the person 1401 are less than or equal to a predetermined threshold value as a high-precision region.

[0055] 14(b) shows a case where a person 1403 is a first target object and an object 1404 is a second target object, and the person 1403 is sitting on the object 1404. In this case, the posture estimation unit 104 derives posture information of the person 1403 and the object 1404. Then, the target region determination unit 1301 determines, based on the posture information of the person 1403, that the distance between the left hand, left and right thighs, and abdomen of the person 1403 and the object 1404 is equal to or less than a predetermined threshold (within a predetermined distance). Next, the target region determination unit 1301 determines voxel groups 1405 and 1406 around the parts of the person 1403 (left hand, left and right thighs, and abdomen) close to the object 1404 as high-precision targets. For example, the target region determining unit 1301 determines regions (voxel groups) 1405 and 1406 in the simplified 3D model whose distances from the left hand, left and right thighs, and abdomen of the person 1403 are equal to or less than a predetermined threshold as high-precision regions.

[0056] The process flow of the model generating device 13 according to this embodiment is the same as that of the first embodiment shown in Fig. 9, and therefore will not be described. However, in the process of determining the high-precision region in S905, the process according to this embodiment is performed.

[0057] As described above, the visual volume intersection method determines areas that are likely to result in low-quality 3D models based on the positional relationships (e.g. distance) of multiple target objects obtained from posture information, and by limiting the model to these areas and processing them to improve accuracy, it becomes possible to generate a high-quality 3D model without increasing the processing load.

[0058] In this way, according to each of the embodiments described above, by limiting the areas of the 3D model obtained by the volume intersection method that are likely to be of low quality to high-precision processing, it is possible to reduce the processing load and generate a high-quality 3D model.

[0059] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0060] The disclosure is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present disclosure. [Explanation of symbols]

[0061] 10 camera array, 102 simple model generation unit, 104 posture estimation unit, 105; 1001; 1301 target region determination unit, 106 high-precision processing unit

Claims

1. a first acquisition means for acquiring a three-dimensional model of an object based on images acquired by imaging using a plurality of imaging devices; a second acquisition means for acquiring orientation information of the object based on images acquired by imaging with a plurality of imaging devices; a specifying means for specifying a region to be corrected, which is a part of the three-dimensional model acquired by the first acquiring means, based on the posture information of the object and information on a portion of the object that is likely to have a recess; a correction unit that performs a correction process on the area identified by the identification unit; A generating device comprising:

2. the second acquisition means acquires, as the posture information, a three-dimensional position of each part constituting the object in a three-dimensional coordinate system having a predetermined position as an origin; The generating device described in claim 1, characterized in that the identification means identifies parts of the three-dimensional model where recesses are likely to occur based on information about parts of the object where recesses are likely to occur, and identifies, as the area to perform the correction processing, an area where a distance from the three-dimensional position of a part of the object including the identified parts where recesses are likely to occur is equal to or less than a predetermined threshold.

3. The generating device according to claim 1 or 2, characterized in that the identification means identifies the area where the correction processing is to be performed based on information about an area in which a depression is likely to occur depending on the posture of the object that has been set in advance and on posture information about the object.

4. The information on the region where a concave portion is likely to occur includes a plurality of body posture patterns and information indicating a region where a concave portion is likely to occur corresponding to each of the plurality of body posture patterns, the second acquisition means acquires, as the posture information, a three-dimensional position of each part constituting the object in a three-dimensional coordinate system having a predetermined position as an origin; The generating device described in claim 3, characterized in that the identification means identifies one posture pattern from the multiple posture patterns based on the three-dimensional position of each part, and identifies in the three-dimensional model an area where the correction processing is performed, the area being a predetermined threshold or less away from the three-dimensional position of a part of the object, including an area where a depression is likely to occur corresponding to the identified posture pattern.

5. The generating device according to claim 1 , wherein the specifying unit specifies the area in which the correction process is to be performed based on a positional relationship between the object and another object.

6. the second acquisition means acquires, as the posture information, a three-dimensional position of each part constituting the object in a three-dimensional coordinate system having a predetermined position as an origin; The generating device according to claim 5, characterized in that the identification means identifies, as an area in which the correction processing is performed, an area whose distance from the other object is within a predetermined distance and whose distance from the three-dimensional position of a part of the object is less than a predetermined threshold.

7. The generating device according to claim 2 , 4 or 6 , wherein the three-dimensional position of the part is a three-dimensional position of a representative point of the part.

8. The generating device according to claim 1 , wherein the object includes a person or an animal.

9. A generating device as described in claim 8, characterized in that the part of the object that is prone to developing depressions is near the neck or armpits of the object.

10. 9. The apparatus of claim 1, wherein the object comprises a car.

11. a first acquisition step of acquiring a three-dimensional model of an object based on images acquired by imaging with a plurality of imaging devices; a second acquisition step of acquiring orientation information of the object based on images acquired by imaging with a plurality of imaging devices; a specifying step of specifying a part of the three-dimensional model acquired in the first acquiring step, the part being subjected to a correction process, based on the posture information of the object and information on a part of the object that is likely to have a recess; a correction step of performing a correction process on the area identified in the identification step; A method for producing the same, comprising:

12. A program for causing a computer to function as the generating device according to any one of claims 1 to 10.

Images