Information processing device, information processing method, and program

Abstract

To generate a more detailed three dimensional shape model on the basis of a plurality of different three dimensional shape models.SOLUTION: A plurality of three dimensional shape models of the same object are acquired, and a likelihood (first likelihood) in each polygon constituting the acquired three dimensional shape models is calculated. A plurality of three dimensional shape models having different postures are converted into the same posture by using the estimated skeleton data, and scale conversion is performed on the posture-converted three dimensional shape model including the posture-converted skeleton data. Then, the likelihood (second likelihood) of the polygon for the arbitrary region is calculated from the calculated likelihood (first likelihood) of the polygon. Partial regions having the highest polygon likelihood (second likelihood) are extracted from the plurality of three dimensional shape models and combined to generate one three dimensional shape model.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] This disclosure relates to information processing technology for processing three-dimensional shape information. [Background technology]

[0002] In recent years, numerous 3D reconstruction techniques using deep learning have been proposed as methods for generating 3D shape models from 2D images. In this method, regions that cannot be observed from 2D images can be reconstructed using estimation results from a pre-trained model.

[0003] For example, in a pre-trained model that estimates a 3D shape model using a 2D image of a person taken from one side and a corresponding ground truth 3D shape model as training data, it is possible to estimate a statistically plausible 3D shape model from a 2D image showing one side of a person. Non-patent document 1 discloses a method for estimating a 3D shape model from a single input image based on statistical information obtained through pre-training. For example, it is possible to make a plausible estimation from a single frontal image of a person, including the back, which is not included in the input information. However, the level of shape detail is lower for unobserved surfaces estimated using only statistical information obtained through pre-training compared to observed surfaces which can be estimated using observational information.

[0004] Non-patent document 2 discloses a method for obtaining a 3D shape model in a standard pose from a video sequence by optimization. In Non-patent document 2, the pose of the object to be reconstructed in the observation space is normalized, the Radiance Field of the person in the normalized space is optimized over the entire sequence, and a 3D shape model of a single person is obtained from the information of the entire sequence. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019. [Non-Patent Document 2] Chung-Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, and Ira Kemelmacher-Shlizerman. “HumanNeRF: Free-viewpoint rendering of moving people from monocular video” In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022 [Summary of the Invention] [Problems to be Solved by the Invention]

[0006] However, in the technology of Non-Patent Document 2, since one 3D shape model is estimated from the averaged observation information of the entire input sequence, the details of the observation information in each instantaneously observed frame may be lost.

[0007] The technology of the present disclosure aims to generate a 3D shape model with a higher level of detail based on a plurality of different 3D shape models. [Means for Solving the Problems]

[0008] The technology of the present disclosure includes acquisition means for acquiring a plurality of different 3D shape models for one object,For each of the plurality of three-dimensional shape models, a first calculation means calculates the likelihood for the constituent unit data that constitutes the three-dimensional shape model, and based on the likelihood of the constituent unit data, The aforementioned multiple three-dimensional shape models Each 3D shape model contains multiple constituent units. Calculate the likelihood for each subdomain. second The system is characterized by comprising a calculation means and a generation means that generates a single three-dimensional shape model for a single object by combining subregions of the plurality of three-dimensional shape models based on the likelihood. [Effects of the Invention]

[0009] According to this disclosure, it is possible to generate a more detailed 3D shape model based on multiple different 3D shape models. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the hardware configuration of the information processing device according to Embodiment 1. [Figure 2] This figure shows an example of input and output images for a 3D reconstruction DNN. [Figure 3] This is a flowchart illustrating the process for generating a 3D shape model according to Embodiment 1. [Figure 4] This figure shows an example of a skeleton representation. [Figure 5] This figure shows an example of input / output of a 3D reconstructed DNN according to Embodiment 1. [Figure 6] This diagram shows the positional relationship between an object placed in a virtual 3D space and a virtual camera. [Figure 7] This diagram shows the positional relationship between the object and the imaging camera. [Figure 8] This is a diagram illustrating a method for estimating the surface position of a 3D shape model. [Figure 9] This figure schematically illustrates the process of generating a three-dimensional shape model with the same orientation from a two-dimensional image according to Embodiment 1. [Figure 10] This figure schematically illustrates the scale adjustment process according to Embodiment 1. [Figure 11]This is a conceptual diagram illustrating a method for generating a 3D shape model based on multiple 3D shape models according to Embodiment 1. [Figure 12] This figure shows an example of input / output of the attitude change DNN according to Embodiment 1. [Figure 13] This is a diagram illustrating a method for combining scale-adjusted 3D shape models according to Embodiment 1. [Figure 14] This is a flowchart illustrating the process for generating a 3D shape model according to Embodiment 2. [Figure 15] This figure shows an example of a UI screen according to Embodiment 2. [Modes for carrying out the invention]

[0011] Embodiments relating to this disclosure will be described below with reference to the drawings. The following embodiments are not intended to limit the technology of this disclosure, and not all combinations of features described in these embodiments are essential to the solutions of this disclosure. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications and various conditions (usage conditions, usage environment, etc.) of the device to which the technology of this disclosure is applied. In addition, in the following embodiments, the same or similar components are denoted by the same reference numerals, and redundant descriptions are omitted.

[0012] <Embodiment 1> Figures 1(a) to 1(c) show examples of the hardware configuration of the information processing device in this embodiment. The information processing device 10 shown in Figure 1(a) is intended for use in PCs, smartphones, tablet terminals, etc., and consists of an imaging unit 101, a CPU 102, RAM 103, ROM 104, a storage device 105, an operation unit 106, and a display unit 107.

[0013] The imaging unit 101 includes an image sensor and an image generation processing unit, and outputs the captured image to the storage unit 105.

[0014] The CPU 102 executes various processes using computer programs and data stored in the RAM 103 and ROM 104. In this way, the CPU 102 executes or controls various processes that are described as controlling the operation of the entire information processing device 10.

[0015] RAM 103 has areas for storing computer programs and data loaded from ROM 104 and storage device 105, and areas for storing data received from the numerous capture groups mentioned above. Furthermore, RAM 103 has a work area used by CPU 102 when executing various processes. In this way, RAM 103 can provide various areas as appropriate.

[0016] ROM104 stores configuration data for the information processing device 10, computer programs and data related to startup, computer programs and data related to basic operation, and so on.

[0017] The storage device 105 is a hard disk drive or the like. The mass storage device 104 stores the OS (operating system) and computer programs and data that cause the CPU 102 to execute or control various processes described as being performed by the information processing device 10. The data stored in the storage device 105 also includes captured images generated by the imaging unit 101 and data related to the DNN model used for 3D reconstruction. The computer programs and data stored in the mass storage device 105 are loaded into the RAM 103 as appropriate according to the control of the CPU 102 and become the target of processing by the CPU 102.

[0018] The operation unit 106 is a user interface such as a keyboard, mouse, or touch panel, and allows the user to input various instructions to the CPU 102 through its operation.

[0019] The display unit 107 has a screen such as an LCD screen or a touch panel screen, and can display the processing results of the CPU 102 as images, text, etc. The display unit 107 may also be a projection device such as a projector that projects images and text.

[0020] The imaging unit 101, CPU 102, RAM 103, ROM 104, mass storage device 105, operation unit 106, and display unit 107 are all connected to the system bus 108.

[0021] Furthermore, as shown in Figure 1(b), the imaging unit 101 may be independent of the information processing device. In that case, the transmission unit 109 transmits imaging information, including the captured image, to the information processing device 11. Alternatively, the system may consist of only the information processing device 11, or, as shown in Figure 1(c), it may have a data acquisition unit 10A and a database 10B that stores imaging information independent of the information processing device 12. Moreover, the configuration of the information processing device is not limited to the configurations shown in Figures 1(a) to (c), and the operation unit 106 and display unit 107 may be included in a separate information processing device different from the information processing device 10.

[0022] The information processing devices 10, 11, and 12 described above perform the acquisition of the 3D shape model, the calculation of the first likelihood, the calculation of the second likelihood, and the generation of the 3D shape model according to this disclosure. The processing of each component of this disclosure will be described below.

[0023] While it is possible to estimate a 3D shape model from a single input image using a DNN model, the estimated 3D shape model will have different regions with high estimation accuracy depending on the differences in the observational information contained in the input image. In this embodiment, a highly accurate 3D shape model is generated by extracting highly accurate subregions from multiple 3D shape models of the same object estimated from multiple input images with different observational information, and combining them.

[0024] Figure 2 shows examples of input and output images for a 3D reconstruction DNN. The 3D shape model 1303 estimated from input 2D image 1301, which contains information about the contours of a jacket, shirt, and tie, captured from the front, successfully reproduces the contours of the jacket, shirt, and tie. In contrast, the 3D shape model 1304 estimated from input 2D image 1302, which was captured from the back, estimates the area around the chest, where the contours of the jacket and tie should be, as a flat area. Thus, since the 3D shape model estimated using a DNN model is estimated only from the observational information contained in the input image and parameter information acquired through prior training, the reproduction accuracy and estimation precision tend to decrease in areas with less observational information compared to areas with more observational information.

[0025] Figures 3(a) to 3(e) show flowcharts illustrating the process of generating a 3D shape model according to this embodiment. The types of objects for which a 3D shape model can be generated in the technology of this disclosure are not limited, but in this embodiment, a 3D shape model of a person will be used as an example.

[0026] Figure 3(a) shows a flowchart illustrating the general process of generating a three-dimensional shape model according to this embodiment.

[0027] In S310, CPU102 acquires multiple 3D shape models of the same object. Here, the same object can be any object in the same category, such as "person," and is not limited to completely identical objects (i.e., the same person in the case of "person").

[0028] In S320, CPU102 calculates the likelihood (first likelihood) for each polygon constituting the 3D shape model acquired in S310 using the 3D reconstruction DNN. Here, the first likelihood represents the estimated confidence level of each polygon constituting the 3D shape model estimated by the 3D reconstruction DNN.

[0029] In S330, CPU102 calculates the likelihood of a polygon for an arbitrary region (second likelihood) from the likelihood of the polygon (first likelihood) calculated in S320.

[0030] In S340, CPU102 generates a 3D shape model by combining the subregions with the highest likelihood (second likelihood) of the polygons calculated in S330.

[0031] <Processing to acquire multiple 3D shape models of the same object (S310)> The following describes the process of acquiring multiple 3D shape models of the same object in S310. Figure 3(b) shows a flowchart illustrating the process of acquiring 3D shape models of the same object in S310.

[0032] In S311, the CPU 102 acquires a 2D video or multiple still images of the person from whom a 3D shape model is to be generated.

[0033] Next, repeat steps S312 and S313 below for the number of images to be acquired.

[0034] In S312, CPU 102 inputs multiple frames of 2D video or multiple still images acquired in S311 into the 3D reconstruction DNN.

[0035] In S313, the CPU 102 acquires the 3D shape model and skeleton (framework model) data output from the 3D reconstruction DNN and stores them in the memory unit 105.

[0036] Figure 5(a) shows an example of the input and output of the 3D reconstruction DNN used in this embodiment. In this embodiment, as shown in Figure 5(a), a 3D shape model 505, ..., 506 and a skeleton 507, ..., 508 are estimated from multiple input images 501, ..., 502 of the same person by 3D reconstruction DNNs 503, ..., 504. Figure 5(b) is used for explanation. Input image 511 is input to 3D reconstruction DNN 512, and a 3D shape model 513 and a skeleton 514 are output. In addition, the 3D reconstruction DNN architecture in Figures 5(a) and (b) shows an Hourglass Network, but the DNN architecture is not limited to this. Furthermore, as shown in Figure 5(c), it may be implemented using separate DNNs, such as a 3D shape model estimation DNN 515 and a skeleton estimation DNN 516.

[0037] Here, we will explain the training method for the 3D reconstruction DNN. In this embodiment, the 3D reconstruction DNN estimates the 3D voxel map of the 3D shape model and skeleton data indicating joint positions. First, we will explain the training method for estimating the 3D voxel map of the 3D shape model. The training data uses pairs of 3DCG objects that serve as Ground Truth (hereinafter referred to as Ground Truth objects) and 2D virtual viewpoint images rendered from the Ground Truth objects using a virtual camera.

[0038] FIG. 6 shows a Ground Truth object 601 and a virtual camera arranged on a virtual three-dimensional space. The input image is each two-dimensional virtual viewpoint image obtained by rendering the Ground Truth object 601 from each virtual camera viewpoint. The loss used for the optimization of the 3D reconstruction DNN is defined by the difference between the Ground Truth object 601 and the three-dimensional shape model estimated from the 3D reconstruction DNN 512. The data format of the three-dimensional shape model handled in learning is a 3D Occupancy Field in which the inside of the target object region is 1 and the outside of the target object region is 0. Therefore, the Ground Truth object 601 holds a binary correct value, and the voxel value f of the Ground Truth object at the voxel position coordinate X v * (X) is as shown in Equation (1).

[0039] [Number] On the other hand, the 3D reconstruction DNN outputs an estimated voxel value f V (X) as a continuous value between 0 and 1 for each voxel position coordinate X in the 3D Occupancy Field. Hereinafter, the 3D Occupancy Field obtained by mapping the estimated voxel value f V (X) output by the 3D reconstruction DNN is called a 3D voxel map. The estimated voxel value f of the 3D reconstruction DNN at the voxel position coordinate X v (X) is determined by the image feature g(I(x)) obtained from the pixel position coordinate x in the input image I by the image encoder g. The voxel position coordinate X and the pixel position coordinate x are represented as x = π(X) (π: two-dimensional projection function). Let F V (x) = g(I(x)), and let the function for obtaining the voxel value at the voxel position coordinate X from the image feature be f V Then, the estimated voxel value can be expressed as f v (F V (x), z(X)). Here, z(X) represents the depth value seen from the shooting camera coordinates at the voxel position coordinate X. The number of sampled voxels is nV Therefore, the loss L V The result is as shown in equation (2).

[0040]

number

[0041] The 3D shape model is obtained by converting the 3D voxel map, which has continuous values ​​output by the 3D reconstruction DNN, into binary values ​​of 0s and 1s using thresholding, or by converting it into a mesh format using the Marching Cube method or similar.

[0042] Next, we will explain the learning method for skeleton estimation. The training data used for skeleton estimation consists of pairs of input images: a Ground Truth3D skeleton and a CG object rendering that has this Ground Truth3D skeleton. There are multiple skeleton data formats, mainly differing in the number of joints, but this proposal does not impose any restrictions on the skeleton data format.

[0043] Figure 4(a) shows an example of a skeleton representation. Furthermore, the position coordinates of each joint in the skeleton representation shown in Figure 4(a) are shown in Figure 4(b) as J h Let (h=1,...,H). As shown in Figure 4(b), J h Let h=1,...,H. In the skeleton representation shown in Figure 4(a), H=21.

[0044] The 3D reconstruction DNN512 outputs H 3D confidence maps. The coordinate system of these 3D confidence maps is the same as that of the 3D voxel map used during 3D shape model estimation. Each 3D confidence map maps the probability of a corresponding joint existing to a voxel position coordinate, and the voxel position coordinate with the highest probability of joint existence becomes the joint position coordinate. Position coordinate X on the 3D confidence map W The probability that a joint exists is P(X) W ) If so, the estimated joint position coordinate J h It can be expressed as in equation (3).

[0045]

number

[0046]

number

[0047]

number

[0048]

number

[0049]

number

[0050] Furthermore, the 3D shape model and skeleton data acquired in S310 may be obtained from a database 10B that has previously stored multiple 3D shape models and corresponding skeleton data of the object to be reconstructed, which have been estimated in the past.

[0051] <Calculation process for the first likelihood (S320)> The following describes the calculation process for the first likelihood in S320.

[0052] Figure 3(c) shows a flowchart illustrating the first likelihood calculation process of S320. Here, object 701 in Figure 7 is set to x in the global coordinate system. WWe consider estimating a 3D shape model from images captured by the imaging camera 702 located on the axis. In this case, the x coordinate system of the imaging camera 702 is c′ ,y c′ Direction (y in the global coordinate system) W ,z W It is easy to accurately estimate the position of the surface of a 3D shape model whose normal consists only of the (direction) component from the information of the captured image. In contrast, the z in the depth direction in the camera coordinate system of the imaging camera 702 c′ Direction (x in global coordinates) W It is difficult to accurately estimate the surface position of a 3D shape model that also has a directional component. Conversely, when estimating a 3D shape model from an image captured by the imaging camera 703, the direction in which accurate estimation of the surface position is easy is the x-axis in global coordinates. W z W Direction, and directions that are difficult to estimate with high accuracy, are represented by the y coordinate system in global coordinates. W This determines the direction. In this way, the region where high-precision estimation by 3D reconstruction DNN is easy changes depending on the positional relationship between the object and the imaging camera.

[0053] The 3D reconstruction DNN outputs a continuous value between 0 and 1 representing the probability that each voxel is located inside the object region. The 3D reconstruction DNN outputs a value close to 1 if it can estimate with high accuracy that the target voxel is located inside the object region, and a value close to 0 if it can estimate with high accuracy that it is located outside the object region. On the other hand, if it is difficult to estimate with high accuracy whether the target voxel is located inside or outside the object region, the 3D reconstruction DNN outputs a value close to the median value of 0.5.

[0054] Figure 8(b) shows a cross-sectional view of an example where the Marching Cube method is applied to an estimated 3D voxel map assuming that the surface position of a 3D shape model is estimated with high accuracy (high likelihood). Here, the threshold in the Marching Cube method is set to 0.5, and it is assumed that the surface exists in the region with a voxel value of 0.5. Figure 8(c) shows a cross-sectional view of an example where the Marching Cube method is applied to an estimated 3D voxel map assuming that the surface position is estimated with high accuracy (low likelihood). The larger the difference in estimated voxel values ​​between adjacent voxels across the surface, the higher the likelihood, and the smaller the difference, the lower the likelihood.

[0055] In S321, CPU102 calculates the normal vector of each polygon. The normal vector of a polygon composed of three points A, B, and C, as shown in Figure 8(a), can be calculated from the cross product of vector AB and vector AC.

[0056] In S322, CPU102 extracts adjacent voxels using the normal direction calculated in S321.

[0057] In S323, CPU102 calculates the likelihood for a polygon based on the difference in voxel values ​​between a voxel touching the polygon and a voxel adjacent to that voxel. As shown in Figure 8(b), when accurate estimation of the surface position is easy, the difference in estimated values ​​between the two voxels is large, and as shown in Figure 8(c), when accurate estimation of the surface position is difficult, the difference in estimated values ​​between the two voxels is small. The likelihood for a polygon is calculated from the difference in the estimated values ​​of the two voxels. Voxels adjacent to the polygon M in the internal and external directions are V M ,V M′ Let each estimate be

[0058]

number

[0059]

number

[0060] In S331, CPU102 uses the estimated skeleton data to convert multiple 3D shape models with different poses into the same pose. Here, we will explain assuming conversion to a Canonical T-pose. The point coordinates on the 3D shape model before pose conversion are s o (∈S o ), the point coordinates after the attitude change are s T (∈S T ) If so, Canonical T-pose S T Pose S o The process of converting to can be expressed as shown in equation (7).

[0061]

number

[0062]

number

[0063]

number

[0064]

number

[0065]

number

[0066]

number

[0067]

number

[0068] In S332, the CPU 102 performs a scale transformation on the pose-transformed 3D shape model, which includes multiple skeleton data obtained in S331. As shown in Figure 9(b), the pose-transformed 3D shape model 915 has a narrower face and torso width compared to the pose-transformed 3D shape model 914, while the pose-transformed 3D shape model 916 has a wider torso and longer arms compared to the pose-transformed 3D shape model 914. Thus, even when the same object representing the same person is reconstructed and transformed into the same pose, the scale of the pose-transformed 3D shape model, such as height and thickness, may not match, and the 3D shape models may not correspond in the global coordinate system. This is because, as explained using Figure 7, it is difficult for 3D reconstruction DNNs to accurately estimate the depth in the camera coordinate system of the imaging camera, resulting in variations in the accuracy of the 3D shape model. Therefore, in S332, the scale of the 3D shape model is adjusted using the skeleton data corresponding to each 3D shape model, making it easier to establish region-specific correspondences between multiple pose-transformed 3D shape models.

[0069] Therefore, in S332, first, the most reliable skeleton vector among the skeleton data after attitude transformation is determined as the reference scale to be used as the adjusted scale, as follows: An object IDi is set for each 3D shape model, and the skeleton vector after attitude transformation is

[0070]

number

[0071]

number

[0072]

number

[0073]

number

[0074] In S332, the most reliable skeleton vector determined in this way is used.

[0075]

number

[0076]

number

[0077]

number

[0078]

number

[0079]

number

[0080]

number

[0081]

number

[0082]

number

[0083] In S334, CPU102 calculates the likelihood (second likelihood) for each sub-region. The first likelihood of the polygon calculated in S323 is used to calculate the likelihood for each sub-region. If the number of divided grids is G, the likelihood C for each grid g is calculated. g N is the number of polygons present within grid g. g Thus, we get equation (12).

[0084]

number

[0085]

number

[0086] <Process for generating a 3D shape model by combining multiple 3D shape models (S340)> The following describes the process of generating a 3D shape model by combining the most likely subregions in S340. A flowchart is shown in Figure 3(e). For the sake of explanation, we consider combining the most likely subregions 1005, 1006, 1007, and 1008 of the scaled 3D shape models 1001, 1002, 1003, and 1004 shown in Figure 13(a).

[0087] In S341, CPU102 extracts the model with the highest polygon likelihood from among the multiple 3D shape models acquired in S310 for each grid.

[0088] In S342, CPU 102 converts the grid polygons extracted in S341 into voxels and combines them to generate a 3D voxel map of a single 3D shape model. When 3D shape models extracted from different 3D shape models on a grid basis are combined, a shift occurs in the polygons at the grid boundaries, as shown in Figure 13(b). If a 3D shape model is generated by combining polygons that have shifted at these grid boundaries, holes will appear in the 3D shape model. Therefore, for each polygon, the voxel values ​​are stored in the voxels adjacent to the polygon in the direction of the polygon normal used when calculating the estimated voxel value of the voxel touching the polygon and the likelihood of the polygon. Among the voxels near the polygon, voxels inside the object region are stored with a voxel value of 1, and voxels outside the object region are stored with a voxel value of 0.

[0089] In S343, CPU102 converts the 3D voxel map generated in S342 into a mesh format using the Marching Cube method or similar.

[0090] As described above, in this embodiment, a highly accurate 3D shape model can be generated by combining the accurate parts of multiple 3D shape models, each independently estimated from multiple frames of a moving image or multiple still images obtained by imaging the same object.

[0091] <Embodiment 2> This embodiment is a variation of Embodiment 1. The system configuration diagram in this embodiment is the same as that of Embodiment 1. In this embodiment, any combination of 3D shape models can be generated by applying any 3D shape model to any sub-region based on the user's preference from among multiple 3D shape models through user GUI operations. In Embodiment 2, user GUI operations are performed via the operation unit 106 and the display device 107. The hardware for operating and displaying the assumed GUI can be a tablet terminal, smartphone, PC, etc., and any device that connects to the information processing device 10 and functions as the operation unit 106 and the display device 107 is acceptable.

[0092] Figure 14(a) shows a flowchart illustrating the process of generating a 3D shape model in this embodiment. Steps S1410, S1420, S1430, and S1440 are the same as steps S310, S320, S331, and S332 in the flowchart of Embodiment 1, so their explanation is omitted here.

[0093] In S1450, the CPU 102 generates a GUI that displays a representative 3D shape model and displays it on the display unit 107. The representative 3D shape model may be one that was generated by dividing a 3D shape model, which was estimated independently from different observation information as described in Embodiment 1, into regions based on a grid of a predetermined size, and combining the subregions with the highest likelihood of polygons in each grid. The representative 3D shape model is not limited to this; it may also be a suitable single 3D shape model, or one generated from multiple frames in a moving image, as disclosed in Non-Patent Document 1.

[0094] Figure 15 shows an example of a GUI screen displayed on a tablet terminal according to this embodiment. Figure 15(a) shows a representative 3D shape model 1501 displayed on the screen of the tablet terminal. In this tablet terminal, user operations on the displayed representative 3D shape model can be received on a screen that serves as both an operation unit 106 and a display unit 107. For example, as shown in Figure 15(b), when the user touches the representative 3D shape model displayed on the tablet terminal and slides it in the direction they want to see, the representative 3D shape model rotates. This makes it possible to display the representative 3D shape model as it appears from the direction the user wants to see it.

[0095] In S1460, the CPU 102 generates a highly accurate 3D shape model based on user input from GUI operations received by the operation unit 106. Figure 14(b) shows a flowchart illustrating the 3D shape model generation process in S1460.

[0096] In S1461, the CPU 102 accepts the selection of a region of the 3D shape model based on the user input of the GUI operation received by the operation unit 106. As shown in Figure 15(c), the user can enlarge the display by tapping the area they want to modify. Then, as shown in Figure 15(d), when the user drags to select the region they want to modify on the enlarged 3D shape model 1502, a grid 1503 is generated, and the polygons within the generated grid become the selected region.

[0097] In S1462, CPU 102 calculates the likelihood of the sub-region representing the selected area for the 3D shape model acquired in S1410 according to equation (12) and scores it. The score may be the likelihood itself or a processed version of the likelihood.

[0098] In S1463, the CPU 102 generates a GUI that displays the calculated likelihood scoring results and displays it on the display unit 107. Here, as shown in Figure 15(e), the top three 3D shape models are displayed as candidates. If the user does not like these candidate 3D shape models, they can press the "Change" button to switch to the next three 3D shape models with the highest scores.

[0099] In S1464, the CPU 102 accepts the user's selection of one 3D shape model from a list of candidate 3D shape models based on the user input of GUI operations received by the operation unit 106.

[0100] In S1465, the CPU 102 replaces the selection area of ​​the representative 3D shape model with the selection area of ​​the 3D shape model selected in S1464, generates a GUI that displays the 3D shape model reflecting the user's selection, and displays it on the display unit 107. The method of combining the 3D shape models at this time is the same as in S341 and S342 in Embodiment 1. As shown in Figure 15(f), if the user selects one of the candidate 3D shape models 1504, a 3D shape model 1505 is generated with the selection area of ​​3D shape model 1504. When "End" is pressed, the 3D shape model generation process is terminated.

[0101] As described above, by incorporating a GUI, users can combine highly accurate sub-regions of the 3D shape model while confirming and selecting them, making it easier to generate the 3D shape model desired by the user. Furthermore, it becomes easier for users to set the granularity of the sub-regions to be modified as desired.

[0102] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0103] This disclosure includes the following configurations and methods: [Configuration 1] A means for obtaining multiple 3D shape models that are different from each other for a single object. A calculation means for calculating the likelihood of each subregion of the plurality of three-dimensional shape models, A generation means for generating a single three-dimensional shape model for a single object by combining subregions of the plurality of three-dimensional shape models based on the likelihood, An information processing device characterized by having the following features. [Configuration 2] The acquisition means is a plurality of three-dimensional shape models generated by a trained model that generates a three-dimensional shape model from an image obtained by imaging one object. The information processing device according to configuration 1, characterized by the above. [Configuration 3] The calculation means calculates the likelihood for the polygons of the three-dimensional shape model and calculates the likelihood for each sub-region based on the likelihood for the polygons. An information processing apparatus according to configuration 1 or 2, characterized by the above. [Structure 4] The acquisition means acquires the plurality of 3D shape models as voxel maps, The calculation means calculates the likelihood for the polygon based on the voxel values ​​in the voxel map of the voxels tangent to the polygon. The information processing apparatus according to configuration 3, characterized by the features described herein. [Composition 5] The calculation means calculates the likelihood for the polygon from the difference in voxel values ​​adjacent to the normal direction of the polygon. The information processing apparatus according to configuration 4, characterized by the features described above. [Composition 6] The generation means generates the one three-dimensional shape model by combining voxel maps corresponding to the subregions. The information processing apparatus according to configuration 4 or 5, characterized by the above. [Composition 7] The acquisition means acquires a skeleton corresponding to the three-dimensional shape model, The generation means performs a posture transformation on the plurality of three-dimensional shape models based on the skeleton so that they have the same posture before combining the partial regions. An information processing device according to any one of configurations 1 to 6. [Structure 8] The generation means, before combining the sub-regions, scales the plurality of three-dimensional shape models based on the skeleton so that they are the same size. The information processing apparatus according to configuration 7, characterized by the features described above. [Composition 9] Display means for displaying the three-dimensional shape model, A means of receiving user input, Furthermore, The generation means uses the selected subregion from one of the plurality of three-dimensional shape models selected based on the user input for the subregion selected based on the user input. An information processing device according to any one of configurations 1 to 8. [Configuration 10] The receiving means accepts the selection of a subregion of the 3D shape model, The generation means is characterized by changing a portion region selected by the user to the same portion region of another 3D shape model. The information processing device described in configuration 9. [Composition 11] The generating means is Referencing the likelihood for the selected subregion, Based on the likelihood for the aforementioned subregion, the three-dimensional shape model of the selected subregion of the plurality of three-dimensional shape models is selected and displayed on the display means. The receiving means accepts the user's selection of the 3D shape model of the relevant sub-region of the displayed plurality of 3D shape models. The information processing apparatus according to configuration 10, characterized by the above. [Composition 12] The steps include obtaining multiple 3D shape models that are different from each other for a single object, and The steps include calculating the likelihood for each subregion of the plurality of three-dimensional shape models, The steps include: generating a single three-dimensional shape model for a single object by combining subregions of the plurality of three-dimensional shape models based on the likelihood; A control method for an information processing device, characterized by having the following features. [Composition 13] A program for causing a computer to function as an information processing device as described in any one of items 1 to 11.

Claims

1. A means for acquiring multiple different 3D shape models for a single object, A first calculation means for calculating the likelihood for each of the aforementioned three-dimensional shape models with respect to the constituent unit data that constitutes the three-dimensional shape model, A second calculation means calculates the likelihood for each sub-region containing multiple constituent units for each of the multiple three-dimensional shape models based on the likelihood of the constituent unit data, A generation means for generating a single three-dimensional shape model for a single object by combining subregions of the plurality of three-dimensional shape models based on the likelihood, An information processing device characterized by having the following features.

2. The plurality of three-dimensional shape models are each generated by a trained model that generates a three-dimensional shape model from a single image, based on a plurality of captured images obtained by imaging the single object. The information processing apparatus according to feature 1.

3. The first calculation means calculates the likelihood for the polygons of the three-dimensional shape model as the likelihood for the constituent unit data, The second calculation means calculates the likelihood for each sub-region based on the likelihood for the polygon. The information processing apparatus according to feature 1.

4. The acquisition means acquires the plurality of three-dimensional shape models as voxel maps, The first calculation means calculates the likelihood for the polygon based on the voxel values ​​in the voxel map of the voxels tangent to the polygon. The information processing apparatus according to claim 3.

5. The first calculation means calculates the likelihood for the polygon from the difference in adjacent voxel values ​​in the normal direction of the polygon. The information processing apparatus according to feature 4.

6. The generation means generates the one three-dimensional shape model by combining voxel maps corresponding to the subregions. The information processing apparatus according to feature 4.

7. The acquisition means acquires a skeleton corresponding to the three-dimensional shape model, The generation means performs a posture transformation on the plurality of three-dimensional shape models based on the skeleton so that they have the same posture before combining the partial regions. The information processing apparatus according to feature 1.

8. The generation means, before combining the sub-regions, scales the plurality of three-dimensional shape models based on the skeleton so that they are the same size. The information processing apparatus according to feature 7.

9. A display means for displaying the three-dimensional shape model, A means of receiving user input, Furthermore, The generation means uses the selected subregion from one of the plurality of three-dimensional shape models selected based on the user input for the subregion selected based on the user input. The information processing apparatus according to feature 1.

10. The receiving means receives the selection of a partial region of the three-dimensional shape model, The generation means is characterized by changing a portion region selected by the user to the same portion region of another three-dimensional shape model. The information processing apparatus according to claim 9.

11. The generating means is Referencing the likelihood for the selected subregion, Based on the likelihood for the aforementioned subregion, the three-dimensional shape model of the selected subregion of the plurality of three-dimensional shape models is selected and displayed on the display means. The receiving means accepts the user's selection of the 3D shape model of the relevant sub-region of the displayed plurality of 3D shape models. The information processing apparatus according to feature 10.

12. The steps include obtaining multiple distinct 3D shape models for a single object, A first calculation step for each of the aforementioned three-dimensional shape models, which calculates the likelihood for the constituent unit data that constitutes the three-dimensional shape model, A second calculation step involves calculating the likelihood for each subregion containing multiple constituent units for each of the multiple three-dimensional shape models, based on the likelihood of the constituent unit data, The steps include: generating a single three-dimensional shape model for a single object by combining subregions of the plurality of three-dimensional shape models based on the likelihood; A control method for an information processing device, characterized by having the following features.

13. A program for causing a computer to function as an information processing device according to any one of claims 1 to 11.

Images