Information processing device, information processing method, and program
The information processing apparatus addresses the issue of discomfort from temporal shape changes in 3D data by aligning and interpolating with reference shapes, resulting in smoother and more comfortable 3D shape data generation.
Patent Information
- Application Number
- JP2023080741
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-16
- Publication Date
- 2024-11-28
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing techniques for generating 3D shape data from temporally continuous images fail to account for temporal changes in the shape of an object, leading to discomfort when viewing the changing 3D shape over time.
An information processing apparatus that acquires and evaluates 3D shape sequence data, selects reference shapes that meet criteria, and generates corrected 3D shape data by aligning and interpolating with these references to minimize temporal discrepancies.
Generates 3D shape data with reduced temporal changes, providing a more comfortable viewing experience by smoothing out shape variations over time.
Smart Images

Figure 2024164975000001_ABST
Abstract
Description
[Technical field]
[0001] This relates to 3D shape data correction technology. [Background technology]
[0002] There is a technique for generating three-dimensional shape data showing the three-dimensional shape of an object based on a plurality of captured image data obtained by capturing images of the object from various positions. There is also a technique for evaluating the accuracy of the three-dimensional shape of the object represented by the three-dimensional shape data. Patent Document 1 discloses a technique for evaluating the accuracy of the three-dimensional shape by projecting the three-dimensional shape onto a two-dimensional image, and further correcting the three-dimensional shape based on the evaluation result, thereby generating accurate three-dimensional shape data. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2003-115042 A Summary of the Invention [Problem to be solved by the invention]
[0004] By generating three-dimensional shape data based on each frame of captured image data obtained by capturing an object as a moving image, three-dimensional shape sequence data including a plurality of three-dimensional shape data that are continuous in time can be generated. The three-dimensional shape sequence data can express a change in the three-dimensional shape of the object over time. When evaluating and generating the three-dimensional shape of an object represented by each three-dimensional shape data included in such three-dimensional sequence data, it is required to reduce the sense of incongruity when appreciating the three-dimensional shape of the object that changes over time. Specifically, in order to reduce such incongruity, it is necessary to consider the three-dimensional shape of the object that changes over time. However, the technology disclosed in Patent Document 1 does not take into consideration the change in the three-dimensional shape over time. Therefore, when the technology disclosed in Patent Document 1 is applied to correct the three-dimensional shape data included in the three-dimensional shape sequence data, a sense of incongruity may occur in the change in the three-dimensional shape of the object represented by the corrected three-dimensional shape sequence data.
[0005] An object of the present disclosure is to generate three-dimensional shape data that reduces the sense of discomfort caused by changes over time in the three-dimensional shape of an object. [Means for solving the problem]
[0006] The information processing device according to the present disclosure includes a shape data acquisition means for acquiring three-dimensional shape sequence data including a plurality of three-dimensional shape data that are consecutive in time; an evaluation acquisition means for acquiring data on evaluation results of the three-dimensional shapes of objects represented by each of the plurality of three-dimensional shape data included in the three-dimensional shape sequence data; and a generation means for generating third three-dimensional shape data to replace second three-dimensional shape data among the plurality of three-dimensional shape data, the evaluation result of which does not satisfy a criterion, based on first three-dimensional shape data among the plurality of three-dimensional shape data, the evaluation result of which satisfies a criterion. Effect of the Invention
[0007] According to the present disclosure, it is possible to generate three-dimensional shape data that reduces the sense of discomfort caused by changes over time in the three-dimensional shape of an object. [Brief description of the drawings]
[0008] [Figure 1] 1 is a block diagram showing an example of a hardware configuration of an information processing device according to a first embodiment. [Diagram 2] 1 is a block diagram showing an example of a functional configuration of an information processing device according to a first embodiment. [Diagram 3] FIG. 4 is a diagram showing an example of three-dimensional shape sequence data according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of evaluation result data according to the first embodiment. [Diagram 5] 4 is a flowchart showing an example of a processing flow in the information processing device according to the first embodiment. [Figure 6] 5 is a flowchart showing an example of a processing flow in a generation unit according to the first embodiment. [Figure 7] 5 is a diagram showing an example of alignment between a reference shape and a shape to be corrected in a generating unit according to the first embodiment. FIG. [Figure 8] 4A to 4C are diagrams for explaining an example of local registration in a generating unit according to the first embodiment. [Figure 9] 11 is a flowchart showing an example of a processing flow in a generation unit according to the second embodiment. [Figure 10] FIG. 11 is a diagram showing an example of a polygon mesh according to the second embodiment. [Figure 11] FIG. 11 is a block diagram showing an example of a functional configuration of an information processing device according to a third embodiment. [Figure 12] FIG. 11 is a diagram showing an example of captured image sequence data included in multi-view image sequence data according to the third embodiment. [Figure 13] 11 is a flowchart showing an example of a processing flow in an information processing device according to a third embodiment. [Figure 14] FIG. 11 is a block diagram showing an example of a functional configuration of an information processing device according to a fourth embodiment. [Figure 15] 13 is a flowchart showing an example of a processing flow in an information processing device according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present disclosure. Note that the same components will be described with the same reference numerals.
[0010] [Embodiment 1] <Hardware configuration of information processing device> FIG. 1 is a block diagram showing an example of a hardware configuration of an information processing device 100 according to the first embodiment. The information processing device 100 includes a CPU 101, a RAM 102, a ROM 103, a serial I / F 104, a VC 105, and a general-purpose I / F 106. The CPU 101 executes an OS (operating system) and various programs stored in the ROM 103 or an external storage device 111, etc., using the RAM 102 as a work memory. At least one of the OS and various programs may be stored in an auxiliary storage device (not shown), such as a hard disk drive, disposed inside the information processing device 100. The CPU 101 controls each piece of hardware shown in FIG. 1 via a system bus 107. Each process shown in a flowchart described later is executed by the CPU 101 after program codes stored in the ROM 103 or the external storage device 111, etc. are expanded in the RAM 102.
[0011] The serial I / F 104 is configured with a serial interface such as a serial ATA, and is connected to an external storage device 111 via a serial bus 108. The external storage device 111 is a storage device configured with an HDD (hard disk drive) or an SSD (solid state drive), etc. The VC 105 is configured with a video card or the like, and is connected to a display 112 via a serial bus 109. The general-purpose I / F 106 is configured with a USB (Universal Serial Bus) interface, for example, and is connected to an input device 113 such as a mouse or a keyboard via a serial bus 110.
[0012] The CPU 101 displays a UI (user interface) provided by a program on a display 112 via a VC 105, and receives input information indicating an instruction from a user obtained via an input device 113. The information processing device 100 is realized, for example, by a desktop PC (personal computer), a notebook PC integrated with a display 112, or a tablet PC. The external storage device 111 may be realized by a storage medium and an external storage drive for accessing the storage medium. Examples of the storage medium include a FD (flexible disk), a CD-ROM, a DVD, a USB memory, an MO, or a flash memory.
[0013] <Functional configuration of information processing device> 2 is a block diagram showing an example of a functional configuration of the information processing device 100 according to the first embodiment. The information processing device 100 has a shape data acquisition unit 201, an evaluation acquisition unit 202, and a correction unit 203 as functional configurations. The correction unit 203 has a generation unit 204 and an output unit 205. Each unit of the information processing device 100 as a functional configuration is realized by the CPU 101 executing a program stored in the ROM 103 using the RAM 102 as a work memory. Note that it is not necessary for all of the processes shown below to be executed by the CPU 101, and the information processing device 100 may be configured so that part or all of the processes are executed by one or more processing circuits other than the CPU 101. Examples of the processing circuits include an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).
[0014] The shape data acquisition unit 201 acquires three-dimensional shape sequence data representing a time-series three-dimensional shape based on input information representing an instruction from a user inputted via the input device 113. The three-dimensional shape sequence data according to the first embodiment is composed of a plurality of three-dimensional shape data, and each of the three-dimensional shape data is associated with a corresponding frame among a plurality of frames successive in time. The three-dimensional shape data according to the first embodiment is assumed to represent the three-dimensional shape of an object by a polygon mesh, and to be composed of a list of spatial coordinates of vertices in each of a plurality of polygons constituting the polygon mesh, and of connection relationships between the vertices. The three-dimensional shape data may include planar coordinates (so-called texture coordinates) of an image corresponding to each vertex constituting the polygon mesh, or color information, etc., for coloring the polygon mesh. The three-dimensional shape sequence data acquired by the shape data acquisition unit 201 is transmitted to the generation unit 204 and the output unit 205.
[0015] 3 is a diagram showing an example of three-dimensional shape sequence data 300 according to the first embodiment. 1 ,t 2 ,t3 ,t 4 ,t 5 Frame f(t 1 ),f(t 2 ),f(t 3 ),f(t 4 ),f(t 5 3 are the appearances of the three-dimensional shapes when viewed from the positive direction of the z-axis. In this way, the three-dimensional shape sequence data 300 is a time-series frame f(t 1 ),f(t 2 ),f(t 3 ),f(t 4 ),f(t 5 . . ), . .
[0016] The evaluation acquisition unit 202 acquires data indicating an evaluation result for the three-dimensional shape sequence data (hereinafter referred to as "evaluation result data") based on input information indicating an instruction from a user inputted via the input device 113. The evaluation result data is information indicating an evaluation result when a predetermined evaluation is performed on each of the time-series three-dimensional shape data constituting the three-dimensional shape sequence data. The predetermined evaluation is, for example, a judgment as to whether or not the accuracy of the three-dimensional shape indicated by each of the three-dimensional shape data constituting the three-dimensional shape sequence data satisfies a predetermined standard, and the evaluation result data lists the judgment results for each of the three-dimensional shape data. The evaluation result data acquired by the evaluation acquisition unit 202 is transmitted to the generation unit 204.
[0017] 4 is a diagram showing an example of the evaluation result data 400 according to the first embodiment. The evaluation result data 400 shown as an example in FIG. 3 ),f(t 4) does not satisfy the standard. Similarly, the evaluation result data 400 indicates that the accuracy of the three-dimensional shapes 303 and 304 indicated by the three-dimensional shape data associated with the other frames f(t 1 ),f(t 2 ),f(t 5 ) satisfies the standard. Note that a predetermined evaluation such as a determination of whether the accuracy of the three-dimensional shape satisfies the standard may be a manual evaluation such as visual inspection by a natural person, or an automatic evaluation using a known technology.
[0018] The generating unit 204 corrects, based on the 3D shape sequence data and the evaluation result data, the 3D shape data corresponding to the 3D shape not satisfying the criteria among the 3D shape sequence data. The correction method in the generating unit 204 will be described later in detail. The output unit 205 outputs the 3D shape sequence data after the generating unit 204 has corrected the 3D shape data to the external storage device 111 or the like. The 3D shape sequence data output by the output unit 205 is stored in, for example, the external storage device 111 or the like.
[0019] <Operation of the information processing device> The operation of the information processing device 100 will be described with reference to Figs. 5 to 8. Fig. 5 is a flowchart showing an example of a processing flow in the information processing device 100 according to the first embodiment. In the following description, the symbol "S" means a step. First, in S501, the shape data acquisition unit 201 acquires three-dimensional shape sequence data based on input information representing an instruction from a user. Next, in S502, the evaluation acquisition unit 202 acquires evaluation result data for the three-dimensional shape sequence data acquired in S501 based on the input information representing an instruction from the user.
[0020] Next, in S503, the generation unit 204 corrects the 3D shape data included in the 3D shape sequence data, the evaluation result of which does not satisfy the criteria, based on the 3D shape sequence data acquired in S501 and the evaluation result data acquired in S502. Hereinafter, the 3D shape data to be corrected is referred to as the shape data to be corrected, and the 3D shape data after correction is referred to as the post-correction shape data. That is, the generation unit 204 generates the post-correction shape data to replace the shape data to be corrected by the process of S503.
[0021] Specifically, for example, the generating unit 204 corrects the correction target shape data as follows. First, the generating unit 204 selects one piece of 3D shape data whose evaluation result satisfies a criterion from among a plurality of pieces of 3D shape data included in the 3D shape sequence data. Next, the generating unit 204 generates 3D shape data by matching the 3D shape of the object indicated by the selected 3D shape data to the position of the 3D shape of the object indicated by the correction target shape data. The generated 3D shape data becomes post-correction shape data that replaces the correction target shape data. The correction process in the generating unit 204, i.e., the process of generating post-correction shape data, will be described in detail later.
[0022] Next, in S504, the output unit 205 outputs modified 3D shape sequence data in which the shape data to be modified in the 3D shape sequence data acquired in S501 is replaced with the modified shape data generated in S503. The modified 3D shape sequence data output in S504 is stored in, for example, the external storage device 111. After S504, the information processing device 100 ends the process of the flowchart shown in FIG.
[0023] <Generation process of corrected shape data> In the first embodiment, the three-dimensional shape of an object represented by each three-dimensional shape data included in the three-dimensional shape sequence data is assumed to change gradually per unit time. That is, when the difference between the timestamps of frames associated with any two three-dimensional shape data included in the three-dimensional shape sequence data is small, the three-dimensional shapes of the objects represented by the two three-dimensional shape data are similar. Therefore, the generating unit 204 selects the three-dimensional shape data associated with a frame whose timestamp difference with the frame associated with the shape data to be modified is small among the three-dimensional shape data whose evaluation result included in the three-dimensional shape sequence data satisfies the criteria. Next, the generating unit 204 generates modified shape data based on the selected three-dimensional shape data as described above.
[0024] Fig. 6 is a flowchart showing an example of a processing flow in the generating unit 204 according to the first embodiment, and is a flowchart showing an example of a detailed flow of the processing in S503 shown in Fig. 5. The processing of the flowchart shown in Fig. 6 is executed after the processing of S502 shown in Fig. 5. First, in S601, the generating unit 204 selects one arbitrary correction target shape data for which post-correction shape data has not yet been generated, from among one or more correction target shape data included in the 3D shape sequence data.
[0025] Next, in S602, the generation unit 204 selects one piece of 3D shape data whose evaluation result satisfies a criterion from among the multiple pieces of 3D shape data included in the 3D shape sequence data. Specifically, the generation unit 204 selects 3D shape data whose evaluation result satisfies the criterion and which is associated with a frame whose timestamp is closest to the timestamp of the frame associated with the correction target shape data selected in S601. Hereinafter, the 3D shape data selected in S602 will be referred to as reference shape data. Also, the 3D shape of the object indicated by the reference shape data will be referred to as the "reference shape", and the 3D shape of the object indicated by the correction target shape data will be referred to as the "correction target shape".
[0026] The method of selecting the reference shape data is not limited to the above-mentioned method. For example, the generating unit 204 may select, as the reference shape data, three-dimensional shape data whose evaluation result satisfies the criteria and whose three-dimensional shape of the object represented by the three-dimensional shape data is expected to be similar to the three-dimensional shape of the object represented by the correction target shape data. The generating unit 204 may also select, as the reference shape data, three-dimensional shape data whose evaluation result satisfies the criteria and whose distance from the surface of the three-dimensional shape of the object represented by the three-dimensional shape data to the surface of the three-dimensional shape of the object represented by the correction target shape data is the shortest. The generating unit 204 may also select, as the reference shape data, three-dimensional shape data whose evaluation result satisfies the criteria and whose three-dimensional shape indicates a three-dimensional shape whose volume of the part overlapping with the three-dimensional shape of the object represented by the correction target shape data is the largest.
[0027] Next, in S603, the generation unit 204 aligns the overall position of the three-dimensional shape of the object indicated by the reference shape data selected in S602 with the overall position of the three-dimensional shape of the object indicated by the correction target shape data selected in S601. Specifically, the generation unit 204 translates and rotates the entire reference shape, and aligns the position and orientation of the reference shape with the position and orientation of the correction target shape so that the volume of the difference shape between the reference shape and the correction target shape after the translation and rotation is minimized. For example, the generation unit 204 translates and rotates the entire reference shape by performing affine transformation or the like using the same transformation parameters for all vertices constituting the reference shape. For example, the generation unit 204 uses the gradient descent method to determine the translation amount and rotation amount as transformation parameters such that the volume of the difference shape between the entire reference shape after the translation and rotation and the entire correction target shape is minimized. Hereinafter, the alignment of the position and orientation of the reference shape with the position and orientation of the correction target shape will be described as the alignment of the reference shape and the correction target shape, or simply as alignment. Note that the method of aligning the reference shape and the shape to be modified is not limited to the method of translating and rotating the reference shape using affine transformation, etc. For example, the generating unit 204 may align the reference shape and the shape to be modified using a known rigid body alignment technique.
[0028] 7A and 7B are diagrams illustrating an example of alignment between a reference shape and a shape to be corrected in the generating unit 204 according to the first embodiment. FIG. 7A is a diagram illustrating an example of a shape to be corrected. In FIG. 7A, the frame f(t 3 3. The three-dimensional shape 303 shown in FIG. 7B is a diagram showing an example of the reference shape. In FIG. 7B, the frame f(t 2 7(c) shows a three-dimensional shape 302 indicated by the three-dimensional shape data associated with the three-dimensional shape 302 shown in FIG. 7(c) is a diagram showing an example of a reference shape 701 after alignment by the generating unit 204. In FIG. 7(c), the three-dimensional shape 303 which is the shape to be corrected is also shown together with the reference shape 701 after alignment. FIG. 7(d) will be described later.
[0029] Next, in S604, the generation unit 204 calculates a difference volume ΔV between the reference shape after the alignment and the shape to be corrected by a predetermined threshold value Th ΔV If the volume ΔV is greater than a predetermined value, it is considered that the shape to be corrected is significantly different from the original shape, such as that the shape to be corrected is missing from the original shape, or that the shape to be corrected has an excessive portion compared to the original shape. Therefore, in S604, the volume ΔV is compared with the threshold value Th ΔV If it is determined that the above is true, in S606, the generating unit 204 further locally corrects the reference shape after alignment, and sets the data of the corrected reference shape as corrected shape data replacing the shape data to be corrected. Specifically, the generating unit 204 first compares the reference shape after alignment with the shape to be corrected, and identifies a portion with a large difference. Next, the generating unit 204 locally aligns the position of the identified portion in the reference shape after alignment with the position of a portion corresponding to the portion in the shape to be corrected. Next, the generating unit 204 sets the data of the reference shape after such local alignment (hereinafter referred to as "local alignment") as corrected shape data replacing the shape data to be corrected.
[0030] Usually, the reference shape and the shape to be corrected are different from each other. Also, before and after the alignment by the process of S603, the position and orientation of the reference shape change, but the shape does not change, as shown in Fig. 7(b) and (c) as an example. Therefore, if the alignment by the process of S603 is performed, there may be some parts where the overlap between the reference shape and the shape to be corrected is small, such as the shaded area in Fig. 7(c). Therefore, for example, the generating unit 204 performs alignment for such parts so that the overlap with the corresponding part of the shape to be corrected is maximized.
[0031] An example of a method of local alignment in the generation unit 204 will be described below. The generation unit 204 first calculates the distance d to the surface of the shape to be corrected for each vertex of the polygon mesh in the reference shape after alignment in the process of S603 (hereinafter simply referred to as "vertices of the reference shape after alignment"). The distance d from the vertex p of the reference shape after alignment to the surface of the shape to be corrected is calculated, for example, using the following formula (1).
[0032] JPEG2024164975000002.jpg8150, where (x p ,y p ,z p ) is the three-dimensional coordinate of the vertex p of the reference shape after alignment, and (x q ,y q ,z q ) is the three-dimensional coordinate of vertex q of the polygon mesh in the shape to be modified. Q t are the vertices of the polygon mesh in the shape to be modified, and the distance d is the distance from vertex p to vertex group Q t represents the shortest distance to each vertex in
[0033] The generation unit 204 then calculates the distance d by multiplying the calculated distance d by a predetermined threshold Th d The generation unit 204 then identifies a vertex of the reference shape after alignment that is larger than the vertex P , such that a plurality of vertices in the polygon mesh that are connected to each other by edges form one vertex group. diff(k) (k is an index to distinguish vertices).
[0034] 8A to 8C are diagrams for explaining an example of local registration in the generation unit 204 according to the first embodiment. diff(k) 8(a) and 8(b) show an example of a shape to be corrected 801 and a reference shape after alignment 802. The points indicated by the white circles in FIG. 8(a) are the points where the distance d is smaller than the threshold value Th d The dots indicated by the filled circles represent the vertices of the reference shape 802 after alignment for which the distance d is less than or equal to the threshold Th d8(a) shows, as an example, a cross section of the shape to be corrected 801 and the reference shape after alignment 802 on a certain plane in a three-dimensional space indicated by three-dimensional coordinate axes. In the example of FIG. 8(a), a total of three vertex groups P diff(1) , P diff(2) , and P diff(3) For example, the vertex group 702 shown in FIG. 7C is generated by dividing the vertex group P diff(1) Included in.
[0035] The generation unit 204 generates a vertex group P diff(k) The generation unit 204 regards the surface constituted by the vertex group P as a portion of the reference shape 802 after the alignment that is significantly different from the correction target shape 801. diff(k) To generate post-correction shape data, the generation unit 204 performs local alignment with the correction target shape 801 by performing different translations or rotations on the vertices P diff(k) Using individual transformation parameters for each vertex group P diff(k) For example, the generation unit 204 performs local registration processing for each vertex group Pdiff(k) by processing similar to the registration of the entire shape in S603. Here, for example, the generation unit 204 uses a gradient descent method to find a vertex group Pdiff(k) that minimizes the volume of the difference shape between the entire reference shape after local registration and the entire shape to be corrected. diff(k) The generating unit 204 obtains the amount of translation and the amount of rotation for each of the local alignments as transformation parameters. The generating unit 204 sets the data of the reference shape after the local alignment as corrected shape data.
[0036] Fig. 8(b) shows an example of corrected shape data 803, which is data of the reference shape after local alignment, generated by performing local alignment on the reference shape after alignment 802 shown in Fig. 8(a). Fig. 7(d) shows an example of corrected shape data 703, which is data of the reference shape after local alignment, generated by performing local alignment on the reference shape after alignment 701 shown in Fig. 7(c). By performing local alignment, the vertex group P diff(k) In other words, a surface that is distant from the shape to be corrected can be brought closer to the surface of the shape to be corrected while maintaining the local shape of the surface.
[0037] In addition, the vertex group P diff(k) The local alignment for the vertex set P diff(k) A vertex group P having a surface area equal to or greater than a predetermined threshold value. diff(k) It is preferable that the local registration is performed on the surface of the reference shape. By limiting the surface on which the local registration is performed in this manner, the area in which the surface shape of the reference shape is maintained is widened. As a result, it is possible to prevent the reference shape after the local registration from becoming too close to the correction target shape that does not satisfy the criteria, such as an erroneous uneven shape or a spike-like uneven shape.
[0038] In S604, the volume ΔV is the threshold value Th ΔV That is, the volume ΔV is not greater than the threshold Th ΔVIf it is determined that the difference is less than the reference shape data, in S605, the generation unit 204 sets the data of the reference shape after the alignment as post-correction shape data replacing the correction target shape data. After S605 or S606, in S607, the generation unit 204 determines whether or not all of the correction target shape data to be corrected has been corrected. If it is determined in S607 that all of the correction target shape data to be corrected has not been corrected, the generation unit 204 returns to S601 and repeatedly executes the processes from S601 to S607 until it is determined in S607 that all of the correction target shape data to be corrected has been corrected. If it is determined in S607 that all of the correction target shape data to be corrected has been corrected, the generation unit 204 ends the process of the flowchart shown in FIG. 6. After the process of the flowchart ends, the process of S504 is executed.
[0039] According to the information processing device 100 configured as above, among a plurality of chronologically consecutive 3D shapes, a 3D shape whose evaluation result does not satisfy the standard is corrected based on a 3D shape that satisfies the standard. According to the information processing device 100, such correction makes it possible to generate 3D shape data that does not feel unnatural due to changes in shape over time.
[0040] In the above description, the generating unit 204 has been described as correcting the correction target shape data using one reference shape data, but the generating unit 204 may correct the correction target shape data using multiple reference shape data. For example, the generating unit 204 selects multiple reference shape data in S602, and then applies the processes from S603 to S605 or S606 to each selected reference shape data to generate multiple corrected shape data corresponding to the selected multiple reference shape data. Next, the generating unit 204 may generate average corrected shape data from the generated multiple corrected shape data, and use the generated average corrected shape data as the final corrected shape data. Here, the average corrected shape data is generated by, for example, calculating the average coordinates of points on the surface that are associated by non-rigid registration between the multiple corrected shape data.
[0041] In the above description, the evaluation result of the three-dimensional shape data is described as a result of determining whether or not the three-dimensional shape of the object indicated by the three-dimensional shape data satisfies the accuracy standard as a whole, but the evaluation result is not limited to this. For example, the evaluation result may be an evaluation result for a predetermined part in the three-dimensional shape of the object indicated by the three-dimensional shape data. Here, the predetermined part in the three-dimensional shape is, for example, a part of the human body such as the head, torso, arms, or legs when the three-dimensional shape of the object is a human body model indicating the three-dimensional shape of a natural person. The evaluation result may be for one predetermined part, or may be for each of a plurality of predetermined parts. When the evaluation result is an evaluation result for a predetermined part in the three-dimensional shape of the object indicated by the three-dimensional shape data, for example, the generating unit 204 corrects the correction target shape data as follows. In this case, the generating unit 204 applies the above-mentioned local alignment or the like to only the part of the three-dimensional shape of the object indicated by the correction target shape data that is determined not to satisfy the standard in the evaluation result, for example, to generate corrected shape data.
[0042] [Embodiment 2] The information processing device 100 according to the first embodiment selects one reference shape data from among a plurality of 3D shapes whose evaluation results satisfy the criteria, and generates modified shape data by aligning the position of the reference shape with the position of the shape to be modified. In contrast, in the second embodiment, a form will be described in which a plurality of reference shape data is selected, and modified shape data is generated using the selected plurality of reference shape data.
[0043] The hardware configuration and functional configuration of the information processing device according to the second embodiment are the same as those shown in Fig. 1 or 2. Therefore, in the second embodiment, the parts different from the first embodiment will be mainly described. Note that the same components as those in the first embodiment will be described with the same reference numerals. In the following description, the information processing device according to the second embodiment will be referred to as the information processing device 100 or the information processing device 100 according to the first embodiment.
[0044] <Generation process of corrected shape data> In the information processing device 100 according to the second embodiment, the processing in the generating unit 204 is different from that in the generating unit 204 according to the first embodiment. That is, in the information processing device 100 according to the second embodiment, the processing for generating modified shape data in S503 shown in FIG. 5 is different from that in the first embodiment. The generating unit 204 according to the second embodiment (hereinafter, simply referred to as the "generating unit 204") selects a plurality of reference shape data, and generates modified shape data using the selected plurality of reference shape data. Specifically, the generating unit 204 predicts the position and shape of an object at the timestamp of a frame corresponding to the shape data to be modified by a three-dimensional shape interpolation process based on a plurality of reference shapes indicated by the selected plurality of reference shape data, and generates modified shape data.
[0045] FIG. 9 is a flowchart showing an example of a processing flow in the generating unit 204 according to the second embodiment, and is a flowchart showing an example of the details of the flow of the generation processing of the modified shape data according to the second embodiment in S503 shown in FIG. 5. The processing of the flowchart is executed after the processing of S502 shown in FIG. 5. First, in S901, one arbitrary modification target shape data for which modified shape data has not yet been generated is selected from one or more modification target shape data included in the three-dimensional shape sequence data. The processing of S901 is the same as the processing of S601. Next, in S902, the generating unit 204 selects, as a plurality of reference shape data, a plurality of three-dimensional shape data whose evaluation results satisfy a criterion from a plurality of three-dimensional shape data included in the three-dimensional shape sequence data. In the second embodiment, the generating unit 204 is described as selecting two reference shape data that sandwich the modification target shape data selected in S601 on the time axis.
[0046] Specifically, for example, the generating unit 204 selects, as the first reference shape data, three-dimensional shape data whose evaluation result satisfies a criterion and corresponds to a frame whose timestamp is earlier than the timestamp of the frame corresponding to the shape data to be corrected selected in S901. For example, the generating unit 204 selects, as the first reference shape data, three-dimensional shape data whose evaluation result satisfies a criterion and corresponds to a frame whose timestamp is earlier than and closest to the timestamp of the frame corresponding to the shape data to be corrected, among the three-dimensional shape data whose evaluation result satisfies a criterion. In addition, the generating unit 204 selects, as the second reference shape data, three-dimensional shape data whose evaluation result satisfies a criterion and corresponds to a frame whose timestamp is later than the timestamp of the frame corresponding to the shape data to be corrected selected in S901. For example, the generating unit 204 selects, as the second reference shape data, three-dimensional shape data whose evaluation result satisfies a criterion and corresponds to a frame whose timestamp is later than and closest to the timestamp of the frame corresponding to the shape data to be corrected, among the three-dimensional shape data whose evaluation result satisfies a criterion.
[0047] Hereinafter, the shape to be corrected indicated by the shape data to be corrected selected in S901 is referred to as the shape to be corrected f t , the reference shape indicated by the first reference shape data is referred to as the first reference shape f ref1 , and the reference shape indicated by the second reference shape data is referred to as the second reference shape f ref2 In the following description, the first reference shape f ref1 and a second reference shape f ref2 It is desirable that the surface shapes of the two surfaces are homeomorphic to each other. This is because, if the two surface shapes are not homeomorphic to each other, it is topologically impossible to continuously transform one surface shape into the other surface shape, making it difficult to perform surface shape interpolation processing.
[0048] Next, in S903, the generating unit 204 uses the multiple reference shape data selected in S902, specifically, the first and second reference shape data, to generate corrected shape data to replace the shape data to be corrected. ref1 and the second reference shape f ref2By the interpolation process using the above, the correction target shape f selected in S901 is t The generating unit 204 generates three-dimensional shape data corresponding to the time stamp of the frame corresponding to the data of the three-dimensional shape. The three-dimensional shape data generated by the generating unit 204 becomes post-correction shape data in place of the shape data to be corrected.
[0049] An example of the interpolation process of the generating unit 204 will be described below. The generating unit 204 first performs correspondence between points constituting the surface shape between a plurality of reference shapes. Specifically, the first reference shape f ref1 Polygon mesh M ref1 in the second reference frame f ref2 Polygon mesh M ref2 By this non-rigid registration, the generation unit 204 determines whether the connection relationship between the vertices is M ref1 and the shape is M ref2 A polygon mesh M' that approximately matches ref2 As a result, we obtain a polygon mesh M ref1 and the vertices of the polygon mesh M ref2 The points on the surface of the polygon mesh M' ref2 A correspondence relationship between the vertices of the vectors is obtained. A well-known technique can be used for the non-rigid registration technique, and since this technique is not the main focus of the present disclosure, a detailed description of the technique will be omitted.
[0050] Next, the generating unit 204 generates a polygon mesh M ref1 Each vertex p n (t ref1 ) and the coordinates of vertex p n (t ref1 ) corresponding polygon mesh M' ref2 Vertex p' of n (t ref2 ) and the coordinates of the shape to be corrected f t Vertex p in n Specifically, for example, the generation unit 204 uses the following equation (2) to interpolate the coordinates of the correction target shape f t Vertex p in n Interpolate the coordinates of (t). Note that n is N VERT is a positive integer less than or equal to NVERT is a polygon mesh M ref1 is a number indicating the number of vertices that make up the
[0051] JPEG2024164975000003.jpg18150Here, t is the shape to be corrected f t The timestamp of the frame corresponding to t ref1 is the first reference shape f ref1 The timestamp of the frame corresponding to t ref2 is the second reference shape f ref2 is the timestamp of the frame corresponding to (x n (t ref1 ),y n (t ref1 ),z n (t ref1 )) is the vertex p n (t ref1 ), and (x n (t ref2 ),y n (t ref2 ),z n (t ref2 )) is the vertex p' n (t ref2 ) coordinates. Furthermore, (x n (t), y n (t),z n (t)) is the vertex p after the interpolation process. n are the coordinates of (t).
[0052] The generation unit 204 generates a plurality of vertices p n (t), and each vertex p n The connection between (t) is M ref1 Matching polygon mesh M t The generation unit 204 obtains the obtained polygon mesh M t Modify the target shape f t The modified shape data at the time stamp of the frame corresponding to the polygon mesh M ref1 , M ref2 , M' ref2 , and M tSpecifically, FIG. 10(a) shows an example of a polygon mesh M ref1 An example of this is shown in Fig. 10(b) as a polygon mesh M ref2 An example of this is shown in Fig. 10(c) as a polygon mesh M' ref2 An example of this is shown in Fig. 10(d) as a polygon mesh M t An example of each is shown.
[0053] After S903, in S904, the generation unit 204 determines whether or not all of the shape data to be corrected has been corrected. If it is determined in S904 that all of the shape data to be corrected has not been corrected, the generation unit 204 returns to S901 and repeatedly executes the processes from S901 to S904 until it is determined in S904 that all of the shape data to be corrected has been corrected. If it is determined in S904 that all of the shape data to be corrected has been corrected, the generation unit 204 ends the process of the flowchart shown in Fig. 9. After the process of the flowchart ends, the process of S504 shown in Fig. 5 is executed.
[0054] According to the information processing device 100 configured as above, among a plurality of temporally consecutive three-dimensional shapes, a three-dimensional shape whose evaluation result does not satisfy the criterion is corrected based on a three-dimensional shape that satisfies the criterion. According to the information processing device 100, such correction makes it possible to generate three-dimensional shape data in which the change in shape over time is less unnatural. Specifically, the information processing device 100 uses a plurality of three-dimensional shape data whose evaluation result satisfies the criterion as reference shape data, predicts the position and shape of the three-dimensional shape of an object represented by the three-dimensional shape data that does not satisfy the criterion, and generates corrected shape data.
[0055] In the second embodiment, the modified shape data is generated by linear interpolation processing using two reference frames, but the method of the interpolation processing is not limited to this. For example, in S902, the generation unit 204 may select three or more reference shape data. In this case, in S903, the generation unit 204 performs correspondence between points constituting the surface shape between a plurality of reference shapes corresponding to the three or more reference shape data selected in S902. Furthermore, in S904, the generation unit 204 estimates the coordinates of the vertices of the three-dimensional shape at the time stamp of the frame corresponding to the shape data to be modified by a spline interpolation process or the like using the coordinates of each corresponding point. Note that the correspondence between points constituting the surface shape between three or more reference shapes can be performed by performing non-rigid positioning of the reference shape with the largest number of vertices among the plurality of reference shapes with respect to the other reference shapes.
[0056] Also, if there is no 3D shape data corresponding to a frame with a time stamp earlier than the time stamp of the frame corresponding to the correction target shape data that satisfies the evaluation criteria, the generating unit 204 selects reference shape data in S902, for example, as follows. In this case, for example, the generating unit 204 selects 3D shape data that satisfies multiple evaluation criteria as reference shape data from among 3D shape data corresponding to frames with time stamps later than the time stamp of the frame corresponding to the correction target shape data. Next, in S903, for example, the generating unit 204 estimates the coordinates of vertices in the correction target shape by extrapolation using the coordinates of vertices in the surface shape between multiple reference shapes corresponding to the reference shape data selected in S902. The same applies when there is no 3D shape data corresponding to a frame with a time stamp later than the time stamp of the frame corresponding to the correction target shape data that satisfies the evaluation criteria. Specifically, in this case, for example, the generating unit 204 may select and extrapolate multiple reference shape data from among 3D shape data corresponding to frames with time stamps earlier than the time stamp of the frame corresponding to the correction target shape data.
[0057] The generation unit 204 also generates a polygon mesh M t 6 may be applied to the reference shape to be processed in the processes from S603 to S606, and the result may be used as the modified shape data. In this case, the generating unit 204 may convert the reference shape to be processed in the processes from S603 to S606 into a polygon mesh M t and execute the processes from S603 to S606.
[0058] [Embodiment 3] The information processing device 100 according to the first and second embodiments generates corrected shape data based on a three-dimensional shape datum whose evaluation result satisfies a criterion. In contrast, in the third embodiment, a form will be described in which corrected shape data is generated based on data of frames of multi-view images corresponding to three-dimensional shape data whose evaluation result satisfies a criterion in a three-dimensional shape sequence estimated based on multi-view images. Here, the multi-view images are multiple captured images obtained by capturing images of an object from different directions in a synchronized manner using multiple imaging devices.
[0059] <Functional configuration of information processing device> FIG. 11 is a block diagram showing an example of a functional configuration of an information processing device 100 according to the third embodiment (hereinafter, simply referred to as "information processing device 100"). The information processing device 100 has an image acquisition unit 1100, a shape data acquisition unit 1101, an evaluation acquisition unit 1102, and a correction unit 1103 as a functional configuration. The correction unit 1103 has an estimation unit 1106 and an output unit 1105. Each unit of the information processing device 100 as a functional configuration is realized by the CPU 101 executing a program stored in the ROM 103 using the RAM 102 as a work memory. Note that it is not necessary for all of the processes shown below to be executed by the CPU 101, and the information processing device 100 may be configured so that part or all of the processes are executed by one or more processing circuits other than the CPU 101.
[0060] The image acquisition unit 1100 acquires data of a multi-view image and camera parameters of an imaging device corresponding to each of a plurality of captured images constituting the multi-view image, based on input information representing an instruction from a user input via the input device 113. Note that in the third embodiment, each captured image constituting the multi-view image will be described as a moving image composed of a plurality of frames. That is, in the third embodiment, the data of the multi-view image is sequence data (hereinafter referred to as "multi-view image sequence data") in which data of a plurality of frames (hereinafter referred to as "multi-view frames") obtained by synchronous imaging is continuous in time.
[0061] 12 is a diagram showing an example of data of a sequence of a plurality of captured images (hereinafter referred to as a "captured image sequence") included in the multi-viewpoint image sequence data acquired by the image acquisition unit 1100 according to the third embodiment. The multi-viewpoint image sequence data 1200 shown in FIG. 12 is made up of data of Ncam captured image sequences. The captured image sequence obtained by imaging with the nth corresponding imaging device starts at time t 1 ,t 2 ,t 3 ,t 4 ,t 5 ,..., the frame image n (t 1 ),image n (t 2 ),image n (t 3 ),image n (t 4 ),image n (t 5 Here, n is a positive integer equal to or smaller than Ncam. In the following, a plurality of captured images obtained by synchronous imaging of a plurality of imaging devices at a time t, i.e., image 1 (t),image 2 (t),...,image Ncom Here, (t) is represented as a multi-view frame f(t) for explanation.
[0062] The camera parameters include, for example, internal parameters, external parameters, and distortion parameters of each of the multiple image capture devices. The internal parameters are parameters that represent the position of the principal point of the image capture device and the focal length of the lens. The external parameters are parameters that represent the position and orientation of the image capture device. The distortion parameters are parameters that represent the distortion of the lens of the image capture device. In the following description, it is assumed that the camera parameters of each image capture device are constant regardless of time. The multi-view image sequence data and camera parameters acquired by the image acquisition unit 1100 are transmitted to the estimation unit 1106, and the estimation unit 1106 receives them.
[0063] The shape data acquisition unit 1101 is the same as the shape data acquisition unit 201 according to the first embodiment. The three-dimensional shape sequence data acquired by the shape data acquisition unit 1101 is transmitted to the output unit 1105, which receives it. The three-dimensional shape sequence data acquired by the shape data acquisition unit 1101 is generated based on the multi-view image sequence data. The evaluation acquisition unit 1102 is the same as the evaluation acquisition unit 202 according to the first embodiment. The evaluation result data acquired by the evaluation acquisition unit 1102 is transmitted to the estimation unit 1106, which receives it.
[0064] The estimation unit 1106 estimates and generates modified shape data to replace the shape data to be modified based on the multi-view image sequence data, the camera parameters, and the evaluation result data. Specifically, the estimation unit 1106 first selects a plurality of reference shape data from among the three-dimensional shape data whose evaluation results satisfy a criterion. Next, the estimation unit 1106 generates, for each captured image, an interpolated image of the captured image included in the multi-view frame corresponding to the shape data to be modified, using the multi-view frame corresponding to each reference shape data. Next, the estimation unit 1106 generates modified shape data to replace the shape data to be modified by estimating a three-dimensional shape using the multi-view frame after the interpolation process made up of the generated interpolated image and the camera parameters. Details of the processing by the estimation unit 1106 will be described later with reference to FIG. 13.
[0065] <Operation of the information processing device> FIG. 13 is a flowchart showing an example of a processing flow in the information processing device 100 according to the third embodiment. First, in S1300, the image acquisition unit 1100 acquires multi-view image sequence data and camera parameters based on input information representing a user's instruction. Next, in S1301, the shape data acquisition unit 1101 acquires sequence data of a three-dimensional shape estimated using the multi-view image sequence data and camera parameters acquired in S1301 based on the input information representing the user's instruction. Next, in S1302, the evaluation acquisition unit 1102 acquires evaluation result data for the three-dimensional shape sequence acquired in S1301 based on the input information representing the user's instruction. Next, in S1303, the estimation unit 1106 selects one arbitrary correction target shape data for which corrected shape data has not yet been generated from one or more correction target shape data included in the three-dimensional shape sequence data. Next, in S1304, the estimation unit 1106 selects, as reference shape data, a plurality of pieces of 3D shape data whose evaluation results satisfy a criterion from among the 3D shape data included in the 3D shape sequence data.
[0066] For example, the estimation unit 1106 selects, as the first reference shape data, three-dimensional shape data corresponding to a multi-view frame with a time stamp earlier than the time stamp of the multi-view frame corresponding to the shape data to be corrected from among the three-dimensional shape data whose evaluation result satisfies the criterion. Specifically, the estimation unit 1106 selects, as the first reference shape data, three-dimensional shape data corresponding to a multi-view frame with a time stamp earlier than and closest to the time stamp of the multi-view frame corresponding to the shape data to be corrected from among the three-dimensional shape data whose evaluation result satisfies the criterion. Also, the estimation unit 1106 selects, as the second reference shape data, three-dimensional shape data corresponding to a multi-view frame with a time stamp later than the time stamp of the multi-view frame corresponding to the shape data to be corrected from among the three-dimensional shape data whose evaluation result satisfies the criterion. For example, the estimation unit 1106 selects, as the second reference shape data, three-dimensional shape data corresponding to a multi-view frame with a time stamp later than and closest to the time stamp of the multi-view frame corresponding to the shape data to be corrected from among the three-dimensional shape data whose evaluation result satisfies the criterion. Hereinafter, the multi-view frame corresponding to the shape data to be corrected is referred to as a multi-view frame f. t , the multi-view frame corresponding to the first reference shape data is called a multi-view frame f ref1 In addition, the multi-view frame corresponding to the second reference shape data is denoted as a multi-view frame f ref2 This will be described as follows.
[0067] Next, in S1305, the estimation unit 1106 estimates a multi-view frame f ref1 ,f ref2 By frame interpolation based on the above, a multi-view frame f corresponding to the shape data to be corrected is obtained. t That is, the estimation unit 1106 generates an interpolated image of multiple viewpoints at a time stamp of the frame image n (t ref1 ) and frame image n (t ref2 ) to obtain the interpolated image ' n(t) is generated. A well-known frame interpolation technique for two-dimensional images can be used to generate the interpolated image. Note that the estimation unit 1106 may select three or more pieces of reference shape data in S1304. In this case, the estimation unit 1106 may generate a multi-viewpoint interpolated image using multi-viewpoint frames corresponding to the selected three or more pieces of reference shape data.
[0068] Next, in S1306, the estimation unit 1106 generates modified shape data to replace the data to be modified, based on the multi-viewpoint interpolation image generated in S1305 and the camera parameters acquired in S1300. Specifically, for example, the estimation unit 1106 first estimates the three-dimensional shape of an object appearing in the interpolation image based on the multi-viewpoint interpolation image and the camera parameters. Then, the estimation unit 1106 sets data indicating the three-dimensional shape obtained by the estimation as modified shape data. Well-known estimation techniques such as the Visual Hull method or the stereo matching method can be used to estimate the three-dimensional shape. The modified shape data estimated and generated by the estimation unit 1106 is estimated based on the data of the multi-viewpoint frame corresponding to the three-dimensional shape data whose evaluation result satisfies the criterion. Therefore, the modified shape data estimated and generated by the estimation unit 1106 can be expected to show a three-dimensional shape with higher accuracy than the three-dimensional shape of the shape data to be modified whose evaluation result does not satisfy the criterion.
[0069] Next, in S1307, the estimation unit 1106 determines whether or not all of the shape data to be corrected has been corrected. If it is determined in S1307 that all of the shape data to be corrected has not been corrected, the estimation unit 1106 executes the process of S1303. Thereafter, the estimation unit 1106 repeatedly executes the processes from S1303 to S1307 until it is determined in S1307 that all of the shape data to be corrected has been corrected. If it is determined in S1307 that all of the shape data to be corrected has been corrected, the output unit 1105 executes the process of S1308. Specifically, in this case, in S1308, the output unit 1105 outputs corrected 3D shape sequence data in which the shape data to be corrected in the 3D shape sequence data acquired in S1301 has been replaced with the corrected shape data generated in S1306. The corrected 3D shape sequence data output in S1308 is stored in, for example, the external storage device 111 or the like. After S1308, the information processing device 100 ends the process of the flowchart shown in FIG.
[0070] According to the information processing device 100 configured as above, among a plurality of 3D shapes successive in time, a 3D shape whose evaluation result does not satisfy the criterion is corrected based on the data of the multi-view frame corresponding to the data of the 3D shape that satisfies the criterion. According to the information processing device 100, such correction makes it possible to generate 3D shape data that does not feel unnatural due to the change in shape over time.
[0071] [Embodiment 4] In the fourth embodiment, a form will be described in which corrected shape data that replaces the shape data to be corrected is generated while switching between correction based on reference shape data and correction based on data of a multi-view frame that corresponds to the reference shape data.
[0072] <Functional configuration of information processing device> FIG. 14 is a block diagram showing an example of a functional configuration of an information processing device 100 according to the fourth embodiment (hereinafter, simply referred to as "information processing device 100"). The information processing device 100 has an image acquisition unit 1401, a shape data acquisition unit 1402, an evaluation unit 1410, and a correction unit 1403. The correction unit 1403 has a switching unit 1411, a generation unit 1404, an estimation unit 1406, and an output unit 1405. Each unit of the information processing device 100 as a functional configuration is realized by the CPU 101 executing a program stored in the ROM 103 using the RAM 102 as a work memory. Note that it is not necessary for all of the processes shown below to be executed by the CPU 101, and the information processing device 100 may be configured so that part or all of the processes are executed by one or more processing circuits other than the CPU 101.
[0073] The image acquisition unit 1401 is similar to the image acquisition unit 1100 according to the third embodiment, and therefore detailed description thereof will be omitted. The multi-view image sequence data and camera parameters acquired by the image acquisition unit 1401 are transmitted to the estimation unit 1406 and the evaluation unit 1410, and are received by the estimation unit 1406 and the evaluation unit 1410, respectively. In addition, in the fourth embodiment, each captured image constituting the multi-view image will be described as a moving image composed of a plurality of frames. That is, in the third embodiment, the data of the multi-view image is sequence data composed of data of a plurality of multi-view frames that are consecutive in time.
[0074] The shape data acquisition unit 1402 is similar to the shape data acquisition unit 1101 according to the third embodiment. The three-dimensional shape sequence data acquired by the shape data acquisition unit 1402 is transmitted to the output unit 1405 and the evaluation unit 1410, and is received by each of the output unit 1405 and the evaluation unit 1410. The three-dimensional shape sequence data acquired by the shape data acquisition unit 1402 is generated based on the multi-view image sequence data.
[0075] The evaluation unit 1410 uses the multi-view image sequence data and the camera parameters to evaluate the three-dimensional shapes of the objects represented by each of the multiple three-dimensional shape data included in the three-dimensional shape sequence data, and obtains evaluation result data. Specifically, the evaluation unit 1410 first uses the camera parameters to project the three-dimensional shape of the object represented by the three-dimensional shape data onto a two-dimensional image for each of the three-dimensional shape data, and calculates a difference such as an average color difference from the multi-view frame included in the multi-view image sequence data. Next, the evaluation unit 1410 evaluates the three-dimensional shapes of the objects represented by each of the three-dimensional shape data by determining whether the calculated difference satisfies a predetermined criterion. For example, when the calculated difference satisfies a predetermined criterion, the evaluation unit 1410 performs evaluation by determining that the accuracy of the three-dimensional shapes of the objects represented by each of the three-dimensional shape data satisfies the criterion. The evaluation unit 1410 transmits the evaluation result data to the switching unit 1411, and the switching unit 1411 receives it.
[0076] The switching unit 1411 selects multiple reference shape data from the 3D shape data whose evaluation results satisfy the criteria, and switches between correction based on the reference shape data and correction based on the multi-view frame data according to the surface shape of the 3D shape indicated by the selected reference shape data. Details of the correction method switching will be described later with reference to FIG. 15. The generating unit 1404 is similar to the generating unit 1104 according to the third embodiment, and the estimating unit 1406 is similar to the estimating unit 1106 according to the third embodiment, and therefore detailed description thereof will be omitted. Note that the processing of the generating unit 1404 and the estimating unit 1406 is executed according to the switching of the correction method by the switching unit 1411.
[0077] <Operation of the information processing device> FIG. 15 is a flowchart showing an example of a processing flow in the information processing device 100 according to the fourth embodiment. First, in S1500, the image acquisition unit 1401 acquires multi-view image sequence data and camera parameters based on input information representing a user's instruction. Next, in S1501, the shape data acquisition unit 1402 acquires sequence data of a three-dimensional shape estimated using the multi-view image sequence data and camera parameters acquired in S1501 based on the input information representing a user's instruction. Next, in S1502, the evaluation unit 1410 evaluates each three-dimensional shape data included in the three-dimensional shape sequence data acquired in S1501 using the multi-view image sequence data and camera parameters acquired in S1500. Next, in S1503, the switching unit 1411 selects one of the correction target shape data for which corrected shape data has not yet been generated from among one or more correction target shape data included in the three-dimensional shape sequence data based on the evaluation result by the process of S1502. Here, the correction target shape data selected by the switching unit 1411 is the three-dimensional shape data that does not satisfy the criteria in the evaluation result in the process of S1502.
[0078] Next, in S1504, the switching unit 1411 selects a plurality of reference shape data from among the three-dimensional shape data that is determined to satisfy the criteria in the evaluation result in the process of S1502, by a method similar to the process of S1304 shown in Fig. 13. Next, in S1505, the switching unit 1411 determines whether or not the surface shapes of the three-dimensional shapes indicated by all of the reference shape data selected in S1504 are homeomorphic to each other. If it is determined in S1505 that the surface shapes of all of the three-dimensional shapes are homeomorphic to each other, in S1506, the switching unit 1411 causes the generation unit 1404 to execute a process similar to the process of S903 according to the second embodiment. By executing this process, the generation unit 1404 generates corrected shape data to replace the shape data to be corrected. If it is determined in S1505 that at least a portion of the surface shapes of all the three-dimensional shapes are not homeomorphic to each other, in S1507 the switching unit 1411 causes the estimation unit 1406 to execute the same process as S1305 according to the third embodiment. After S1507, in S1508 the estimation unit 1406 executes the same process as S1306 according to the third embodiment. By executing this process, the estimation unit 1406 generates corrected shape data to replace the shape data to be corrected.
[0079] After S1506 or S1508, in S1509, the switching unit 1411 determines whether or not all of the shape data to be corrected has been corrected. If it is determined in S1509 that all of the shape data to be corrected has not been corrected, the information processing device 100 executes the process of S1503. Thereafter, the information processing device 100 repeatedly executes the processes from S1503 to S1509 until it is determined in S1307 that all of the shape data to be corrected has been corrected. If it is determined in S1509 that all of the shape data to be corrected has been corrected, the output unit 1405 executes the process of S1510. Specifically, in S1510, the output unit 1405 outputs corrected three-dimensional shape sequence data in which the shape data to be corrected in the three-dimensional shape sequence data acquired in S1501 has been replaced with the corrected shape data generated in S1506 or S1508. The corrected three-dimensional shape sequence data output in S1510 is stored in, for example, the external storage device 111 or the like. After S1510, the information processing device 100 ends the process of the flowchart shown in FIG.
[0080] According to the information processing device 100 configured as above, among a plurality of 3D shapes successive in time, a 3D shape whose evaluation result does not satisfy the standard can be corrected by switching the correction method according to the surface shape of the 3D shape indicated by the reference shape data. According to the information processing device 100, such correction makes it possible to generate 3D shape data that is less unnatural in the change of the shape over time.
[0081] The evaluation unit 1410 may re-evaluate the modified shape data generated by executing the processes from S1503 to S1508. For example, when the result of the re-evaluation by the evaluation unit 1410 does not satisfy a predetermined criterion, the information processing device 100 may be configured so that the modification unit 1403 re-generates the modified shape data. For example, the information processing device 100 reselects, as a reference frame, three-dimensional shape data different from the three-dimensional shape data selected so far in S1504, and performs the processes from S1505 to S1508 until the result of the re-evaluation by the evaluation unit 1410 satisfies the criterion. The configuration of the information processing device 100 when the result of the re-evaluation by the evaluation unit 1410 does not satisfy the criterion is not limited to this, and the information processing device 100 may be configured as follows. For example, a plurality of candidates for modified shape data may be generated using a plurality of different algorithms such as a plurality of frame interpolation processing algorithms or a plurality of three-dimensional shape estimation algorithms, and the one with the best evaluation result may be set as the modified shape data.
[0082] [Other embodiments] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.
[0083] It should be noted that, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.
[0084] [Configuration of this disclosure] The present disclosure includes the following configurations, methods, and programs.
[0085] <Configuration 1> A shape data acquisition means for acquiring three-dimensional shape sequence data including a plurality of three-dimensional shape data that are successive in time; an evaluation acquisition means for acquiring data of an evaluation result of a three-dimensional shape of an object represented by each of the plurality of three-dimensional shape data included in the three-dimensional shape sequence data; a generating means for generating third three-dimensional shape data to replace second three-dimensional shape data, the evaluation result of which does not satisfy a criterion, among the plurality of three-dimensional shape data, based on first three-dimensional shape data, the evaluation result of which satisfies a criterion, among the plurality of three-dimensional shape data; 13. An information processing device comprising:
[0086] <Configuration 2> Each of the plurality of three-dimensional shape data and the three-dimensional shape data generated by the generating means includes information indicating a position of the object. 2. The information processing device according to configuration 1,
[0087] <Configuration 3> the generating means selects at least one of the first three-dimensional shape data from the plurality of three-dimensional shape data, and generates the third three-dimensional shape data by aligning the three-dimensional shape indicated by the selected first three-dimensional shape data with the three-dimensional shape indicated by the second three-dimensional shape data. 3. The information processing device according to configuration 1 or 2,
[0088] <Configuration 4> The generating means performs rigid body registration to register the three-dimensional shape indicated by the selected first three-dimensional shape data with respect to the three-dimensional shape indicated by the first three-dimensional shape data. 4. The information processing device according to configuration 3,
[0089] <Component 5> The generating means sets the 3D shape data indicating the 3D shape indicated by the first 3D shape data after the alignment as the third 3D shape data when a volume of a difference area between the 3D shape indicated by the first 3D shape data after the alignment and the 3D shape indicated by the second 3D shape data is less than a predetermined threshold value. 5. The information processing device according to configuration 3 or 4,
[0090] <Component 6> the generating means specifies, as a first portion, a portion of the three-dimensional shape indicated by the first three-dimensional shape data after alignment, where a difference between the three-dimensional shape indicated by the first three-dimensional shape data after alignment and the three-dimensional shape indicated by the second three-dimensional shape data is equal to or greater than a predetermined threshold value, and generates the third three-dimensional shape data by aligning the specified first portion with a second portion in the three-dimensional shape indicated by the second three-dimensional shape data, which corresponds to the first portion. 6. The information processing device according to any one of configurations 3 to 5,
[0091] <Component 7> The generating means aligns the first portion with respect to the second portion by performing a rigid body alignment. 7. The information processing device according to configuration 6,
[0092] <Component 8> the generating means selects a plurality of the first three-dimensional shape data from the plurality of three-dimensional shape data, and generates the third three-dimensional shape data by performing an interpolation process or an extrapolation process using a plurality of three-dimensional shapes represented by the selected plurality of the first three-dimensional shape data. 8. The information processing device according to any one of configurations 1 to 7,
[0093] <Component 9> The generating means selects the first three-dimensional shape data based on a time stamp corresponding to the second three-dimensional shape data. 9. The information processing device according to any one of configurations 3 to 8,
[0094] <Component 10> an image acquisition means for acquiring multi-viewpoint image data obtained by capturing moving images synchronously using a plurality of imaging devices that capture images of an object from different directions; a shape data acquisition means for acquiring three-dimensional shape sequence data including a plurality of time-sequential three-dimensional shape data generated based on the multi-viewpoint image data; an evaluation acquisition means for acquiring data of an evaluation result of a three-dimensional shape of an object represented by each of the plurality of three-dimensional shape data included in the three-dimensional shape sequence data; a generating means for generating third three-dimensional shape data to replace second three-dimensional shape data, the evaluation result of which does not satisfy a criterion among the plurality of three-dimensional shape data, by estimating a three-dimensional shape of the object based on camera parameters of each of the plurality of image capturing devices and a set of frame data included in the multi-viewpoint image data used in generating first three-dimensional shape data, the evaluation result of which satisfies a criterion among the plurality of three-dimensional shape data; 13. An information processing device comprising:
[0095] <Component 11> the generating means selects a plurality of the first three-dimensional shape data from among the plurality of three-dimensional shape data based on time stamps corresponding to the second three-dimensional shape data, performs an interpolation process or an extrapolation process using a set of data of a plurality of the frames used in generating the selected plurality of the first three-dimensional shape data, thereby generating a frame data set replacing the set of data of the frames used in generating the second three-dimensional shape data, and generates the third three-dimensional shape data by estimating a three-dimensional shape of the object based on the generated frame data set and each of the camera parameters of the plurality of image capturing devices. 11. The information processing device according to configuration 10,
[0096] <Component 12> The generating means includes: a first generating means for generating the third three-dimensional shape data by estimating a three-dimensional shape of the object based on a set of data of the frames used in generating the first three-dimensional shape data; a second generating means for generating the third three-dimensional shape data based on the first three-dimensional shape data; a switching means for switching between generating the third three-dimensional shape data by the first generating means and generating the third three-dimensional shape data by the second generating means based on the first three-dimensional shape data; 12. The information processing device according to configuration 10 or 11, comprising:
[0097] <Component 13> the switching means selects a plurality of the first three-dimensional shape data from among the plurality of three-dimensional shape data based on time stamps corresponding to the second three-dimensional shape data, switches to generation of the third three-dimensional shape data by the second generation means when a plurality of three-dimensional shapes indicated by the selected plurality of the first three-dimensional shape data are homeomorphic to each other, and switches to generation of the third three-dimensional shape data by the first generation means when at least some of the plurality of three-dimensional shapes indicated by the selected plurality of the first three-dimensional shape data are not homeomorphic to each other. 13. The information processing device according to configuration 12,
[0098] <Component 14> the evaluation result includes a result of evaluation of a three-dimensional shape corresponding to each part constituting the object in the three-dimensional shape indicated by each of the plurality of three-dimensional shape data; The generating means generates the third three-dimensional shape data by correcting a three-dimensional shape corresponding to a portion of the three-dimensional shape represented by the second three-dimensional shape data where the result of the evaluation for each portion does not satisfy a standard. 14. The information processing device according to any one of configurations 1 to 13,
[0099] <Component 15> The evaluation acquisition means has an evaluation means for evaluating the three-dimensional shape indicated by the three-dimensional shape data. 15. The information processing device according to any one of configurations 1 to 14.
[0100] <Component 16> The generating means generates the third three-dimensional shape data so that a result of evaluation by the evaluating means of the three-dimensional shape represented by the third three-dimensional shape data is improved compared with the evaluation result of the three-dimensional shape represented by the second three-dimensional shape data. 16. The information processing device according to configuration 15,
[0101] <Component 17> a correcting means for correcting the three-dimensional shape sequence data by replacing the second three-dimensional shape data included in the three-dimensional shape sequence data with the third three-dimensional shape data; 17. The information processing device according to any one of configurations 1 to 16, further comprising:
[0102] <Component 18> an output means for outputting the modified three-dimensional shape sequence data; 18. The information processing device according to configuration 17, further comprising:
[0103] <Method 1> a shape data acquisition step of acquiring three-dimensional shape sequence data including a plurality of three-dimensional shape data that are successive in time; an evaluation acquisition step of acquiring data of an evaluation result of a three-dimensional shape of an object represented by each of the plurality of three-dimensional shape data included in the three-dimensional shape sequence data; a generating step of generating third three-dimensional shape data to replace second three-dimensional shape data, the evaluation result of which does not satisfy a criterion, among the plurality of three-dimensional shape data, based on first three-dimensional shape data, the evaluation result of which satisfies a criterion; 13. An information processing method comprising:
[0104] <Method 2> an image acquisition step of acquiring multi-viewpoint image data obtained by capturing moving images synchronously using a plurality of imaging devices that capture images of an object from different directions; a shape data acquisition step of acquiring three-dimensional shape sequence data including a plurality of time-sequential three-dimensional shape data generated based on the multi-viewpoint image data; an evaluation acquisition step of acquiring data of an evaluation result of a three-dimensional shape of an object represented by each of the plurality of three-dimensional shape data included in the three-dimensional shape sequence data; a generating step of generating third three-dimensional shape data to replace second three-dimensional shape data, the evaluation result of which does not satisfy the criterion among the plurality of three-dimensional shape data, by estimating a three-dimensional shape of the object based on camera parameters of each of the plurality of image capturing devices and a set of frame data included in the multi-viewpoint image data used in generating first three-dimensional shape data, the evaluation result of which satisfies the criterion among the plurality of three-dimensional shape data; 13. An information processing method comprising:
[0105] <Program> A program for causing a computer to function as the information processing device according to any one of configurations 1 to 18. [Explanation of symbols]
[0106] 100 Information processing device 201 Shape data acquisition unit 202 Evaluation Department 204 Generation part
Claims
1. a shape data acquisition means for acquiring three-dimensional shape sequence data including a plurality of three-dimensional shape data that are successive in time; an evaluation acquisition means for acquiring data of an evaluation result of a three-dimensional shape of an object represented by each of the plurality of three-dimensional shape data included in the three-dimensional shape sequence data; a generating means for generating third three-dimensional shape data to replace second three-dimensional shape data, the evaluation result of which does not satisfy a criterion, among the plurality of three-dimensional shape data, based on first three-dimensional shape data, the evaluation result of which satisfies a criterion, among the plurality of three-dimensional shape data; 13. An information processing device comprising:
2. Each of the plurality of three-dimensional shape data and the three-dimensional shape data generated by the generating means includes information indicating a position of the object.
2. The information processing apparatus according to claim 1,
3. the generating means selects at least one of the first three-dimensional shape data from the plurality of three-dimensional shape data, and aligns a three-dimensional shape represented by the selected first three-dimensional shape data with a three-dimensional shape represented by the second three-dimensional shape data, thereby generating the third three-dimensional shape data.
2. The information processing apparatus according to claim 1,
4. The generating means performs rigid body registration to register the three-dimensional shape indicated by the selected first three-dimensional shape data with respect to the three-dimensional shape indicated by the first three-dimensional shape data.
4. The information processing apparatus according to claim 3,
5. The generating means sets the three-dimensional shape data indicating the three-dimensional shape indicated by the first three-dimensional shape data after the alignment as the third three-dimensional shape data when a volume of a difference area between the three-dimensional shape indicated by the first three-dimensional shape data after the alignment and the three-dimensional shape indicated by the second three-dimensional shape data is less than a predetermined threshold value.
4. The information processing apparatus according to claim 3,
6. the generating means specifies, as a first portion, a portion of the three-dimensional shape indicated by the first three-dimensional shape data after alignment, where a difference between the three-dimensional shape indicated by the first three-dimensional shape data after alignment and the three-dimensional shape indicated by the second three-dimensional shape data is equal to or greater than a predetermined threshold value, and aligns the specified first portion with a second portion in the three-dimensional shape indicated by the second three-dimensional shape data, which corresponds to the first portion, to generate the third three-dimensional shape data.
4. The information processing apparatus according to claim 3,
7. The generating means aligns the first portion with respect to the second portion by performing rigid body alignment. The information processing device according to claim 6 .
8. the generating means selects a plurality of the first three-dimensional shape data from the plurality of three-dimensional shape data, and performs an interpolation process or an extrapolation process using a plurality of three-dimensional shapes represented by the selected plurality of the first three-dimensional shape data, thereby generating the third three-dimensional shape data.
2. The information processing apparatus according to claim 1,
9. The generating means selects the first three-dimensional shape data based on a time stamp corresponding to the second three-dimensional shape data.
4. The information processing apparatus according to claim 3,
10. an image acquisition means for acquiring multi-viewpoint image data obtained by capturing moving images synchronously using a plurality of imaging devices that capture images of an object from different directions; a shape data acquisition means for acquiring three-dimensional shape sequence data including a plurality of time-sequential three-dimensional shape data generated based on the multi-viewpoint image data; an evaluation acquisition means for acquiring data of an evaluation result of a three-dimensional shape of an object represented by each of the plurality of three-dimensional shape data included in the three-dimensional shape sequence data; a generating means for generating third three-dimensional shape data to replace second three-dimensional shape data, the evaluation result of which does not satisfy a criterion among the plurality of three-dimensional shape data, by estimating a three-dimensional shape of the object based on camera parameters of each of the plurality of image capturing devices and a set of frame data included in the multi-viewpoint image data used in generating first three-dimensional shape data, the evaluation result of which satisfies a criterion among the plurality of three-dimensional shape data; 13. An information processing device comprising:
11. the generating means selects a plurality of the first three-dimensional shape data from the plurality of three-dimensional shape data based on time stamps corresponding to the second three-dimensional shape data, performs an interpolation process or an extrapolation process using a set of data of a plurality of the frames used in generating the selected plurality of the first three-dimensional shape data, thereby generating a frame data set replacing the set of data of the frames used in generating the second three-dimensional shape data, and generates the third three-dimensional shape data by estimating a three-dimensional shape of the object based on the generated frame data set and each of the camera parameters of the plurality of image capturing devices. The information processing device according to claim 10 .
12. The generating means includes: a first generating means for generating the third three-dimensional shape data by estimating a three-dimensional shape of the object based on a set of data of the frames used in generating the first three-dimensional shape data; a second generating means for generating the third three-dimensional shape data based on the first three-dimensional shape data; a switching means for switching between generating the third three-dimensional shape data by the first generating means and generating the third three-dimensional shape data by the second generating means based on the first three-dimensional shape data; 11. The information processing apparatus according to claim 10, further comprising:
13. the switching means selects a plurality of the first three-dimensional shape data from among the plurality of three-dimensional shape data based on time stamps corresponding to the second three-dimensional shape data, switches to generation of the third three-dimensional shape data by the second generation means when a plurality of three-dimensional shapes indicated by the selected plurality of the first three-dimensional shape data are homeomorphic to each other, and switches to generation of the third three-dimensional shape data by the first generation means when at least some of the plurality of three-dimensional shapes indicated by the selected plurality of the first three-dimensional shape data are not homeomorphic to each other. The information processing device according to claim 12 .
14. the evaluation result includes a result of evaluation of a three-dimensional shape corresponding to each part constituting the object in the three-dimensional shape indicated by each of the plurality of three-dimensional shape data; The generating means generates the third three-dimensional shape data corrected for a three-dimensional shape corresponding to a portion of the three-dimensional shape represented by the second three-dimensional shape data that does not satisfy a standard as a result of the evaluation for each portion.
2. The information processing apparatus according to claim 1,
15. The evaluation acquisition means has an evaluation means for evaluating the three-dimensional shape indicated by the three-dimensional shape data.
2. The information processing apparatus according to claim 1,
16. The generating means generates the third three-dimensional shape data so that a result of evaluation by the evaluating means of the three-dimensional shape represented by the third three-dimensional shape data is improved compared with the evaluation result of the three-dimensional shape represented by the second three-dimensional shape data. The information processing device according to claim 15 .
17. a correcting means for correcting the three-dimensional shape sequence data by replacing the second three-dimensional shape data included in the three-dimensional shape sequence data with the third three-dimensional shape data; 2. The information processing apparatus according to claim 1, further comprising:
18. an output means for outputting the corrected three-dimensional shape sequence data; 20. The information processing apparatus according to claim 17, further comprising:
19. a shape data acquisition step of acquiring three-dimensional shape sequence data including a plurality of three-dimensional shape data that are successive in time; an evaluation acquisition step of acquiring data of an evaluation result of a three-dimensional shape of an object represented by each of the plurality of three-dimensional shape data included in the three-dimensional shape sequence data; a generating step of generating third three-dimensional shape data to replace second three-dimensional shape data, the evaluation result of which does not satisfy a criterion, among the plurality of three-dimensional shape data, based on first three-dimensional shape data, the evaluation result of which satisfies a criterion; 13. An information processing method comprising:
20. an image acquisition step of acquiring multi-viewpoint image data obtained by capturing moving images synchronously using a plurality of imaging devices that capture images of an object from different directions; a shape data acquisition step of acquiring three-dimensional shape sequence data including a plurality of time-sequential three-dimensional shape data generated based on the multi-viewpoint image data; an evaluation acquisition step of acquiring data of an evaluation result of a three-dimensional shape of an object represented by each of the plurality of three-dimensional shape data included in the three-dimensional shape sequence data; a generating step of generating third three-dimensional shape data to replace second three-dimensional shape data, the evaluation result of which does not satisfy the criterion among the plurality of three-dimensional shape data, by estimating a three-dimensional shape of the object based on camera parameters of each of the plurality of image capturing devices and a set of frame data included in the multi-viewpoint image data used in generating first three-dimensional shape data, the evaluation result of which satisfies the criterion among the plurality of three-dimensional shape data; 13. An information processing method comprising:
21. A program for causing a computer to function as the information processing device according to any one of claims 1 to 18.
Citation Information
Patent Citations
Method for evaluating three-dimensional shape model and method and device for generating the model
JP2003115042A
