Information processing device, information processing method, and program

Abstract

[Problem] Desired is the provision of a technology that makes it possible to more accurately estimate the movement of a three-dimensional model between a plurality of frames. [Solution] Provided is an information processing device comprising a movement amount calculation unit that calculates a movement amount associated with a first apex which is included in a first frame, such calculation being on the basis of statistical processing performed according to colour information associated with the first apex, colour information associated with a second apex which is included in a second frame that follows after the first frame, a three-dimensional coordinate associated with the first apex, and a three-dimensional coordinate associated with the second apex.

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, techniques for estimating the movement of a three-dimensional model between multiple frames have become known. For example, a technique for estimating the movement of a three-dimensional model based on the degree of correspondence between the shapes of the three-dimensional model between multiple frames has been disclosed (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-136943 Summary of the Invention [Problem to be solved by the invention]

[0004] However, it is desirable to provide a technique that can more accurately estimate the movement of a three-dimensional model between multiple frames. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, there is provided an information processing device including a movement amount calculation unit that calculates a movement amount associated with a first vertex based on statistical processing according to color information associated with a first vertex included in a first frame, color information associated with a second vertex included in a second frame that comes after the first frame, three-dimensional coordinates associated with the first vertex, and three-dimensional coordinates associated with the second vertex.

[0006] According to another aspect of the present disclosure, there is provided an information processing method including a processor calculating a movement amount associated with a first vertex based on statistical processing according to color information associated with a first vertex included in a first frame, color information associated with a second vertex included in a second frame that comes after the first frame, three-dimensional coordinates associated with the first vertex, and three-dimensional coordinates associated with the second vertex.

[0007] According to another aspect of the present disclosure, there is provided a program that causes a computer to function as an information processing device, the program including a movement amount calculation unit that calculates a movement amount associated with a first vertex based on statistical processing according to color information associated with a first vertex included in a first frame, color information associated with a second vertex included in a second frame that comes after the first frame, three-dimensional coordinates associated with the first vertex, and three-dimensional coordinates associated with the second vertex. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram for explaining an example of three-dimensional data extracted from imaging data in a volumetric capture technique. [Figure 2] FIG. 10 is a diagram showing details of a polygon structure. [Figure 3] FIG. 10 is a diagram illustrating an example of the configuration of polygon data representing a polygon. [Figure 4] FIG. 10 is a diagram illustrating an example of the configuration of vertex data indicating vertices. [Figure 5] FIG. 1 is a diagram illustrating a configuration example of an information processing device according to an embodiment of the present disclosure. [Figure 6] 10 is a flowchart illustrating an example of optical flow calculation. [Figure 7] FIG. 10 is a diagram illustrating an example of optical flow calculation. [Figure 8] 10A and 10B are diagrams illustrating examples of optical flows calculated by an optical flow calculation unit. [Figure 9] 10 is a flowchart illustrating an example of calculation of a destination position of an effect. [Figure 10] FIG. 10 is a diagram illustrating an example of calculation of a destination position of an effect. [Figure 11] FIG. 10 is a diagram showing the position of the effect in frame N. [Figure 12] FIG. 10 is a diagram showing the destination position of the effect in frame N+1. [Figure 13] FIG. 10 is a diagram illustrating an example of the configuration of an information processing device according to a first modified example. [Figure 14] FIG. 10 is a diagram illustrating an example of the configuration of an information processing device according to a second modified example. [Figure 15] FIG. 2 is a block diagram illustrating an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0010] Furthermore, in this specification and drawings, multiple components having substantially the same or similar functional configurations may be distinguished by adding different numbers after the same reference numeral. However, if there is no need to particularly distinguish between multiple components having substantially the same or similar functional configurations, only the same reference numeral will be used. Furthermore, similar components in different embodiments may be distinguished by adding different letters after the same reference numeral. However, if there is no need to particularly distinguish between similar components, only the same reference numeral will be used.

[0011] The explanation will be given in the following order. 0. Overview 1. Details of the embodiment 1.1. Device configuration example 1.2.Function details 2. Various Modifications 3. Hardware configuration example 4. Summary

[0012] <0. Overview> First, an overview of the embodiment of the present disclosure will be described.

[0013] In recent years, volumetric capture technology has become known as an example of a technology that extracts three-dimensional data of an object (e.g., a person) captured in image data (image data) captured continuously over time by multiple cameras. The object from which the three-dimensional data is extracted may be a three-dimensional model. Such volumetric capture technology uses the extracted three-dimensional data to reproduce three-dimensional moving images of the object from any viewpoint.

[0014] The three-dimensional data extracted by the volumetric capture technology is also called volumetric data. Volumetric data is three-dimensional moving image data composed of three-dimensional data (hereinafter also referred to as "frames") at each of a plurality of consecutive times. Here, examples of three-dimensional data extracted from imaging data by the volumetric capture technology will be described with reference to FIGS. 1 to 4.

[0015] Fig. 1 is a diagram illustrating an example of three-dimensional data extracted from imaging data using volumetric capture technology. Referring to Fig. 1, a three-dimensional model F1 in a certain frame is shown. Also referring to Fig. 1, a three-dimensional model F2 in the next frame is shown. The three-dimensional models F1 and F2 correspond to the same three-dimensional model at different times.

[0016] 1 illustrates a person as an example of a three-dimensional model. However, the three-dimensional model according to the embodiment of the present disclosure may include objects other than a person. The three-dimensional model is a model represented by three-dimensional data extracted from imaging data using volumetric capture technology.

[0017] Also, referring to FIG. 1, a polygon structure D1 is shown that shows a portion of a three-dimensional model F1 in detail. Similarly, a polygon structure D2 is shown that shows a portion of a three-dimensional model F2 in detail. That is, the three-dimensional model F1 and the three-dimensional model F2 are configured by polygon structures. Note that a polygon can mean a polygonal shape. Also, in the example shown in FIG. 1, the polygon structure D1 and the polygon structure D2 are configured by a combination of triangles (polygons with three vertices), but they may also be configured by polygons other than triangles (polygons with four or more vertices).

[0018] Fig. 2 is a diagram showing details of the polygon structure D1. Referring to Fig. 2, there is shown a diagram showing details of the polygon structure D1 shown in Fig. 1. The polygon structure D1 is made up of a combination of a plurality of polygons.

[0019] 2 shows polygons T0, T1, and T2 as examples of multiple polygons that make up polygon structure D1. Polygon T0 is made up of vertices V0, V1, and V2. Similarly, polygon T1 is made up of vertices V1, V2, and V3, and polygon T2 is made up of vertices V0, V2, and V4.

[0020] Fig. 3 is a diagram showing an example of the configuration of polygon data representing a polygon. As shown in Fig. 3, the polygon data includes the name of the polygon and the names of the vertices that make up the polygon. The name of the polygon is information for uniquely identifying the polygon in that frame. The names of the vertices that make up the polygon are information for uniquely identifying the vertices that make up the polygon in that frame.

[0021] FIG. 4 is a diagram showing an example of the configuration of vertex data indicating a vertex. As shown in FIG. 4, the vertex data includes the name of the vertex, the coordinates of the vertex, and color information of the vertex. As described above, the name of the vertex is information for uniquely identifying the vertex in that frame. The coordinates of the vertex are coordinates that represent the position of the vertex. As an example, the coordinates of the vertex can be represented by three-dimensional coordinates (x coordinate, y coordinate, z coordinate).

[0022] The color information of a vertex is information that indicates the color to be used to paint the surface formed by the vertices (hereinafter also referred to as a "mesh"). For example, the color information may be expressed using the RGB method, but may also be expressed using any method. As an example, the color information of the mesh formed by vertices V0, V1, and V2 (i.e., the color information of the mesh inside polygon T0) is determined based on the color information of vertex V0, vertex V1, and vertex V2.

[0023] Although volumetric data has the above-described structure, it is independent between frames and does not contain information indicating the correspondence between the positions of the same 3D model in multiple consecutive frames. This makes it difficult to accurately grasp the movement of the 3D model. For example, it is not easy to grasp to which polygon a polygon that exists near a certain part (e.g., a hand) in one frame has moved in the next frame.

[0024] For example, existing technologies include those that detect the positions of human body parts from imaging data. However, such existing technologies do not easily detect the positions of objects other than human body parts (e.g., clothing, props, etc.). Furthermore, other existing technologies include those that detect the positions of markers that have been attached in advance from imaging data. However, such existing technologies do not easily detect the positions of objects for which it is difficult to attach markers.

[0025] Further differences between the technology according to the embodiment of the present disclosure and other existing technologies will be described in more detail at the end of this specification. Furthermore, if the movement of a three-dimensional model is not accurately grasped, the following events may occur.

[0026] That is, a 3D model may be processed to add visual objects (hereinafter also referred to as "effects"). In this case, unless the movement of the 3D model is accurately grasped, the position of the effect, which changes in accordance with the movement of the 3D model, cannot be accurately estimated. If the position of the effect cannot be accurately estimated, the creator will have to decide the position of the effect manually, but if the creator has to decide all the positions of the effects, this will impose a heavy workload on the creator. In particular, the burden on the creator of making the effect follow the limbs or props is likely to be heavy.

[0027] Therefore, in an embodiment of the present disclosure, a technique is mainly proposed that can more accurately estimate the movement of a three-dimensional model between multiple frames. More specifically, in an embodiment of the present disclosure, the amount of movement associated with a vertex between frames is estimated based on color information associated with the vertex between frames. Here, the amount of movement may include at least one of the direction of movement and the distance of movement. The direction of movement and the distance of movement may correspond to a movement vector (hereinafter also referred to as "optical flow").

[0028] Optical flow is generally used in two-dimensional video. In two-dimensional video, the movement amount of each pixel between frames is generally calculated as two-dimensional optical flow. In the embodiment of the present disclosure, three-dimensional optical flow is calculated, but two-dimensional optical flow and three-dimensional optical flow differ in the following points.

[0029] That is, in two-dimensional video, the relative position between the subject and the light source does not change, but the camera moves. On the other hand, in three-dimensional video, the relative position between the light source and the camera does not change, but the subject moves. Furthermore, two-dimensional video is divided into pixels by a grid, but in three-dimensional video, the positions of vertices and polygons are random. Therefore, in an embodiment of the present disclosure, three-dimensional optical flow is calculated using a calculation method different from the calculation method for two-dimensional optical flow.

[0030] The outline of the embodiments of the present disclosure has been described above.

[0031] <1. Details of the embodiment> Next, embodiments of the present disclosure will be described in detail.

[0032] (1.1. Device configuration example) First, a configuration example of an information processing device according to an embodiment of the present disclosure will be described.

[0033] 5 is a diagram illustrating a configuration example of an information processing device according to an embodiment of the present disclosure. As shown in FIG. 5, the information processing device 10 according to an embodiment of the present disclosure is realized by a computer, and includes a control unit 120, a display unit 130, an operation unit 140, and a storage unit 150.

[0034] (control unit 120) The control unit 120 may be configured, for example, by one or more CPUs (Central Processing Units). When the control unit 120 is configured by a processing device such as a CPU, the processing device may be configured by an electronic circuit. The control unit 120 can be realized by the execution of a program by the processing device.

[0035] 5, the control unit 120 includes a motion capture unit 121, an optical flow calculation unit 122, an effect position calculation unit 123, an effect position proposal unit 124, an effect position correction unit 125, and a recording control unit 126. Details of the motion capture unit 121, the optical flow calculation unit 122, the effect position calculation unit 123, the effect position proposal unit 124, the effect position correction unit 125, and the recording control unit 126 will be described later.

[0036] (Display section 130) The display unit 130 presents various information to the creator under the control of the control unit 120. For example, the display unit 130 may include a display. The type of display is not limited. For example, the display included in the display unit 130 may be an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display, a PDP (Plasma Display Panel), or the like.

[0037] (Operation unit 140) The operation unit 140 has a function of accepting operations input by a creator of a 3D image. For example, the operation unit 140 may be configured with a mouse and a keyboard. Alternatively, the operation unit 140 may be configured with a touch panel, buttons, or an input device such as a microphone.

[0038] (Storage unit 150) The storage unit 150 is a recording medium that includes a memory and stores programs executed by the control unit 120 and data required for executing these programs. The storage unit 150 also temporarily stores data for calculations by the control unit 120. The storage unit 150 is configured by a magnetic storage device, a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.

[0039] The configuration example of the information processing device 10 according to the embodiment of the present disclosure has been described above.

[0040] (1.2. Function details) Next, detailed functions of the information processing device 10 according to an embodiment of the present disclosure will be described. In the embodiment of the present disclosure, multiple cameras are installed around a three-dimensional object (e.g., a person), and the multiple cameras capture images of the three-dimensional object. The multiple cameras are connected to the information processing device 10, and data (image data) captured continuously in time series by the multiple cameras is transmitted to the information processing device 10.

[0041] (Motion Capture Section 121) The motion capture unit 121 extracts three-dimensional data of a three-dimensional object based on image data captured by multiple cameras. This allows three-dimensional data at multiple consecutive times to be obtained as frames. The motion capture unit 121 continuously outputs the multiple frames thus obtained to the optical flow calculation unit 122.

[0042] The motion capture unit 121 may output multiple frames to the optical flow calculation unit 122 in real time, or may output multiple frames to the optical flow calculation unit 122 on demand in response to a request from the optical flow calculation unit 122. Each frame includes three-dimensional coordinates associated with the vertices and color information associated with the vertices.

[0043] (Optical flow calculation unit 122) The optical flow calculation unit 122 functions as a movement amount calculation unit that calculates the movement vectors associated with the vertices included in frame N as the optical flow associated with the vertices based on statistical processing according to color information associated with the vertices included in a target frame (hereinafter also referred to as "frame N") out of two consecutive frames, three-dimensional coordinates associated with the vertices included in the target frame, color information associated with the vertices included in a frame after frame N (hereinafter also referred to as "frame N+1"), and three-dimensional coordinates associated with the vertices included in frame N+1.

[0044] This allows for more accurate estimation of the movement of the three-dimensional model between multiple frames. Note that frame N may be an example of a first frame. Frame N+1 may be an example of a second frame. A vertex included in frame N may be an example of a first vertex. A vertex included in frame N+1 may be an example of a second vertex.

[0045] Furthermore, any of the three color attributes, hue, lightness, and saturation, may be used as color information. However, it is preferable to use hue as color information, which is relatively less affected by the relative position between the three-dimensional object and the light source or the intensity of the light emitted by the light source. In other words, color information associated with vertices included in frame N may include hue, and color information associated with vertices included in frame N+1 may include hue.

[0046] Note that in the embodiments of the present disclosure, it is mainly assumed that the optical flow is calculated on a mesh basis. However, the optical flow may also be calculated on a vertex basis. That is, in the embodiments of the present disclosure, it is mainly assumed that the optical flow is calculated using the three-dimensional coordinates of the mesh and the color information of the mesh. However, the three-dimensional coordinates of the vertices and the color information of the vertices may also be used to calculate the optical flow.

[0047] More specifically, in the embodiments of the present disclosure, it is mainly assumed that the color information associated with the vertices included in frame N is the color information of the mesh (first surface) formed by those vertices, and the color information associated with the vertices included in frame N+1 is the color information of the mesh (second surface) formed by those vertices.

[0048] Furthermore, in the embodiments of the present disclosure, it is mainly assumed that the three-dimensional coordinates associated with a vertex included in frame N are the three-dimensional coordinates of a mesh formed by that vertex, and the three-dimensional coordinates associated with a vertex included in frame N+1 are the three-dimensional coordinates of a mesh formed by that vertex.

[0049] In the embodiment of the present disclosure, it is mainly assumed that the movement amount associated with a vertex included in frame N is the movement amount of a mesh formed by that vertex.

[0050] However, as will be described in a later modified example, the color information associated with a vertex included in frame N may be the color information of that vertex, and the color information associated with a vertex included in frame N+1 may be the color information of that vertex. Furthermore, the three-dimensional coordinates associated with a vertex included in frame N may be the three-dimensional coordinates of that vertex, and the three-dimensional coordinates associated with a vertex included in frame N+1 may be the three-dimensional coordinates of that vertex. In this case, the movement amount associated with a vertex included in frame N may be the movement amount of that vertex.

[0051] The optical flow calculation unit 122 obtains color information of each vertex in frame N from frame N obtained by the motion capture unit 121, and obtains the three-dimensional coordinates of each vertex in frame N. Furthermore, the optical flow calculation unit 122 obtains color information of each vertex in frame N+1 from frame N+1, and obtains the three-dimensional coordinates of each vertex in frame N+1.

[0052] The optical flow calculation unit 122 calculates color information of each mesh in frame N based on the color information of each vertex obtained from frame N. Furthermore, the optical flow calculation unit 122 calculates three-dimensional coordinates of each mesh in frame N based on the three-dimensional coordinates of each vertex obtained from frame N. Similarly, the optical flow calculation unit 122 calculates color information of each mesh in frame N+1 based on the color information of each vertex obtained from frame N+1. Furthermore, the optical flow calculation unit 122 calculates three-dimensional coordinates of each mesh in frame N+1 based on the three-dimensional coordinates of each vertex obtained from frame N+1.

[0053] The color information of a mesh can be calculated by combining (for example, averaging) the color information of each of the three vertices that form the mesh, and the three-dimensional coordinates of the mesh can be calculated by the barycentric coordinates of the three-dimensional coordinates of each of the three vertices that form the mesh.

[0054] Fig. 6 is a flowchart showing an example of optical flow calculation. Fig. 7 is a diagram for explaining an example of optical flow calculation. An example of optical flow calculation S10 will be described in detail with reference to Figs. 6 and 7.

[0055] As shown in FIG. 6, the optical flow calculation unit 122 extracts, from each mesh obtained from frame N, a mesh for which an optical flow is to be calculated as a target mesh (S11).

[0056] Referring to FIG. 7, "Frame N" and "Frame N+1" are shown. Each circle included in "Frame N" and "Frame N+1" indicates a mesh included in that frame. The pattern of each mesh corresponds to the color information of the mesh. Therefore, meshes with the same pattern have the same color information. On the other hand, meshes with different patterns have different color information. For example, suppose that a target mesh M1 is extracted from frame N.

[0057] 6, the optical flow calculation unit 122 extracts one or more meshes that satisfy a predetermined relationship with the target mesh M1 from the meshes included in frame N+1 as meshes (third planes) in the statistical processing range of the target mesh M1. Then, the optical flow calculation unit 122 calculates a provisional optical flow for each mesh in the statistical processing range (S12).

[0058] Here, a mesh that satisfies the predetermined relationship with the target mesh M1 may be a mesh whose distance from the target mesh M1 is smaller than a second threshold (Y cm). For example, Y cm may be 6 cm, or may be a fixed value or may be variable. In "Frame N" shown in FIG. 7, a set of three-dimensional coordinates that are Y cm away from the three-dimensional coordinates of mesh M1 is shown as range Y.

[0059] More specifically, the optical flow calculation unit 122 extracts, from the plurality of meshes included in frame N+1, one or more meshes having three-dimensional coordinates whose distance from the three-dimensional coordinates of the target mesh M1 is smaller than Y cm as meshes in the statistical processing range of the target mesh M1. Here, it is assumed that meshes M1 to M6 are extracted as meshes in the statistical processing range of the target mesh M1.

[0060] Next, the optical flow calculation unit 122 extracts one or more meshes having three-dimensional coordinates whose distance from the three-dimensional coordinates of mesh M1 in the statistical processing range is smaller than a first threshold (defined as X cm). X cm may be a fixed value or may be variable. Note that the first threshold X cm may be the same as or different from the second threshold Y cm described above.

[0061] Then, the optical flow calculation unit 122 extracts, from the extracted one or more meshes, the mesh that has the smallest difference from the color information of the mesh M1 in the statistical processing range as a destination candidate mesh for the mesh M1 in the statistical processing range. The optical flow calculation unit 122 calculates the movement vector from the three-dimensional coordinates of the mesh M1 in the statistical processing range to the three-dimensional coordinates of the destination candidate mesh as a tentative optical flow of the mesh M1.

[0062] Similarly, the optical flow calculation unit 122 extracts a destination candidate mesh (fourth plane) for each of the meshes M2 to M6 in the statistical processing range. Then, the optical flow calculation unit 122 calculates a provisional optical flow for each of the meshes M2 to M6 in the statistical processing range.

[0063] The "provisional optical flow" shown in Fig. 7 is a diagram in which the meshes in "frame N" and the meshes in "frame N+1" shown in Fig. 7 are superimposed. However, in the "provisional optical flow", the pattern applied to the mesh in frame N is shown lighter than the pattern applied to the mesh in frame N+1. Furthermore, the "provisional optical flow" indicates the provisional optical flow of each mesh in frame N with arrows.

[0064] Next, the optical flow calculation unit 122 calculates the optical flow of the target mesh M1 based on the statistical processing of the provisional optical flows of each of the meshes M1 to M6 in the statistical processing range. By performing statistical processing on the provisional optical flows in this way, errors contained in the provisional optical flows can be removed.

[0065] In the embodiment of the present disclosure, a case where an average value calculation process is used as an example of statistical processing will be mainly described. That is, the optical flow calculation unit 122 calculates the average value of the provisional optical flows of each of the meshes M1 to M6 in the statistical processing range as the optical flow of the target mesh M1 (S13). The optical flow W1 shown in FIG. 7 is the optical flow of the target mesh M1.

[0066] However, the statistical processing is not limited to the processing of calculating an average value. For example, the statistical processing may be the processing of extracting a mode value. Note that whether the statistical processing is the processing of calculating an average value or the processing of extracting a mode value may be appropriately determined depending on the characteristics of the volumetric data, etc.

[0067] The optical flow calculation unit 122 calculates the optical flow of all meshes included in the frame N using a method similar to the method for calculating the optical flow of the target mesh M1. However, the meshes for which the optical flow is calculated do not necessarily have to be all meshes included in the frame N, and may be only a portion of the meshes included in the frame N.

[0068] For example, the optical flow calculation unit 122 may change the mesh for which the optical flow is calculated depending on the purpose of use of the optical flow. For example, there may be cases where the optical flow of a mesh located at a position lower than a predetermined height is not used. Therefore, the optical flow calculation unit 122 may exclude meshes located at a position lower than a predetermined height from the meshes for which the optical flow is calculated.

[0069] Alternatively, the optical flow calculation unit 122 may increase the error contained in the provisional optical flow as the number of vertices or meshes per unit volume included in frame N decreases. Therefore, the optical flow calculation unit 122 may increase the proportion of meshes for which the optical flow is calculated as the number of vertices or meshes per unit volume included in frame N decreases.

[0070] Fig. 8 is a diagram showing an example of optical flows calculated by the optical flow calculation unit 122. A three-dimensional model F1 is a three-dimensional model in frame N. A three-dimensional model F2 is a three-dimensional model in frame N+1. Referring to Fig. 8, optical flows W1, W7 to W9 show to which meshes included in the three-dimensional model F2 in frame N+1 four meshes included in the three-dimensional model F1 in frame N have moved.

[0071] (Effect position calculation unit 123) The effect position calculation unit 123 acquires information indicating the effect position in frame N. The effect position may be input by the creator or may be stored in advance in the storage unit 150. Then, the effect position calculation unit 123 calculates the destination position of the effect based on statistical processing of the optical flow of each of a plurality of meshes that exist within a predetermined distance from the effect position in frame N.

[0072] This allows the destination position of the effect to be estimated with greater accuracy. Furthermore, by accurately estimating the destination position of the effect, the creator does not need to determine the position of all the effects, which reduces the workload on the creator. In particular, the workload on the creator of making the effect follow the limbs or props can be reduced.

[0073] As described above, in the embodiments of the present disclosure, a case where a process of calculating an average value is used as an example of statistical processing will be mainly described. However, statistical processing is not limited to a process of calculating an average value. For example, statistical processing may be a process of extracting a mode value. Whether the statistical processing is a process of calculating an average value or a process of extracting a mode value may be determined appropriately depending on the characteristics of the volumetric data, etc.

[0074] Fig. 9 is a flowchart showing an example of calculation of the destination position of the effect. Fig. 10 is a diagram for explaining an example of calculation of the destination position of the effect. An example of calculation S20 of the destination position of the effect will be described in detail with reference to Figs. 9 and 10.

[0075] 9, the effect position calculation unit 123 calculates the average value of the optical flow of each of a plurality of meshes that exist within a predetermined distance (defined as Z cm) from the effect position in frame N as the movement vector of the effect. Z cm may be a fixed value or may be variable. For example, Z cm may be changed depending on the size of the body part (e.g., hand or foot) to which the effect is applied.

[0076] Referring to Figure 10, effect E1 in frame N is shown. A set of three-dimensional coordinates that are Z cm away from the position of effect E1 is shown as range Z. Optical flows W1 to W4 are the optical flows of the meshes (fifth planes) that exist within Z cm from the position of effect E1. The effect position calculation unit 123 calculates the average value of optical flows W1 to W4 as a movement vector G1 of the position of effect E1.

[0077] 10, the number of meshes existing within Z cm from the position of effect E1 is four, but the number of meshes existing within Z cm from the position of effect E1 does not have to be four and may be one or more. The effect position calculation unit 123 calculates the destination position of the effect based on the position of effect E1 in frame N and the movement vector G1.

[0078] More specifically, the effect position calculation unit 123 calculates the temporary destination position of the effect by adding the position of the effect E1 in frame N and the movement vector G1 (S22). Then, the effect position calculation unit 123 calculates the position of the mesh that is closest to the calculated temporary destination position of the effect as the effect position in frame N+1 (S23).

[0079] (Effect position suggestion unit 124) The effect position proposal unit 124 proposes an effect position in frame N+1. More specifically, the effect position proposal unit 124 functions as an example of an output control unit that controls the output of information related to the destination position of the effect and frame N+1 by an output unit. This allows the creator, when producing a three-dimensional video, to apply an effect to frame N+1 while referring to information related to the destination position of the effect.

[0080] Fig. 11 is a diagram showing the position of an effect in frame N. Referring to Fig. 11, a three-dimensional model F1 in frame N is shown. In frame N, an effect E1 is applied to a position based on a specific part of the three-dimensional model F1 (here, the foot). Note that, although waves generated by a foot immersed in water are shown as an example of effect E1, effect E1 is not particularly limited as long as it is a visual object.

[0081] FIG. 12 is a diagram showing the destination position of an effect in frame N+1. Referring to FIG. 12, a three-dimensional model F2 in frame N+1 is shown. Frame N+1 is assigned an effect E1 that has been moved to the destination position calculated by the effect position calculation unit 123. As in this example, information about the destination position of the effect may include the effect E1 that has been moved to the destination position in frame N+1. This allows the creator to easily confirm the destination position of the effect.

[0082] It is assumed here that the output unit includes the display unit 130, and the effect position proposal unit 124 controls the display of information regarding the destination position of the effect, frame N+1, and the display unit 130. However, it is also possible that the terminal used by the creator for work and the information processing device 10 are different devices. Therefore, the output unit may include a communication unit, and the effect position proposal unit 124 may control the transmission of information regarding the destination position of the effect and frame N+1 to the terminal by the communication unit.

[0083] (Effect position correction unit 125) The creator applies an effect to frame N+1 while checking the effect position in frame N+1 proposed by the effect position proposal unit 124. For example, if the creator wants to accept the proposed effect position, the creator inputs an operation to confirm the effect position into the operation unit 140. On the other hand, the creator inputs an operation to correct the proposed effect position into the operation unit 140, and then inputs an operation to confirm the effect position into the operation unit 140.

[0084] (Recording control unit 126) Based on the input of an operation to confirm the effect position in frame N+1, the recording control unit 126 controls the recording of the effect position in frame N+1 in the storage unit 150. For example, if the effect position proposed by the effect position proposal unit 124 is modified by the creator, the recording control unit 126 controls the recording of the modified position of the effect in frame N+1 in the storage unit 150 based on the modification made to the proposed effect position.

[0085] The detailed functions of the information processing device 10 according to the embodiment of the present disclosure have been described above.

[0086] <2. Various Modifications> Next, various modified examples of the information processing device 10 according to the embodiment of the present disclosure will be described.

[0087] (First Modification) In the above embodiment, an example has been described in which a creator determines the effect position in frame N+1. Below, a first modification will be described, which is a modification in which the effect position in frame N+1 is automatically determined. For example, this first modification is suitable for cases in which a user views a video to which an effect has been automatically applied.

[0088] Fig. 13 is a diagram illustrating an example of the configuration of an information processing device according to the first modified example. As shown in Fig. 13, the information processing device 20 according to the first modified example is realized by a computer, and includes a control unit 120 and a communication unit 160. The control unit 120 includes a motion capture unit 121, an optical flow calculation unit 122, an effect position calculation unit 123, and a transmission control unit 127.

[0089] (Communication unit 160) The communication unit 160 is configured by a communication interface. For example, the communication unit 160 communicates with a user terminal via a network (not shown).

[0090] (Transmission control unit 127) The transmission control unit 127 adds the effect to the position of the effect destination in frame N+1 calculated by the effect position calculation unit 123. Then, the transmission control unit 127 controls transmission of frame N+1 to which the effect has been added to the user's terminal by the communication unit 160. This allows the user to view a moving image to which an effect that automatically follows the movement of the three-dimensional object has been added.

[0091] (Second Modification) In the above embodiment, an example in which both frame N and frame N+1 are transmitted has been described. Below, a modified example in which frame N+1 is not transmitted, and the optical flow of each mesh in frame N is transmitted will be described as the second modified example. At this time, the receiving side moves each mesh in frame N+1 based on the optical flow of each mesh in frame N and frame N.

[0092] Since the optical flow of each mesh in frame N is expected to have a smaller amount of data than that in frame N+1, this example can reduce the amount of data transmission. For example, this second modified example is suitable for when a user on the receiving side watches a moving image.

[0093] 14 is a diagram illustrating an example of the configuration of an information processing device according to the second modified example. As shown in FIG. 14, the information processing device 30 according to the second modified example is realized by a computer, and includes a control unit 120 and a communication unit 160. The control unit 120 includes a motion capture unit 121, an optical flow calculation unit 122, and a transmission control unit 127.

[0094] (Transmission control unit 127) The transmission control unit 127 controls the transmission of the optical flow of each mesh in frame N and frame N to the user's terminal by the communication unit 160. The user's terminal moves each mesh in frame N+1 based on the optical flow of each mesh in frame N and frame N. This can reduce the amount of data transmission.

[0095] (Third Modification) In the above embodiment, the case where the optical flow is calculated on a mesh basis has been mainly described. However, the optical flow may be calculated on a vertex basis. That is, in the above embodiment, the case where the optical flow is calculated using the three-dimensional coordinates of the mesh and the color information of the mesh has been mainly described. However, the three-dimensional coordinates of the vertices and the color information of the vertices may also be used to calculate the optical flow.

[0096] More specifically, the optical flow calculation unit 122 extracts one or more vertices that satisfy a predetermined relationship with the target vertex (assumed to be C1) from the multiple vertices included in frame N+1 as vertices (third vertices) in the statistical processing range of the target vertex C1. Then, the optical flow calculation unit 122 calculates a provisional optical flow for each vertex in the statistical processing range.

[0097] Here, the vertex that satisfies the predetermined relationship with the target vertex C1 may be a mesh whose distance from the target vertex C1 is smaller than a second threshold. More specifically, the optical flow calculation unit 122 extracts, from the multiple vertices included in frame N+1, one or more vertices whose three-dimensional coordinates are closer to the three-dimensional coordinates of the target vertex C1 than the second threshold, as vertices in the statistical processing range of the target vertex C1. Here, it is assumed that vertices C1 to C18 are extracted as vertices in the statistical processing range of the target vertex C1.

[0098] Next, the optical flow calculation unit 122 extracts one or more vertices having three-dimensional coordinates whose distance from the three-dimensional coordinates of the vertex C1 of the statistical processing range is smaller than a first threshold.The optical flow calculation unit 122 then extracts, from the one or more extracted vertices, the vertex whose difference from the color information of the vertex C1 of the statistical processing range is smallest as a destination candidate vertex of the vertex C1 of the statistical processing range.The optical flow calculation unit 122 calculates a movement vector from the three-dimensional coordinates of the vertex C1 of the statistical processing range to the three-dimensional coordinates of the destination candidate vertex as a tentative optical flow of the vertex C1.

[0099] Similarly, the optical flow calculation unit 122 extracts a destination candidate vertex (fourth vertex) for each of the vertices C2 to C18 in the statistical processing range, and then calculates a provisional optical flow for each of the vertices C2 to C18 in the statistical processing range.

[0100] Next, the optical flow calculation unit 122 calculates the optical flow of the target vertex C1 based on statistical processing of the provisional optical flows of each of the vertices C1 to C18 in the statistical processing range. The optical flow calculation unit 122 calculates the optical flows of all the vertices included in frame N using a method similar to the method for calculating the optical flow of the target vertex C1.

[0101] The effect position calculation unit 123 calculates, as the movement vector of the effect, the average value of the optical flow of each of a plurality of vertices that exist within a predetermined distance from the effect position in frame N. The effect position calculation unit 123 calculates the destination position of the effect based on the position of the effect E1 in frame N and the movement vector of the effect.

[0102] More specifically, the effect position calculation unit 123 calculates the temporary destination position of the effect by adding the position of the effect E1 in frame N and the movement vector. Then, the effect position calculation unit 123 calculates the position of the mesh that is closest to the calculated temporary destination position of the effect as the effect position in frame N+1.

[0103] Various modified examples of the information processing device 10 according to the embodiment of the present disclosure have been described above.

[0104] <3. Hardware configuration example> Next, a hardware configuration example of an information processing device 900 as an example of the information processing device 10 according to an embodiment of the present disclosure will be described with reference to Fig. 15. Fig. 15 is a block diagram showing a hardware configuration example of the information processing device 900. Note that the information processing device 10 does not necessarily have to have all of the hardware configuration shown in Fig. 15, and some of the hardware configuration shown in Fig. 15 may not be present in the information processing device 10.

[0105] 15, the information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 903, and a RAM (Random Access Memory) 905. The information processing device 900 may also include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925. The information processing device 900 may include a processing circuit such as a DSP (Digital Signal Processor) or an ASIC (Application Specific Integrated Circuit) instead of or in addition to the CPU 901.

[0106] The CPU 901 functions as an arithmetic processing unit and control unit, and controls all or part of the operations within the information processing device 900 in accordance with various programs recorded in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 927. The ROM 903 stores programs and calculation parameters used by the CPU 901. The RAM 905 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during the execution. The CPU 901, the ROM 903, and the RAM 905 are interconnected by a host bus 907 constituted by an internal bus such as a CPU bus. Furthermore, the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.

[0107] The input device 915 is a device operated by a user, such as a button. The input device 915 may include a mouse, a keyboard, a touch panel, a switch, a lever, or the like. The input device 915 may also include a microphone that detects the user's voice. The input device 915 may be, for example, a remote control device that uses infrared or other radio waves, or an externally connected device 929, such as a mobile phone, that operates the information processing device 900. The input device 915 includes an input control circuit that generates an input signal based on information input by the user and outputs the signal to the CPU 901. The user operates the input device 915 to input various data and instruct the information processing device 900 to perform processing operations. The imaging device 933, described below, may also function as an input device by capturing images of the user's hand movements, fingers, etc. In this case, the pointing position may be determined based on the hand movements and finger orientations.

[0108] The output device 917 is configured with a device capable of visually or audibly notifying the user of acquired information. The output device 917 may be, for example, a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker or headphones. The output device 917 may also include a PDP (Plasma Display Panel), a projector, a hologram, a printer, or the like. The output device 917 outputs the results obtained by processing by the information processing device 900 as video such as text or images, or as sound such as voice or audio. The output device 917 may also include a light for illuminating the surroundings.

[0109] The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900. The storage device 919 is configured, for example, by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores programs and various data executed by the CPU 901, as well as various data acquired from the outside.

[0110] The drive 921 is a reader / writer for a removable recording medium 927 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is built into or externally attached to the information processing device 900. The drive 921 reads information recorded on the attached removable recording medium 927 and outputs the information to the RAM 905. The drive 921 also writes information to the attached removable recording medium 927.

[0111] The connection port 923 is a port for directly connecting a device to the information processing device 900. The connection port 923 may be, for example, a USB (Universal Serial Bus) port, an IEEE 1394 port, or a SCSI (Small Computer System Interface) port. The connection port 923 may also be an RS-232C port, an optical audio terminal, or an HDMI (registered trademark) (High-Definition Multimedia Interface) port. By connecting an external device 929 to the connection port 923, various types of data can be exchanged between the information processing device 900 and the external device 929.

[0112] The communication device 925 is, for example, a communication interface configured with a communication device for connecting to the network 931. The communication device 925 may be, for example, a communication card for a wired or wireless local area network (LAN), Bluetooth (registered trademark), or wireless USB (WUSB). The communication device 925 may also be a router for optical communication, a router for asymmetric digital subscriber line (ADSL), or a modem for various types of communication. The communication device 925 transmits and receives signals, for example, between the Internet and other communication devices using a predetermined protocol such as TCP / IP. The network 931 connected to the communication device 925 is a network connected by wire or wirelessly, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication.

[0113] <4. Summary> According to an embodiment of the present disclosure, a movement amount calculation unit is provided that calculates a movement amount associated with a first vertex based on statistical processing according to color information associated with a first vertex included in a first frame, color information associated with a second vertex included in a second frame subsequent to the first frame, three-dimensional coordinates associated with the first vertex, and three-dimensional coordinates associated with the second vertex. According to the present invention, an information processing device is provided, and with this configuration, the movement of a three-dimensional model between multiple frames can be estimated with higher accuracy.

[0114] Finally, further differences between the technology according to the embodiment of the present disclosure and other existing technologies will be summarized. First, a first existing technology is the technology described in Japanese Patent Application Laid-Open No. 2020-136943. The first existing technology is a technology that compares 3D models between consecutive frames, links 3D models with similar shapes, associates parts between the frames of the 3D models based on the linking, and estimates the movement of the parts.

[0115] The first existing technology can also estimate the movement of each body part. However, with the first existing technology, the estimation of the movement of each body part depends on the accuracy of capturing the mesh. Therefore, the first existing technology is difficult to apply to objects that deform significantly between successive frames (for example, costumes during dancing). Furthermore, the first existing technology is difficult to apply to cases where there are multiple objects with similar shapes (for example, a large number of balls of the same size rolling around).

[0116] Furthermore, the first existing technology requires the use of other known technologies, which increases the amount of processing and the processing time. Also, the first existing technology requires the use of external libraries, which makes it difficult to implement the functions related to the first existing technology and increases the amount of processing.

[0117] In the first existing technology, the movement of parts of a three-dimensional model is estimated based on shape information using shape fitting, whereas the technology according to an embodiment of the present disclosure estimates the movement of parts of a three-dimensional model based on color information (e.g., hue, etc.).

[0118] Therefore, the first existing technology is difficult to apply to small objects or objects that deform significantly, etc. In contrast, the technology according to the embodiment of the present disclosure is suitable for application to colorful objects that deform significantly (for example, objects that appear in dance, live performances, etc.).

[0119] Furthermore, a second existing technique is described in JP-A-2002-517859, which captures an image of a person with a marker on their face and detects the movement of the facial mesh from the captured image data.

[0120] The second existing technology uses markers on the face, making it difficult to directly capture the 3D data extracted from the captured image data as volumetric content. Therefore, it is necessary to remove the markers before capturing the 3D data as volumetric content. Furthermore, the second existing technology can only recognize mesh movement at positions where markers are attached.

[0121] The technology according to the embodiment of the present disclosure is a markerless technology. Therefore, the technology according to the embodiment of the present disclosure requires less burden during imaging than the second existing technology. In addition, the markerless technology can easily perform processing during generation of volumetric content.

[0122] For example, as a modification of the second existing technology, it may be expected that markers are attached to objects other than faces (for example, props, etc.) and their locations are tracked. However, even in such a modification, the markers must be removed from the 3D model of the object to which the markers are attached, so the technology according to the embodiment of the present disclosure requires lower costs for shooting and generating the 3D model.

[0123] Furthermore, in the second existing technology, it is necessary to clarify what is to be tracked at the time of capturing an image, whereas in the technology according to the embodiment of the present disclosure, it is possible to determine or change the tracking target after capturing an image.

[0124] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0125] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0126] The following configurations also fall within the technical scope of the present disclosure. (1) a movement amount calculation unit that calculates a movement amount associated with a first vertex based on statistical processing according to color information associated with a first vertex included in a first frame, color information associated with a second vertex included in a second frame subsequent to the first frame, three-dimensional coordinates associated with the first vertex, and three-dimensional coordinates associated with the second vertex; Information processing device. (2) the information processing device includes an effect position calculation unit that calculates a destination position of the effect based on a movement amount associated with the first vertex and a position of the effect in the first frame; The information processing device according to (1) above. (3) the information processing device includes an output control unit that controls output of information about a destination position of the effect and the second frame by an output unit; The information processing device according to (2) above. (4) the information about the destination position of the effect includes an effect that has been moved to the destination position of the effect in the second frame; The information processing device according to (3) above. (5) the information processing device includes a recording control unit that controls recording of the modified position of the effect based on the modification of the destination position of the effect; The information processing device according to (4) or (5). (6) the output unit includes a communication unit or a display unit, the output control unit controls the communication unit to transmit information about the destination position of the effect and the second frame or the display unit to display the information. The information processing device according to any one of (3) to (5) above. (7) the information processing device includes a transmission control unit that controls transmission of the first frame and the movement amount associated with the first vertex by a communication unit; The information processing device according to (1) above. (8) The statistical processing is a process of extracting a mode value or a process of calculating an average value. The information processing device according to any one of (1) to (7). (9) each of the color information associated with the first vertex and the color information associated with the second vertex includes a hue; The information processing device according to any one of (1) to (8). (10) the first vertex forms a first face; the second vertex forms a second face; the color information associated with the first vertex is color information of the first face, the color information associated with the second vertex is color information of the second face, the three-dimensional coordinates associated with the first vertex are the three-dimensional coordinates of the first face, the three-dimensional coordinates associated with the second vertex are the three-dimensional coordinates of the second face, The movement amount associated with the first vertex is the movement amount of the first face. The information processing device according to (1) above. (11) The movement amount calculation unit calculating color information of the first face based on color information of the first vertex, and calculating color information of the second face based on color information of the second vertex; calculating three-dimensional coordinates of the first surface based on the three-dimensional coordinates of the first vertex, and calculating three-dimensional coordinates of the second surface based on the three-dimensional coordinates of the second vertex; The information processing device according to (10) above. (12) the movement amount calculation unit extracts, from a plurality of surfaces included in the second frame, one or more surfaces having three-dimensional coordinates whose distance from the three-dimensional coordinates of the first surface is smaller than a first threshold, and extracts, from the one or more extracted surfaces, a surface having the smallest difference from color information of the first surface as the second surface; The information processing device according to (10) or (11). (13) the movement amount calculation unit extracts, from the plurality of surfaces included in the second frame, one or more surfaces having three-dimensional coordinates whose distance from the three-dimensional coordinates of a third surface is smaller than the first threshold value, and extracts, from the one or more extracted surfaces, a surface having the smallest difference from color information of the third surface as a fourth surface; when the distance between the three-dimensional coordinates of the first surface and the three-dimensional coordinates of the third surface is smaller than a second threshold, calculating the amount of movement of the first surface based on statistical processing of the distance between the three-dimensional coordinates of the first surface and the three-dimensional coordinates of the second surface and the distance between the three-dimensional coordinates of the third surface and the three-dimensional coordinates of the fourth surface; The information processing device according to (12) above. (14) the information processing device includes an effect position calculation unit that calculates a destination position of the effect based on statistical processing of a movement amount of the first plane and a movement amount of the fifth plane when the first plane and the fifth plane are present within a predetermined distance from a position of the effect in the first frame; The information processing device according to any one of (10) to (13) above. (15) the color information associated with the first vertex is color information of the first vertex, the color information associated with the second vertex is color information of the second vertex, the three-dimensional coordinates associated with the first vertex are the three-dimensional coordinates of the first vertex, the three-dimensional coordinates associated with the second vertex are the three-dimensional coordinates of the second vertex, The movement amount associated with the first vertex is the movement amount of the first vertex. The information processing device according to (1) above. (16) the movement amount calculation unit extracts, from a plurality of vertices included in the second frame, one or more vertices having three-dimensional coordinates whose distance from the three-dimensional coordinates of the first vertex is smaller than a first threshold, and extracts, from the one or more extracted vertices, a vertex having the smallest difference from color information of the first vertex as the second vertex; The information processing device according to (15) above. (17) the movement amount calculation unit extracts, from a plurality of vertices included in the second frame, one or more vertices having three-dimensional coordinates whose distance from the three-dimensional coordinates of a third vertex is smaller than the first threshold value, and extracts, from the one or more extracted vertices, a vertex whose difference with color information of the third vertex is smallest as a fourth vertex; when the distance between the three-dimensional coordinates of the first vertex and the three-dimensional coordinates of the third vertex is smaller than a second threshold, calculate the amount of movement of the first vertex based on statistical processing of the distance between the three-dimensional coordinates of the first vertex and the three-dimensional coordinates of the second vertex and the distance between the three-dimensional coordinates of the third vertex and the three-dimensional coordinates of the fourth vertex; The information processing device according to (16) above. (18) the information processing device includes an effect position calculation unit that calculates a destination position of the effect based on statistical processing of a movement amount of the first vertex and a movement amount of the fifth vertex when the first vertex and the fifth vertex are present within a predetermined distance from a position of the effect in the first frame; The information processing device according to any one of (15) to (17) above. (19) a processor calculating a movement amount associated with the first vertex based on statistical processing according to color information associated with a first vertex included in a first frame, color information associated with a second vertex included in a second frame subsequent to the first frame, three-dimensional coordinates associated with the first vertex, and three-dimensional coordinates associated with the second vertex; Information processing methods. (20) Computer, a movement amount calculation unit that calculates a movement amount associated with a first vertex based on statistical processing according to color information associated with a first vertex included in a first frame, color information associated with a second vertex included in a second frame subsequent to the first frame, three-dimensional coordinates associated with the first vertex, and three-dimensional coordinates associated with the second vertex; A program that functions as an information processing device. [Explanation of symbols]

[0127] 10, 20, 30 Information processing device 120 control section 121 Motion Capture Department 122 Optical flow calculation unit 123 Effect position calculation unit 124 Effect position suggestion section 125 Effect position correction section 126 Recording control section 127 Transmission control section 130 Display section 140 Operation section 150 Storage section 160 Communications Department

Claims

1. a movement amount calculation unit that calculates a movement amount associated with the first vertex based on statistical processing according to color information associated with a first vertex included in a first frame, color information associated with a second vertex included in a second frame subsequent to the first frame, three-dimensional coordinates associated with the first vertex, and three-dimensional coordinates associated with the second vertex; Information processing device.

2. the information processing device includes an effect position calculation unit that calculates a destination position of the effect based on a movement amount associated with the first vertex and a position of the effect in the first frame; The information processing device according to claim 1 .

3. the information processing device includes an output control unit that controls output of information about a destination position of the effect and the second frame by an output unit; The information processing device according to claim 2 .

4. the information about the destination position of the effect includes an effect that has been moved to the destination position of the effect in the second frame; The information processing device according to claim 3 .

5. the information processing device includes a recording control unit that controls recording of the modified position of the effect based on the modification of the destination position of the effect; The information processing device according to claim 4 .

6. the output unit includes a communication unit or a display unit, the output control unit controls the communication unit to transmit information about a destination position of the effect and the second frame or the display unit to display the information. The information processing device according to claim 3 .

7. the information processing device includes a transmission control unit that controls transmission of the first frame and the movement amount associated with the first vertex by a communication unit; The information processing device according to claim 1 .

8. The statistical processing is a process of extracting a mode value or a process of calculating an average value. The information processing device according to claim 1 .

9. each of the color information associated with the first vertex and the color information associated with the second vertex includes a hue; The information processing device according to claim 1 .

10. the first vertex forms a first face; the second vertex forms a second face; the color information associated with the first vertex is color information of the first face, the color information associated with the second vertex is color information of the second face, the three-dimensional coordinates associated with the first vertex are the three-dimensional coordinates of the first face, the three-dimensional coordinates associated with the second vertex are the three-dimensional coordinates of the second face, the movement amount associated with the first vertex is the movement amount of the first face; The information processing device according to claim 1 .

11. The movement amount calculation unit calculating color information of the first face based on color information of the first vertex, and calculating color information of the second face based on color information of the second vertex; calculating three-dimensional coordinates of the first surface based on the three-dimensional coordinates of the first vertex, and calculating three-dimensional coordinates of the second surface based on the three-dimensional coordinates of the second vertex; The information processing device according to claim 10.

12. the movement amount calculation unit extracts, from a plurality of surfaces included in the second frame, one or more surfaces having three-dimensional coordinates whose distance from the three-dimensional coordinates of the first surface is smaller than a first threshold, and extracts, from the one or more extracted surfaces, a surface having a smallest difference from color information of the first surface as the second surface; The information processing device according to claim 10.

13. the movement amount calculation unit extracts, from the plurality of surfaces included in the second frame, one or more surfaces having three-dimensional coordinates whose distance from the three-dimensional coordinates of a third surface is smaller than the first threshold value, and extracts, from the one or more extracted surfaces, a surface having a smallest difference from color information of the third surface as a fourth surface; when the distance between the three-dimensional coordinates of the first surface and the three-dimensional coordinates of the third surface is smaller than a second threshold, calculating the amount of movement of the first surface based on statistical processing of the distance between the three-dimensional coordinates of the first surface and the three-dimensional coordinates of the second surface and the distance between the three-dimensional coordinates of the third surface and the three-dimensional coordinates of the fourth surface; The information processing device according to claim 12.

14. the information processing device includes an effect position calculation unit that, when the first plane and the fifth plane are present within a predetermined distance from a position of the effect in the first frame, calculates a destination position of the effect based on statistical processing of a movement amount of the first plane and a movement amount of the fifth plane; The information processing device according to claim 10.

15. the color information associated with the first vertex is color information of the first vertex, the color information associated with the second vertex is color information of the second vertex, the three-dimensional coordinates associated with the first vertex are the three-dimensional coordinates of the first vertex, the three-dimensional coordinates associated with the second vertex are the three-dimensional coordinates of the second vertex, the movement amount associated with the first vertex is the movement amount of the first vertex; The information processing device according to claim 1 .

16. the movement amount calculation unit extracts, from a plurality of vertices included in the second frame, one or more vertices having three-dimensional coordinates whose distance from the three-dimensional coordinates of the first vertex is smaller than a first threshold, and extracts, from the one or more extracted vertices, a vertex having a smallest difference from color information of the first vertex as the second vertex; The information processing device according to claim 15.

17. the movement amount calculation unit extracts, from a plurality of vertices included in the second frame, one or more vertices having three-dimensional coordinates whose distance from the three-dimensional coordinates of a third vertex is smaller than the first threshold, and extracts, from the one or more extracted vertices, a vertex whose difference from color information of the third vertex is smallest as a fourth vertex; when the distance between the three-dimensional coordinates of the first vertex and the three-dimensional coordinates of the third vertex is smaller than a second threshold, calculating the amount of movement of the first vertex based on statistical processing of the distance between the three-dimensional coordinates of the first vertex and the three-dimensional coordinates of the second vertex and the distance between the three-dimensional coordinates of the third vertex and the three-dimensional coordinates of the fourth vertex; The information processing device according to claim 16.

18. the information processing device includes an effect position calculation unit that calculates a destination position of the effect based on statistical processing of a movement amount of the first vertex and a movement amount of the fifth vertex when the first vertex and the fifth vertex are present within a predetermined distance from a position of the effect in the first frame; The information processing device according to claim 15.

19. a processor calculating a movement amount associated with the first vertex based on statistical processing according to color information associated with a first vertex included in a first frame, color information associated with a second vertex included in a second frame subsequent to the first frame, three-dimensional coordinates associated with the first vertex, and three-dimensional coordinates associated with the second vertex; Information processing methods.

20. Computer, a movement amount calculation unit that calculates a movement amount associated with the first vertex based on statistical processing according to color information associated with a first vertex included in a first frame, color information associated with a second vertex included in a second frame subsequent to the first frame, three-dimensional coordinates associated with the first vertex, and three-dimensional coordinates associated with the second vertex; A program that functions as an information processing device.

Images