Method for processing data representing a 3D volumetric scene

Abstract

A method for digitally processing data representing a three-dimensional volumetric scene, the present method is implemented by a computer and comprises the following steps: accessing said data representing the scene; specifying a first frame associated with a first mesh (M0); specifying a first position (P) within the first mesh; determining a first side (T) of the first mesh where the first position is located; accessing a list of pre-set associations between the original positions of vertices (V0, V1, V2) of the first side (T) and their respective associated positions (PV0', PV1', PV2') represented by a reference frame that is specified in the second mesh (M1) of the second frame and has a topology different from that of the first mesh (M0), wherein these associations represent the evolution of the scene over time; and determining a position (P') located in the second mesh (M1) associated with a position specified in the first mesh based on their respective associated positions.

Description

Technology Field

[0001] The present invention relates to a method for processing data representing a three-dimensional volumetric scene, that is, including an object extending in three directions of space. Background Technology

[0002] The present invention relates to a method for establishing a continuous link between consecutive frames of volumetric data, that is, a method for representing three-dimensional volumetric scenes, and an application of such method to digital processing of consecutive frames of such volumetric data.

[0003] To the knowledge of the inventors, this subject has not been developed in the field of three-dimensional infographics until now, and no technically satisfactory solution has been known, particularly in the case of sequences with large changes in topology.

[0004] In the case of a two-dimensional video sequence or a 2D video sequence, it is possible to apply effects or filters to a sequence of frames that form the video sequence by a known calculation based on the pixels of a sequence of images.

[0005] This type of application is performed, for example, through "optical flow" technology based on the identification of pixels moving from one frame to the next in a digital video sequence; since this identification can track the movement of objects across consecutive frames, tracking can be performed, and if desired, processing tasks can be automatically applied to each of these frames thanks to object tracking.

[0006] In the case of a three-dimensional volumetric video sequence or a 3D video sequence, the volumetric data does not correspond to pixels but defines a mesh consisting of specific sides for each frame.

[0007] Therefore, methods applicable to 2D video sequences are not applicable to 3D video sequences.

[0008] Therefore, associating the mouth of a moving character in a 3D scene with a sound source requires manually monitoring the movement of the mouth, which is a tedious and time-consuming method that requires manually identifying the position of the mouth for each frame of the scene defined by a topology different from the previous one, resulting in qualitatively unsatisfactory results.

[0009] In addition, there is an approach that detects faces in each frame, but this is not satisfactory when there are multiple faces in the scene.

[0010] Furthermore, this approach applies only to faces and cannot be extended to more general cases.

[0011] The article "Tracking Textured Deformable Objects with Finite-Element Meshes" by Sotiris Malassiotis and Michael G. Strintzis (IEEE Transactions on Circuits and Systems for Video Technology, Vol. 8, No. 6, October 1998) is about an algorithm for estimating the motion of objects subjected to non-stiff deformation.

[0012] The article "Hybrid Human Modeling: Making Volumetric Video Animable" by Peter Eisert and Anna Hilsmann (Computer Vision-ECCV 2020: The 16th European Conference, Glasgow, UK, Springer International Publishers) is about methods for generating hybrid video / animation representations of the human body and face.

[0013] Farrer et al.’s patent US 9,224,245 B2 relates to an animation method that combines a depth map and a two-dimensional image to form a three-dimensional sequence.

[0014] The article by Andreas Schoob et al., "Stereo Vision-Based Tracking of Soft Tissue Movements Applied to Online Resection Control Laser Microsurgery" (Medical Image Analysis 40 (2017), pp. 80–95, Elsevier), concerns a non-rigid tracking method using stereo imaging and its application to laser surgical resection in the field of image-based laser microsurgery.

[0015] The article "High-resolution passive facial expression capture" by Derek Bradley et al. (2010, Vol. 29, No. 4, Article 41 of ACM Transactions on Graphics) is about a method for video capturing human facial expressions.

[0016] According to this article, a single reference mesh is used to uniformly track the geometric structure and texture of the face during a video sequence, serving to generate a set of meshes with the same topology—that is, corresponding vertices and the same connectivity between these vertices.

[0017] In this way, it is possible to propagate from one mesh to another and from one frame to another across the entire sequence, and to modify the geometric structure and texture of the initial frames by the artist.

[0018] However, this method is limited to video sequences described by meshes having the same topology, and cannot be applied to scenes involving changes in the mesh's topology, for example, while a scene of an object not visible in the reference mesh appears.

[0019] Therefore, there is an unmet need for the digital processing of 3D video sequences, more specifically, the digital processing of successive frames of such sequences involving changes in the mesh topology.

[0020] The present invention aims to provide a digital processing method for a three-dimensional volumetric scene that implements time tracking of the location of a point.

[0021] More specifically, the present invention relates to a method for digitally processing data representing a three-dimensional volumetric scene comprising a 3D video sequence having a frame associated with each mesh composed of respective sides, and the method comprises, in a data storage and processing system, the step of evaluating the data representing the three-dimensional volumetric scene, the step of designating a first frame associated with a first mesh of each mesh in the sequence of frames, and the step of designating a first position within the first mesh, the method further comprises the step of determining a first side of the first mesh where the first position is located, in the system accessing a list of pre-set associations between the original position of the vertex of the first side and the respective associated position expressed in a reference frame specified in a second mesh of a second frame of the sequence of frames, wherein these associations represent the development of the scene over time, and the first mesh and the second mesh have different topologies, and the step of determining a position located in the second mesh associated with the position specified in the first mesh from the respective associated position.

[0022] The above method enables the automation of spatial and temporal tracking of points in a three-dimensional volumetric scene modeled by a sequence of frames, each having a mesh associated with it.

[0023] Furthermore, this tracing is general, that is, applicable to any type of visual content including cases of strong topological variations between frames, requires less computing power, and enables real-time tracing and the application of digital processing to these successive frames.

[0024] The method for processing a volumetric video stream according to the present invention may have the following features:

[0025] Each associated location can be defined by the side of the second mesh to which it is located and the centroid coordinates associated with this side, and

[0026] The first frame and the second frame may be two consecutive frames;

[0027] The above method may further include the step of associating the location of the sound source with the location located in the second mesh;

[0028] A frame may belong to at least two blocks of frames having different topologies, and a list may include pre-set associations between the location of each vertex of each frame and the location of neighboring frames, and these associations represent the spatial evolution of scene elements over time, and if two frames under consideration belong to the same frame block, the pre-set associations are defined by an identification operator;

[0029] The method may further include a preliminary step of generating a list of pre-set associations between the original position of a vertex of a first side and each associated position expressed in a reference frame specific to a second mesh, which may include a step of specifying a current vertex of the first mesh; a step of deforming the first mesh to approximate the second mesh; a step of calculating the closest approximate position of the side of the second mesh for a position of the first deformed mesh corresponding to the current vertex; a step of identifying the side where the position of the second mesh is located; and a step of calculating center coordinates that define the approximate position in a reference frame specific to the second mesh.

[0030] The present invention can be extended to the following:

[0031] A data processing system including means for performing steps of a method;

[0032] A computer program comprising instructions that cause the computer to implement steps of a method when the computer program is executed by the computer; and

[0033] A computer-readable medium containing instructions that cause the computer to implement steps of a method when executed by a computer. Brief explanation of the drawing

[0034] The present invention will be better understood by reading the detailed description of the embodiments provided as non-limiting examples and illustrated by the accompanying drawings, and other advantages will become apparent, and the drawings are as follows: FIG. 1a illustrates a videogrammetry studio; FIG. 1b illustrates the modeling of a scene by a mesh; FIG. 1c illustrates a series of frames; FIG. 1d illustrates an approximation of the mesh of the target frame by deformation of the mesh of the source frame; FIG. 2a illustrates a series of frames as well as a bridge according to the present invention; FIG. 2b illustrates a method for calculating the bridge of FIG. 2a; FIG. 3a illustrates a simple application of the bridge of FIG. 2a; FIG. 3b illustrates a general application of the bridge of FIG. 2a; FIG. 4a illustrates a calculation method based on the use of the bridge of FIG. 2a; FIG. 4b illustrates a method for creating a bridge, and Figure 5 illustrates an application for tracking points in consecutive frames. Specific details for implementing the invention

[0035] The volume data that is the subject of the method according to the present invention can be inferred by a set of cameras (130) that captures images of a scene (125) performed by an actor (105) in a video measurement studio (100) at a frequency of generally 30 to 60 times per second, by spatially dispersing the images of the scene (125) as shown in FIG. 1a by video measurement of a three-dimensional scene (125).

[0036] The camera (130) is functionally connected to a data storage and processing system (140) as well as a user interface (150) that enables control of the camera and system, such as a computer equipped with one or more hard disks.

[0037] The above images are used to recreate the action of the scene by calculating a point cloud for each moment of capture.

[0038] Each point cloud is continuously connected to one another and is modeled by a mesh (M) of sides defined by specific points that are their vertices.

[0039] These sides are often triangles, as shown in Fig. 1b, which represents a mesh (M) modeling a human head.

[0040] In the following, this embodiment will be illustrated by a mesh composed of triangles, but other types of sides may also be used.

[0041] The mesh represents the surface of scene elements visible to the camera.

[0042] Each triangle is identified by three vertices (V) defining V0, V1, and V2 respectively in the form of a triplet {V0, V1, V2}, which is the location of each vertex known in a 3D reference frame associated with the scene.

[0043] Each capture moment of a scene corresponds to the frame associated with the volumetric data set composed of the mesh associated with that frame.

[0044] Before processing, the number of triangles used and their connectivity generally differ from frame to frame.

[0045] Typically, to limit the amount of data to be processed, geometric tracing of the mesh is performed over time, consisting of approximating the mesh of a specific frame by a transformation of a reference mesh belonging to a reference frame called a "keyframe," so that the mesh of the following frame is composed of the same triangles with the same connectivity as this reference mesh, and these frames constitute a block of frames.

[0046] Therefore, the mesh of the same frame block has the same topology in terms of composition, that is, it consists of the same triangles with the same connectivity.

[0047] If the mesh of the reference frame cannot be approximated acceptablely by deforming it, define a new reference frame as the new reference mesh, and repeat the operation until the mesh of all frames is processed to define all blocks of the frame.

[0048] The mesh of two consecutive frame blocks, and two blocks that are actually distinct from each other, are given different topologies by structure, that is, they consist of different triangles with different connectivity.

[0049] As described above, the mesh is defined based on the mesh of the keyframe, and therefore, the frame located between two consecutive keyframes is referred to as an "interframe."

[0050] FIG. 1c illustrates a block set (FB) of frames (F) in which each block of the frame is composed of a single key frame (KF) and an interframe (IF).

[0051] Document "High-quality streamable free-viewpoint video" ( ACM Trans. Graphics (SIGGRAPH), 34(4), 2015) provides an overview of the well-known method described above.

[0052] The inventors can see that volume data does not correspond to pixels as in the case of 2D video, but is defined by a mesh consisting of a surface itself defined by triangles having known positions in a coordinate system.

[0053] This embodiment defines an approximate position on the surface of a mesh in a second frame, called a "target frame," for each vertex of a mesh in a first frame, called a "source frame," based on the use of so-called "bridge" elements, and this approximate position is represented by a reference frame specific to the mesh in the target frame.

[0054] These bridges enable time tracking of points defining the mesh and allow filters to be applied to the 3D video sequence.

[0055] A reference frame specific to a given mesh is defined by the identification of the vertices of the components and their respective connectivity to one another.

[0056] The direct result of this definition is that a mesh-specific reference frame allows any point on the surface of the mesh to be easily connected to the coordinates of its vertices. In particular, it can be easily guaranteed that a location point defined by such a specific reference frame is situated on the side of the mesh.

[0057] An example using such a reference frame consists of defining the location of a point in a mesh by the centroid coordinates associated with the side on which the point is located and the vertex defining that side, and these centroid coordinates define the location of the point on the side.

[0058] As described in the present invention, the use of a reference frame specific to each mesh enables the tracking of locations defined by these meshes in a video sequence with simple and efficient use of computing resources.

[0059] It should be noted that a Euclidean reference frame (which defines a position by three coordinates along three axes defining three-dimensional space) is not considered as a reference frame specific to the frame mesh of a video sequence: the point defining the Euclidean reference frame is not defined by the frame mesh, and the Euclidean reference frame is the same for all frames, and therefore, it is not possible to track the position defined by the mesh during a frame using computing resources efficiently.

[0060] In a complete 3D video sequence consisting of a first frame, a last frame, and an intermediate frame formed between these two frames, three types of bridges are defined as empty bridge, delta bridge, and fixed bridge, as illustrated in FIG. 2a.

[0061] FIG. 2a illustrates the entire sequence of frames (F) starting with a start frame (BF) and ending with a end frame (EF), wherein the frames are grouped into at least one block (BL0) and a block (BL1) as defined above, and the block (BL1) comes immediately after the block (BL0) in the direction of the video sequence, and the change from BL0 to BL1 is visualized by a block change line (BC).

[0062] In the direction of the video stream (from BL0 to BL1), the last frame of block (BL0) is frame (F0), the first frame of block (BL1) is frame (F1), and the second frame of block (BL1) immediately following frame (F1) is frame (F2).

[0063] Empty bridges

[0064] As shown in FIG. 2a, the first frame and the last frame of the sequence (BF and EF, respectively) each have no left frame and right frame, that is, no frame facing outward from the sequence.

[0065] In this particular case, the inventors define a so-called empty bridge, which arbitrarily represents the absence of vertex correspondence due to the absence of a frame identified as "Empty" in FIG. 2a.

[0066] In general cases

[0067] For other cases of drawings, the preliminary method (200) is implemented to define an association list between the original position of the vertex of the mesh of the source frame and the respective associated position expressed in a reference frame specific to the mesh of the target frame, as illustrated in (a) of FIG. 2b, and starting with the step (205) of specifying the source frame and the target frame, a test step (T) is implemented to determine whether the source frame and the target frame belong to the same frame block (Y) or two different frame blocks (N).

[0068] If two frames belong to the same frame block, the bridge associated with these two frames will be a delta bridge determined by step (205D), whereas if the source frame and the target frame belong to two different blocks (N) and are consequently associated with two different mesh topologies and two different specific reference frames, the bridge associated with these two frames will be a fixed bridge determined by step (205F).

[0069] Delta Bridge

[0070] Within a given block, as in block (BL1) of FIG. 2a, the mesh of each frame has the same topology, that is, they are composed of the same triangles.

[0071] Therefore, tracking vertices from one frame to another is extremely simple, because it is sufficient to indicate that the vertices of the target frame have the same identifier as the corresponding vertices of the source frame through the delta bridge identified by "delta" in FIG. 2a.

[0072] This operation corresponds to the expression of identity between a given vertex of a frame and the same vertex of an immediately adjacent frame, such as between frames (F1 and F2) of the embodiment shown in FIG. 2a, and this is for each of the vertices of the frame.

[0073] Therefore, a bridge (BR) that tracks the vertices of frame (F1) in frame (F2) 12 ) and a bridge (BR) that tracks the vertices of frame (F2) from frame (F1) 21 Each can be denoted as {Id}, where Id is an identification operator indicating that the identifiers of each vertex in two frames (F1 and F2) are the same.

[0074] Fixed bridge

[0075] The most complex case is when the source frame mesh and the target frame mesh have different topologies; that is, they consist of different numbers of triangles and their connections are also different.

[0076] This situation corresponds to a change from one block to another, such as the change from frame (F0) to frame (F1) in Fig. 2a.

[0077] Tracking of the position between these two frames will be ensured by a fixed bridge identified as "fixed" in FIG. 2a and calculated according to steps (210 to 250) of the method (200), which can be considered as a sub-step of step (205F).

[0078] In practice, the fixed bridge will define an association that enables a change from a defined location within a specific reference frame of the source frame's mesh to a corresponding location within a specific reference frame of the target frame, thereby allowing a point in the scene to be tracked in time even when the topology changes in the mesh representing the scene.

[0079] In step (210), the current vertex (Vi) of frame (F0) is designated as the source frame and frame (F1) is designated as the target frame, and these frames are each exemplified by silhouettes (a) and (e) of FIG. 1d defined by meshes, and (c) shows the superposition of these meshes.

[0080] In a reference frame specific to the target frame (F1), if the vertices (Vi) of the mesh in the source frame (F0) are transformed to follow the overall movement of the scene being displayed, for example, the movement of a character in the scene and / or the position (PVi') corresponding to his movement is to be represented.

[0081] In step (220), the mesh of the source frame is deformed to approximate the mesh of the target frame so that the deformed mesh of the frame (F0) is superimposed on the mesh of the frame (F1).

[0082] FIG. 1d illustrates that in (b), the mesh of the source frame (a) is transformed to approximate the mesh of the target frame (e).

[0083] In addition, Fig. 1d illustrates the superposition of the mesh of (b) and the deformed mesh of (a) in (d).

[0084] A transformation of a mesh maintains its topology, that is, the number of triangles constituting the mesh and their connectivity, but modifies the spatial positions of the vertices of these triangles.

[0085] This step can be performed manually, but preferably, it is performed by applying conventional matching algorithms such as ARAP (As Rigid As Possible) method, ASAP (As Similar As Possible) method, LABF (Linear Angle Based Flattening) method, IC (Inverse Curvature) method, or CP (Curvature Prescription) method, as described in the document ["A local / global approach to mesh parameterization" (Liu et al., Eurographics Symposium on Geometry Processing 2008, Vol. 27 (2008) No. 5)].

[0086] At the end of step (220), the two meshes are spatially very close to each other but have different topologies and vertices.

[0087] The inventors have identified the vertices (Vi) of the mesh of the frame (F0) and the positions (PVi) of the deformed mesh. def ) is considered, which corresponds to vertices (Vi) that have been moved to approximate the mesh of frame (F1) by the mesh deformation of frame (F0).

[0088] Location (Vi def ) is generally not located on the side of the mesh of the frame (F1).

[0089] In step (230), at the location (PVi def For ), the nearest position (PVi') on the side of the mesh of the target frame is calculated by a conventional calculation method.

[0090] In step (240), the side on which the location (PVi'), where the triangle (Tj) is located, is identified by a conventional method as shown in (b) of FIG. 2b.

[0091] The triangle (Tj) is defined by three vertices of the mesh of the frame (F1): V0 (F1), V1 (F1) and V2 (F1).

[0092] In step (250), the centroid coordinates (C0, C1 and C2) representing the position (PVi') of triangle (Tj) are calculated in a conventional manner, and these coordinates are each associated with the vertices (V0(F1), V1(F1) and V2(F1)).

[0093] By doing so, the position (PV') is associated with the vertex (Vi) of the source frame (F0), this position is represented as a specific reference frame in the target frame (F1), and this position (PVi') reflects the movement over time of the volume defined by the mesh of the frames (F0 and F1).

[0094] By applying steps (210 to 250) to each vertex of the mesh of the source frame (F0), a bridge (BR 01 ) is defined as collecting all respective associations between the vertices (V) of the mesh in the source frame (F0) and the triangles (T) of the mesh in the target frame (F1), as well as the centroid coordinates (C0, C1 and C2) that specify the positions of these triangles, and this bridge represents the evolution of the vertex (V) positions over time from frame (F0) to frame (F1).

[0095] Bridge (BR 01Each association listed in ) can be represented as {T, C0, C1, C2}, and, for example, the bridge takes the form of a table that collects each of these associations.

[0096] If normalized centroid coordinates are used and their sum is 1, the bridge (BR 01 The expression of ) can be simplified to {Tj, C0, C1}, and C2 is inferred from C0 and C1.

[0097] In addition, to perform tracking in both directions of time, a bridge (BR 01 In addition to ), this time, by assigning frame (F1) as the source frame and frame (F0) as the target frame, the bridge (BR 10 It is necessary to calculate ).

[0098] Bridge (BR 01 In the step of calculating the bridge (BR), the transformation of the source frame (F0) toward the previously calculated target frame (F1) is used, and the nearest position (PVi') on the side of the mesh of frame (F0) is defined for each vertex of the mesh of frame (F1). 10 The calculation of ) can be optimized.

[0099] However, according to the experience of the inventors, the result is that the deformation of the source frame (F1) passes to the target frame (F0) and the bridge (BR 01 It is less satisfactory than repeating all the aforementioned calculation modes to obtain ).

[0100] As described in detail above, by defining a Delta, Empty, or Fixed bridge for each frame, an association is established between the position of each vertex in each frame and the position of an adjacent frame, and this association represents the spatial evolution of scene elements over time.

[0101] Based on this association, it is possible to track points of scene elements simply and very efficiently in terms of computing resources across the entire 3D video sequence.

[0102] Direct application of the bridge (VTransfer and PTransfer functions, bridge combination)

[0103] For each frame of a 3D video sequence, a bridge as defined above tracks the position of any point on the surface of the mesh of a first frame, called a source frame, which is either before or after the sequence, thereby enabling tracking of the movement of elements or elements modeled by the mesh.

[0104] The inventors intend to discuss associating the location of a source frame with the location of a target frame.

[0105] VTransfer

[0106] A simple application of the bridge consists of associating a vertex of interest (Vint) of position (PVint') in the source frame with a surface of the mesh in the target frame.

[0107] The position (PVint') can be expressed as PVint'=VTransfer(Vint, BR), where VTransfer is a function that associates the position on the surface of the mesh in the target frame with the vertex (Vint) of the source frame by using the bridge (BR) defined above.

[0108] If two frames belong to the same block, VTransfer is defined as the upper position (PVint') of the target frame having the same identifier as Vint through a delta bridge.

[0109] When two frames belong to two different blocks, Vtransfer defines the position (PVint') on the triangle (T') of the target frame's mesh through a fixed bridge as (V0'*C0+V1'*C1+V2'*C2), where V0', V1', and V2' correspond to the vertices of the triangle (T') indicated by the fixed bridge for Vint, and C0, C1, and C2 are normalized centroid coordinates indicated by this bridge, as shown in FIG. 3a.

[0110] In the above notation, Vj*Cj corresponds to the product of the respective coordinates of the vertex (Vj) in the reference frame associated with the scene and the centroid coefficient (Cj), resulting in the respective coordinates of the position (Vint').

[0111] The function VTransfer makes it possible to determine the positions associated with vertices after converting the mesh of the source frame to the target frame, which is actually two consecutive frames of a 3D video sequence.

[0112] PTransfer

[0113] A more general application of the bridge as defined by the VTransfer function is to associate a location of interest (P) belonging to a surface defined by the mesh (M0) of the source frame (F0) with a location (P') specified on a triangle (T') of the surface defined by the mesh (M1) of the target frame (F1), as shown in (a) of FIG. 3b.

[0114] Unlike VTransfer, PTransfer does not require that the location (P) be a vertex of the mesh.

[0115] By the method (400) illustrated in FIG. 3b and FIG. 4a, the position (P') is P'=PTransfer (P BR 01It is expressed as ), where Ptransfer is a function that associates a position on the surface of the target frame's mesh with an arbitrary position (P) on the mesh surface of the source frame, and a bridge (BR) for tracking between frames (F0 and F1) as defined above. 01 Apply ).

[0116] In step (410) of the method (400), the location of interest (P) is represented by the centroid coordinates (C0, C1 and C2) of a triangle (T) containing it in the mesh of the source frame, and this triangle is defined by three vertices (V0, V1 and V2).

[0117] In step (420), the positions of the target frames (PV0', PV1', and PV2') are each calculated by associating them with the three vertices (V0, V1, and V2) of the source frames by the VTransfer function defined above:

[0118] PV0'=VTransfer (V0, BR 01 )

[0119] PV1'=VTransfer (V1, BR 01 )

[0120] PV2'=VTransfer (V2, BR 01 ).

[0121] As illustrated in FIG. 3b (b) and using the same notation used above, in step (430), the centroid coordinates of a position (Q) within a triangle where the vertex is located by three positions (PV0', PV1' and Pv2') are calculated by applying the centroid coordinates (C0, C1 and C2) of the position (P) that defines the position (Q) corresponding to the position (P) in the target frame (F1) to these three positions: PV0'*C0 + PV1'*C1 + PV2'*C2.

[0122] The position (Q) is generally not located on the surface of the mesh (M1) of the target frame (F1).

[0123] In step (440), and similar to step (230) of method (200), as illustrated in (c) of FIG. 3b, a position (P') that is closest to and is the desired position on the surface (Q) of the mesh of the target frame (F1) is determined using a view from a point located on a plane defined by positions (PV1', PV2' and PV3').

[0124] In step (450), and similar to step (240) of method (200), the centroid coordinates (C0', C1' and C2') are calculated to represent the position (P') within the triangle (T') of the frame (M1) where it is located.

[0125] In doing so, the position (P') associated with the position (P) of the source frame (F0) is represented as a specific reference frame in the target frame (F1), and this position takes into account the displacement over time of the entire volume defined by the mesh.

[0126] The application of the PTransfer function can be repeated frame by frame, enabling the tracking of the location of points or elements in the scene throughout the entire 3D video sequence.

[0127] Bridge connection

[0128] In a given 3D video sequence, tracking can be performed on two immediately consecutive frames (Fp and Fq) as described in detail above, but tracking can also be performed when a frame (Fq) is interposed between two non-consecutive frames such as Fp and Fr.

[0129] If there are bridges (Bpq and Bqr) that clarify tracking between frames (Fp and Fq) and between frames (Fq and Fr), respectively, a combination of these bridges can be implemented according to the rules set below to obtain a bridge (Bpr) that allows tracking between discontinuous frames (Fp and Fr).

[0130] Combining an empty bridge with any delta or fixed bridge creates an empty bridge.

[0131] Combining a Delta Bridge with another Delta Bridge creates a Delta Bridge.

[0132] Combining a delta bridge and a fixed bridge creates the same fixed bridge.

[0133] The three aforementioned combination rules are commutative in an algebraic sense, that is, in this order, the combination of bridge (Ba) and bridge (Bb) is strictly equivalent to the combination of bridge (Bb) and bridge (Ba).

[0134] If two bridges are fixed bridges, and the bridges (BRpq and BRqr) are combined in this order as defined above, i.e., BRpq*BRqr, then the bridge (Bpr) is obtained as calculated below, and this rule is not commutative.

[0135] We want to express the position that a vertex (V) of a mesh in a source frame (Fp) will have when it moves in accordance with the overall movement of the scene represented by the mesh cell of the frame, in a reference frame specific to the target frame (Fr).

[0136] To this end, the aforementioned function VTransfer (V, BRpq) is applied to the corresponding vertex (V) of the mesh in frame (Fp) to obtain the position (V') on the surface of the mesh in frame (Fq), and then the aforementioned function PTransfer (V', BRqr) is applied to this position (V') to obtain the position (V") on the surface as shown in FIG. 4b.

[0137] By repeating this operation at each vertex of the mesh of the frame (Fp), a bridge (BRpr) can be constructed as part of the bridge (BRpq and BRqr).

[0138] The application of the bridge combination changes a sequence of 60 frames per second to 30 frames per second (or fps), and in this operation, every other frame is removed.

[0139] In this case, calculating a new set of bridges for a 30 fps sequence can be easily done from the bridges already calculated for a 60 fps sequence: from the two bridges calculated for tracking over three consecutive frames (BRpq and BRqr), a single bridge (BRpr) is calculated for tracking between the first and third frames, and the intermediate frames are removed according to the combination procedure described above.

[0140] The aforementioned bridge combination operator consequently provides a new bridge, and because this operator is associative, it can optimize tracking for one or more points throughout the sequence due to known approaches based on prior calculation, such as parallel summation or other similar approaches.

[0141] Furthermore, it is understood that if the necessary bridges are calculated and an optimization approach like the one cited above is followed, it is possible to track in real time the point where most of the necessary calculations have already been performed.

[0142] specific application

[0143] Generally, any application requiring tracking the location of points on the surface of a mesh in a 3D video sequence can benefit from the bridge and function described above.

[0144] This is, for example, the case of arbitrary graphic processing on a frame of this sequence intended to be applied to another frame of the sequence.

[0145] A first embodiment of the tracking application is audio tracking, and it is sought to associate the 3D location of an audio source in a 3D video scene over time, such as when an actor speaks while moving in a scene.

[0146] In the context of virtual reality applications, it is important that this audio source is consistently placed in space and follows the actor's movements.

[0147] In this example, the operator selects a frame and manually defines the position of the audio source from the actor's mouth, and then automatically tracks this position over time by recursively applying the PTransfer function from this manually defined position to each frame.

[0148] More specifically, data representing a 3D video scene (or a three-dimensional volumetric scene), comprising a sequence of consecutive frames each associated with a mesh representing the scene at a given moment, is stored in a data storage and processing system (140).

[0149] The tracking method illustrated in the diagram (500) of FIG. 5 is implemented by a data storage and processing system (140) and includes the step (510) of accessing data of the data storage system (140) and the step (520) of designating a first frame associated with a first mesh by a human operator in a sequence of consecutive frames.

[0150] In this first frame, the human operator designates a first position corresponding to the mouth of the actor in the scene during the first mesh and during the designating step (530).

[0151] Next, in step (540), the system (140) determines a first side of a first mesh where a first position is located by means of a conventional calculation method performed by a data storage and processing system (140) that also performs calculations of subsequent operations.

[0152] The first position can be expressed as the centroid coordinates within the reference frame formed by the vertex of the first side.

[0153] Before the start of the tracking method, a list of associations is defined between the location of each vertex of each frame and the locations of adjacent frames, and these associations represent the spatial evolution of the scene elements over time, and these associations can be formed by the bridge defined above and can be stored in a data storage and processing system (140).

[0154] In step (550), access to this pre-set association list indicates a specific association between the position of the vertex of the first side and the respective associated position expressed in a specific reference frame of the second mesh of the second frame of the sequence.

[0155] From each associated position, in step (560), a position located in the second mesh is determined, which is associated with a position specified in the first mesh and can be expressed as the centroid coordinates of the side of the second mesh to which it belongs.

[0156] In a specific case of audio tracking, the sound source is additionally associated with the position of the second mesh associated with the position of the first mesh in step (570).

[0157] The two steps (550 and 560) can be implemented, for example, by the Ptransfer function defined above.

[0158] By repeating steps (540 to 570) in frame units, starting for each repetition from the position determined in the previous repetition step (560) instead of the first position, a step (580) of automatically tracking a position manually defined by a human operator and associating it with the 3D position of the audio source can be performed.

[0159] Note that the first frame may be arbitrarily selected at the beginning, middle, or end of the video sequence, the second frame may be located at a time before or after the first frame, and a pre-set association may associate the location independently of the order of the frames being considered.

[0160] Additionally, the first frame and the second frame may be continuous or discontinuous, that is, separated by one or more other frames, and the association between the position of each vertex of each frame and the position of an adjacent frame makes it possible to establish an association between discontinuous frames by, for example, a combination of bridges as described above.

[0161] A second embodiment of the application is a geometric anti-vibration filter that tracks the position of a point in a scene in a given sequence as described in detail above and then compares it to adjust the mesh of the frame around the averaged position to eliminate or reduce vibration at this point.

[0162] Unlike the first embodiment, the step (530) of specifying a point in an image is automated and supported by the data storage and processing system (140), which can be configured to specify, for example, a set of vertices of a mesh of a given frame consecutively.

[0163] Of course, the application of the embodiments disclosed in this specification is not limited to the embodiments cited above, and those skilled in the art will immediately understand that any processing of a 3D video sequence involving tracking a point of a scene over space and time can benefit from the present invention.

[0164] Although for the sake of brevity in the description of the method according to the present invention, the examples used process a sequence of consecutive frames such as frames (F0, F1 and F2), the present invention is not limited to consecutive frames, particularly while defining a bridge, but extends to the definition of a bridge between two distinct frames of one or more different frames in a video sequence.

[0165] The present invention is not limited to the embodiments disclosed above, and modifications are possible within the scope of the invention.

Claims

Claim 1 A method (500) for digitally processing data comprising a 3D video sequence that represents a three-dimensional volumetric scene, is derived from a video measurement capture of the three-dimensional scene (125), is configured laterally, and has frames (F0, F1, F2) each associated with a respective mesh representing volumetric data of the three-dimensional scene at a given moment, wherein the method is implemented by a computer and comprises the following steps: accessing the data representing the three-dimensional volumetric scene (125) in a data storage and processing system (140) (510); and specifying a first frame associated with a first mesh (M0) of each mesh in the sequence of frames (520). The method comprises the step (530) of specifying a first position (P) within the first mesh (M0), and the method further comprises the following steps: determining a first side (T) of the first mesh (M0) where the first position is located (540); and in the system (140), accessing a pre-set (fixed) list of associations (550) between the original positions of the vertices (V0, V1, V2) of the first side (T) and their respective associated positions (PV0', PV1', PV2') represented by a specific reference frame in the second mesh (M1) of the second frame of the sequence of frames, wherein these associations represent the evolution of the scene over time and the first mesh (M0) and the second mesh (M1) have different topologies (550). A method further comprising the step (560) of determining a position (P') located in the second mesh (M1) associated with a designated position in the first mesh from each of the above associated positions. Claim 2 A method according to claim 1, wherein each of the respective associated positions is defined by the associated side of the second mesh on which it is located and the centroid coordinates associated with the associated side. Claim 3 A method according to claim 1 or 2, wherein the first frame and the second frame are two consecutive frames. Claim 4 A method according to claim 1 or 2, further comprising the step (570) of associating the location of a sound source with the location of the location within the second mesh. Claim 5 A method according to claim 1 or 2, wherein the frames (F0, F1, F2) belong to at least two blocks (BL0, BL1) of frames (F) having different topologies, and the list includes pre-set associations between the locations of each vertex of each frame and the locations of adjacent frames, and these associations represent the spatial evolution of elements of the scene over time, and when the two frames considered (F1, F2) belong to the same block (BL1) of frames, the pre-set associations are defined by an identification operator (Delta). Claim 6 In claim 1 or 2, the method further comprises a preliminary step of generating a list of pre-set associations (Fix) between the original positions of the vertices (V0, V1, V2) of the first side (T) and their respective associated positions (PV0', PV1', PV2') expressed in the reference frame specific to the second mesh, and the following steps: a step (210) of specifying a current vertex (Vi) of the first mesh (M0); a step (220) of modifying the first mesh (M0) to be close to the second mesh (M1); and a position (PVi) of the first modified mesh corresponding to the current vertex (Vi). def A method comprising: a step (230) of calculating the closest approximate location (PVi') to the side (Tj) of the second mesh; a step (240) of identifying the side where the approximate location (PVi') is located; and a step (250) of calculating centroid coordinates (C0, C1, C2) that define the approximate location (PVi') within the reference frame specific to the second mesh (M1). Claim 7 A data processing system comprising means for implementing the steps of the method according to paragraph 1 or 2. Claim 8 A computer program stored on a computer-readable recording medium, comprising instructions that allow the computer to implement the steps of the method according to claim 1 or 2 when the program is executed by the computer. Claim 9 A computer-readable medium comprising instructions that allow the computer to implement the steps of the method according to claim 1 or 2 when executed by a computer.

Images