Method for processing data representing a three-dimensional volumetric scene - Patents.com

Abstract

1. A computer-implemented method for digitally processing data representing a three-dimensional volumetric scene, the method comprising the steps of: accessing the data representing the scene; specifying a first frame associated with a first mesh (M0); specifying a first position (P) within the first mesh; determining a first facet (T) of the first mesh in which the first position is located; accessing a list of pre-established associations between original positions of vertices (V0, V1, V2) of the first facet (T) and respective associated positions (PV0', PV1', PV2') represented in a reference frame specific to a second mesh (M1) in a second frame and having a different topology than the first mesh (M0), wherein these associations represent the evolution of the scene over time; and determining a position (P') located in the second mesh (M1) that is associated with the specified position in the first mesh based on the respective associated positions.

Description

[Technical field]

[0001] The present invention relates to a method for processing data representing a three-dimensional volumetric scene, i.e. data containing objects extending in three directions in space. [Background technology]

[0002] The present invention relates to a method for establishing a contiguous link between successive frames of volumetric data, i.e. for representing a three-dimensional volumetric scene, and to the application of this method to the digital processing of successive frames of these volumetric data.

[0003] To the best of the inventors' knowledge, this subject has not been developed so far in the field of three-dimensional infographics and no technically satisfactory solutions are known, especially in the case of sequences with large changes in topology.

[0004] In the case of a two-dimensional or 2D video sequence, it is possible to apply effects or filters to successive frames forming the video sequence by known calculations based on the pixels of the successive images of the frames.

[0005] This kind of application is, for example, carried out by the "optical flow" technique, based on the identity of pixels from one frame of a digital video sequence to the next, which makes it possible to track the movement of an object in successive frames and thus to carry out said tracking and, if desired, to automatically apply processing to each of these frames thanks to the tracking of the object.

[0006] In the case of three-dimensional volumetric or 3D video sequences, the volumetric data does not correspond to pixels, but defines a mesh made up of facets that are unique to each frame.

[0007] Therefore, methods that are applicable to 2D video sequences are not applicable to 3D video sequences.

[0008] Therefore, to associate a sound source with the mouth of a person moving in a three-dimensional scene, the mouth movements must be monitored manually, a tedious and time-consuming method that involves manually identifying the mouth position for each frame of a scene defined by a topology different from the previous one, and which produces qualitatively less than satisfactory results.

[0009] There are approaches that rely on face detection in each frame, but this is not satisfactory when there are several faces in the scene.

[0010] Furthermore, these approaches only work for faces and cannot be extended to the more general case.

[0011] The paper by Sotiris Malassiotis and Michael G. Strintzis, "Tracking Textured Deformable Objects Using a Finite-Element Mesh," IEEE transactions on circuits and Systems for video technology, Vol. 8, No. 6 (October 1998), is about algorithms for estimating the motion of objects undergoing non-rigid deformations.

[0012] Paper by Peter Eisert and Anna Hilsmann, "Hybrid Human Modeling: Making Volumetric Video Animatable," Computer Vision-ECCV 2020:16 th European Conference, Glasgow, UK, Springer International Publishing, on methods for creating hybrid video / animation representations of the human body and face.

[0013] US Patent No. 9,224,245 (B2) to Farrer et al. relates to an animation method that combines depth maps and two-dimensional images to form a three-dimensional sequence.

[0014] The article by Andreas Schoob et al., “Stereo vision-based tracking of soft tissue motion with application to online ablation control laser microsurgery”, Medical Image Analysis 40 (2017), pp. 80-95, Elsevier, relates to a method for non-rigid tracking by stereo imaging and its application to laser surgical ablation in the field of image-guided laser microsurgery.

[0015] The paper by Derek Bradley et al., "High Resolution Passive Facial Performance Capture," ACM Transactions on Graphics, volume 29, number 4, item 41 (2010), concerns a method for video capture of human facial expressions.

[0016] According to this paper, in order to facilitate uniform tracking of facial geometry and texture during a video sequence, a single reference mesh is used, which serves to generate a set of meshes of the same topology, i.e., having corresponding vertices and the same connectivity between these vertices.

[0017] In this way, modifications made by an artist to the geometry and textures of a first frame can be propagated from one mesh to another, and from one frame to another for the entire sequence.

[0018] However, this method is limited to video sequences described by meshes with the same topology and cannot be applied to scenes with changes in the topology of the mesh, for example during which an object that is not visible in the reference mesh appears in the scene.

[0019] There is therefore an unmet need for digital processing of 3D video sequences, and more specifically, for digital processing of successive frames of such sequences with changing mesh topology. DISCLOSURE OF THEINVENTION

[0020] The invention aims to provide a method for digital processing of three-dimensional volumetric scenes implementing time tracking of point positions.

[0021] More specifically, the invention relates to a method for digitally processing data representing a three-dimensional volumetric scene including a 3D video sequence having frames each associated with a respective mesh consisting of facets and representing volumetric data of the scene at a given instant in time, the method being implemented by a computer and comprising the steps of: accessing in a data storage and processing system said data representing the three-dimensional volumetric scene; specifying in a sequence of frames a first frame associated with a first mesh of said respective meshes; specifying a first position in the first mesh, the method further comprising the steps of determining a first facet of the first mesh in which the first position is located; accessing in the system a list of pre-established associations between original positions of vertices of the first facet and the respective associated positions represented in a reference frame specific to a second mesh of a second frame of the sequence of frames, these associations representing the evolution of the scene over time, the first mesh and the second mesh having different topologies; and determining from the respective associated positions a position located in the second mesh associated to the specified position in the first mesh.

[0022] The above method makes it possible to automate the tracking in space and time of points of a three-dimensional volumetric scene modelled by a sequence of frames, each of which has a respective mesh associated with it.

[0023] In addition, the tracking is general, i.e. applicable to any type of visual content, including cases with strong topological variations between frames, and has low computational requirements, making it possible to perform real-time tracking and apply digital processing to the sequence of frames.

[0024] A method for processing a volumetric video stream according to the present invention comprises the steps of: each respective associated location can be defined by the belonging facet of the second mesh in which it is located and by the barycentric coordinates associated with this belonging facet; the first frame and the second frame may be two consecutive frames; The method may further comprise a step of associating the position of the sound source with a position located in the second mesh; the frame may belong to at least two blocks of frames having different topologies, the list includes pre-established associations between the position of each vertex of each frame and its position in the adjacent frames, these associations expressing the evolution in the space of the elements of the scene over time, the characteristic that when the two considered frames belong to the same block of frames, the pre-established associations are defined by an identity operator; - the method may further include a preliminary step of generating a list of pre-established associations between original positions of the vertices of the first facet and their respective associated positions represented in a reference frame specific to the second mesh, the preliminary step may include the steps of specifying a current vertex of the first mesh, deforming the first mesh to approximate the second mesh, calculating, for the position of the deformed first mesh corresponding to the current vertex, an approximate position closest to a facet of the second mesh, identifying the facet in which the position of the second mesh is located, and calculating barycentric coordinates defining the approximate position in the reference frame specific to the second mesh.

[0025] The present invention relates to - a data processing system including means for performing the steps of the method; - a computer program comprising instructions which, when the program is executed by a computer, perform the steps of a method; The present invention may extend to a computer-readable medium containing instructions that, when executed by a computer, perform the steps of a method. [Brief description of the drawings]

[0026] The invention will be better understood and other advantages will become apparent on reading the detailed description of embodiments thereof, illustrated by way of non-limiting example and by the appended drawings, in which: [Figure 1A] Shown is a videogrammetry studio. [Figure 1B] 1 shows modeling of a scene using meshes. [Figure 1C] A series of frames is shown. [Figure 1D] 4 illustrates the approximation of a mesh in a target frame by transforming a mesh in a source frame. [Figure 2A] 1 shows a series of frames and bridges according to the present invention. [Figure 2B] 2B shows a method for calculating the bridge of FIG. 2A. [Figure 3A] 2B shows a simple application of the bridge of FIG. 2A. [Figure 3B] 2B illustrates a general application of the bridge of FIG. 2A. [Figure 4A] 2B shows a calculation method based on the use of the bridge of FIG. 2A. [Figure 4B] Here's how to create a bridge. [Diagram 5] We demonstrate an application for tracking points in successive frames. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] The volumetric data that is the subject of the method according to the invention can be estimated, for example, from a videogrammetric capture of a three-dimensional scene 125 as shown in FIG. 1A, i.e. by a set of spatially distributed cameras 130 capturing images of the scene 125 acted out by actors 105 in a videogrammetry studio 100, typically at a frequency of around 30-60 images per second.

[0028] The camera 130 is operatively connected to a data storage and processing system 140, and to a user interface 150 allowing control of the system, such as a computer equipped with the camera and one or more hard disks.

[0029] The images are used to recreate the action of the scene by computing a point cloud for each instant of capture.

[0030] Each point cloud is then modeled by a mesh M of facets that are continuously connected to each other and defined by the particular points that are their vertices.

[0031] These facets are often triangles, such as those shown in FIG. 1B, which represents a mesh M that models a human head.

[0032] In the following, this embodiment is illustrated by a mesh of triangles, although other types of facets can also be used.

[0033] A mesh represents the surface of a scene element as seen by the camera.

[0034] Each triangle is identified by the three vertices V that define it, i.e., V0, V1 and V2 respectively, in the form of a triplet {V0, V1, V2}, and the position of each vertex is known in a three-dimensional reference frame associated with the scene.

[0035] Each instant of the capture of the scene corresponds to a frame associated with a volumetric dataset consisting of the mesh associated with this frame.

[0036] Before processing, the number of triangles used and their connectivity generally varies from frame to frame.

[0037] Conventionally, in order to limit the amount of data to be processed, a geometric tracking of the mesh is performed over time, which consists of approximating the mesh of a particular frame by deformation of a reference mesh belonging to a reference frame called a "key frame", so that the mesh of subsequent frames consists of the same triangles with the same connectivity as this reference mesh, these frames constituting a block of frames.

[0038] Thus, meshes of the same block of a frame have, by construction, the same topology, i.e., consist of the same triangles with the same connectivity.

[0039] If the mesh cannot be acceptably approximated by deforming the mesh of the reference frame, a new reference mesh is defined using the new reference frame and the operation is repeated until the meshes of all frames have been processed, thus defining all blocks of the frames.

[0040] The meshes of two consecutive blocks of the frame, in fact two separate blocks, are by construction given different topologies, i.e. they consist of different triangles with different connectivity.

[0041] A frame whose mesh is defined as described herein above based on the mesh of a keyframe and which is located between two consecutive keyframes is called an "interframe."

[0042] FIG. 1C shows a set of blocks FB of a frame F, where each block of the frame consists of a single key frame KF and an inter frame IF.

[0043] The paper “High-quality streamable free-viewpoint video”, ACM Trans. Graphics (SIGGRAPH), 34(4) (2015) gives an overview of the known methods mentioned above.

[0044] It can be seen that volumetric data does not correspond to pixels, as in the case of 2D video, but is defined by a mesh made up of surfaces, the surfaces themselves being defined by triangles whose vertices have known positioning in a coordinate system.

[0045] This embodiment is based on the use of so-called "bridge" elements, which define, for each vertex of a mesh in a first frame, called the "source frame", an approximate position on the surface of the mesh in a second frame, called the "target frame", where this approximate position is expressed in a reference frame specific to the mesh of the target frame.

[0046] These bridges make it possible to perform time tracking of the points that define the mesh and to apply filters to the 3D video sequence.

[0047] The unique reference frame for a given mesh is defined by the identity of its component vertices and their respective connectivity with respect to one another.

[0048] A direct consequence of this definition is that a reference frame specific to a mesh makes it easy to connect any point of the surface of this mesh to the coordinates of its vertices. In particular, it is easy to guarantee that a position point defined by this unique reference frame lies on a facet of the mesh.

[0049] An example of the use of such a reference frame is in defining the position of a point of a mesh by the facet on which it lies and the barycentric coordinates associated with the vertices that define this facet, these barycentric coordinates defining the position of the point on the facet.

[0050] The use of a unique reference frame for each mesh as described in this invention allows tracking of positions defined by these meshes within a video sequence with a simple and efficient use of computational resources.

[0051] It should be noted that the Euclidean reference frame (which defines positions by three coordinates along three axes that define a three-dimensional space) is not considered a reference frame specific to the mesh of a frame of a video sequence: the points that define the Euclidean reference frame are not defined by the mesh of the frame, the Euclidean reference frame is the same for all frames, and therefore, in an efficient use of computational resources, it is not possible to track positions defined by the mesh during a frame.

[0052] In a complete 3D video sequence, using the first frame, the last frame, and the intermediate frames contained between these two frames, three types of bridges are defined, namely, empty bridge, delta bridge, and fixed bridge, as shown in FIG. 2A.

[0053] FIG. 2A shows a complete sequence of frames F starting with a start frame BF and ending with an end frame EF, the frames being grouped into at least one block BL0 and the above defined block BL1, which block BL1 immediately follows block BL0 in the direction of the video sequence, and the transition from BL0 to BL1 being visualized by a block change line BC.

[0054] In the direction of the video stream (from BL0 to BL1), the last frame of block BL0 is frame F0, the first frame of block BL1 is frame F1, and the second frame of block BL1 immediately after frame F1 is frame F2.

[0055] Sky Bridge The first and last frames of the sequence, BF and EF respectively, have no left and right frames, ie frames towards the outside of the sequence, as shown in FIG. 2A.

[0056] In these particular cases, we define so-called empty bridges, which arbitrarily indicate the absence of vertex correspondence due to the absence of a frame, identified by "empty" in FIG. 2A.

[0057] In general For the other cases in the figure, a preliminary method 200 is performed to define a list of associations between the original positions of the vertices of the mesh of the source frame and their respective associated positions represented in a reference frame specific to the mesh of the target frame, as shown in (a) of FIG. 2B, starting with a step 205 of specifying the source frame and the target frame, followed by a test step T of determining whether the source frame and the target frame belong to the same frame block (Y) or to two separate frame blocks (N).

[0058] If the two frames belong to the same frame block, the bridge associated with these two frames is a delta bridge determined by step 205D, whereas if the source frame and the target frame belong to two different blocks (N) and are therefore associated with two different mesh topologies and two different unique reference frames, the bridge associated with these two frames is a fix bridge determined in step 205F.

[0059] Delta Bridge Within a given block, such as block BL1 in FIG. 2A, the mesh in each frame has the same topology, i.e., is formed from the same triangles.

[0060] Thus, with the delta bridge identified by "Delta" in FIG. 2A, tracking vertices from one frame to another is quite simple, as it is sufficient to indicate that a vertex in the target frame has the same identifier as the corresponding vertex in the source frame.

[0061] This operation corresponds to the expression of identity between a given vertex of a frame and the same vertex in an immediately adjacent frame, such as between frames F1 and F2 in the example shown in Figure 2A, for each of the vertices of the frame.

[0062] Therefore, a bridge BR that tracks the vertices of frame F1 in frame F2 12 and a bridge BR that tracks the vertices of frame F2 within frame F1. 21 can be written as {Id}, respectively, where Id identifies the identity operator that indicates that the respective identifiers of the vertices are the same for the two frames F1 and F2.

[0063] Fixbridge The most complicated case is when the meshes in the source and target frames have different topologies, i.e. they contain different numbers of triangles, the connections of which are also different.

[0064] This situation corresponds to a change from one block to another, such as from frame F0 to frame F1 in FIG. 2A.

[0065] The tracking of the position between these two frames is ensured by a fix bridge, identified in FIG. 2A by "fix" and calculated according to steps 210-250 of method 200, which can be considered as sub-steps of step 205F.

[0066] In effect, the fixbridge defines associations that allow changing from defined positions in a reference frame specific to the mesh of the source frame to corresponding positions in a reference frame specific to the target frame, thus making it possible to track points of the scene in time even when the topology in the mesh representing the scene is changed.

[0067] In step 210, the current vertex Vi in frame F0 is designated as the source frame and frame F1 is designated as the target frame, which are illustrated by silhouettes (a) and (e) in FIG. 1D defined by their respective meshes, and (c) shows the superposition of these meshes.

[0068] In a reference frame specific to the target frame F1, it is sought to represent the positions PVi' that the vertices Vi of the mesh of the source frame F0 would have when deformed to follow the overall movement of the depicted scene, e.g., corresponding to the movement of people in the scene and / or their movements.

[0069] In step 220, the mesh of the source frame is deformed to approximate the mesh of the target frame by tending to create a superimposed deformed mesh of frame F0 onto the mesh of frame F1.

[0070] FIG. 1D shows, in (b), a mesh from a source frame (a) being deformed to approximate that of a mesh in a target frame (e).

[0071] In addition, FIG. 1D shows the superposition, at (d), of the deformed mesh in (a) with the mesh in (b).

[0072] The deformation of a mesh preserves its topology, ie the number of triangles that compose it and their connectivity, but modifies the positions in space of the vertices of these triangles.

[0073] This step can be performed manually, but is preferably performed by applying a conventional matching algorithm such as the As Rigid As Possible (ARAP), As Similar As Possible (ASAP), Linear Angle Based Flattening (LABF), Inverse Curvature (IC) or Curvature Prescription (CP) methods as described in Liu et al., “A local / global approach to mesh parameterization,” Eurographics Symposium on Geometry Processing 2008, Vol. 27, No. 5 (2008).

[0074] At the end of step 220, the two meshes are very close to each other in space, but have different topologies and vertices.

[0075] Vertex Vi of the mesh in frame F0 and the position PVi of the deformed mesh corresponding to the vertex Vi moved by the deformation of the mesh in frame F0 to approximate the mesh in frame F1. def Think about it.

[0076] Vi def The positions are generally not located on facets of the mesh of frame F1.

[0077] In step 230, the position PVi def For ,the closest position PVi' on a facet of the mesh of the target frame is calculated by conventional computational methods.

[0078] In step 240, the facet in which the position PVi' lies, here triangle Tj, is identified by conventional methods, as shown in FIG. 2B(b).

[0079] A triangle Tj is defined by three vertices V0(F1), V1(F1) and V2(F1) of the mesh of frame F1.

[0080] In step 250, barycentric coordinates C0, C1 and C2 representing a position PVi' within triangle Tj are calculated in a conventional manner, and these coordinates are associated with vertices V0(F1), V1(F1) and V2(F1), respectively.

[0081] In doing so, a position PV' is associated with a vertex Vi in the source frame F0, this position is expressed in a reference frame specific to the target frame F1, and this position PVi' reflects the temporal movement of the volume defined by the meshes of frames F0 and F1.

[0082] By applying steps 210 to 250 to each of the vertices of the mesh in the source frame F0, a bridge BR 01 is defined as the collection of all respective associations of vertices V of the mesh in source frame F0 to triangles T of the mesh in target frame F1, as well as the barycentric coordinates C0, C1, and C2 that identify their positions within these triangles, and this bridge represents the evolution of the position of vertex V over time from frame F0 to frame F1.

[0083] Bridge BR 01 Each of the associations listed in can be represented by {T, C0, C1, C2} and the bridge takes the form of, for example, a table that collects each of these associations.

[0084] If normalized barycentric coordinates are used and their sum is 1, then the bridge BR 01 can be simplified to {Tj,C0,C1}, where C2 is deduced from C0 and C1.

[0085] In order to perform tracking in both time directions, a bridge BR 01 Not only that, but this time we assign frame F1 as the source frame and frame F0 as the target frame, so that we can create a bridge BR 10 needs to be calculated.

[0086] Bridge BR 10 Calculation of bridge BR 01 It is possible to optimize the mesh in frame F0 by using the transformation of the source frame F0 towards the target frame F1 previously calculated during the step of calculating the mesh in frame F1 and defining for each vertex of the mesh in frame F1 the closest position PVi' on a facet of the mesh in frame F0.

[0087] However, in our experience, the result is that the source frame F1 passes through the transformation to the target frame F0 and the bridge BR 01 This is less satisfactory than repeating all the above modes of calculation to obtain

[0088] By defining a delta bridge, null bridge or fixed bridge for each frame as detailed above, associations are established between the position of each vertex in each frame and positions in adjacent frames, and these associations represent the evolution in space of the scene elements over time.

[0089] Based on these associations, it is possible to track scene element points simply and very efficiently in terms of computing resources throughout a 3D video sequence.

[0090] Direct application of bridges (VTransfer and PTransfer functions, combined bridges) The bridge defined above for each frame of a 3D video sequence makes it possible to track the position of any point on the surface of the mesh of the first frame, called the source frame, either before or after in the sequence, so as to track the movement of one or more elements modelled by the mesh.

[0091] It talks about relating the location of a source frame to the location of a target frame.

[0092] VTransfer A simple application of a bridge is to associate a vertex of interest Vint in the source frame at a position PVint' with the surface of a mesh in the target frame.

[0093] The PVint' position can be expressed by PVint' = VTransfer(Vint,BR), where VTransfer is a function that relates a vertex Vint of the source frame to a position on the surface of the mesh of the target frame by using the bridge BR as defined above.

[0094] If the two frames belong to one and the same block, VTransfer defines, by the delta bridge, the position PVint' as the position of the beginning of the target frame with the same identifier as Vint.

[0095] If the two frames belong to two separate blocks, VTransfer transfers the position PVint' of the target frame's mesh triangle T' to (V0' * C0+V1' * C1+V2' * C2), where V0', V1', and V2' correspond to the vertices of the triangle T' indicated by the fix bridge to Vint, and C0, C1, and C2 are the normalized barycentric coordinates indicated by this bridge, as shown in FIG. 3A.

[0096] In the above notation, Vj * Cj corresponds to multiplying each of the coordinates of the vertex Vj in the reference frame associated with the scene by the barycentric coefficient Cj, resulting in each of the coordinates of the position Vint'.

[0097] The function VTransfer makes it possible to determine the positions associated with the vertices after transforming a mesh of a source frame into a target frame, in fact into two consecutive frames in a 3D video sequence.

[0098] PTransfer A more general application of a bridge as defined by the VTransfer function is the association of a position of interest P belonging to a surface defined by mesh M0 of source frame F0 with a position P' located on triangle T' of the surface defined by mesh M1 of target frame F1, as shown in (a) of Figure 3B.

[0099] Unlike VTransfer, Ptransfer does not require that the position P be the position of a vertex of a mesh.

[0100] According to the method 400 shown in FIG. 3B and FIG. 4A, the position P′ is calculated as P′=PTransfer(PBR 01 ), where PTransfer is the bridge BR for tracking between frames F0 and F1, as defined above. 01 is a function that relates a position on the surface of the mesh in the target frame to an arbitrary position P on the surface of the mesh in the source frame by applying

[0101] In step 410 of the method 400, a position of interest P is represented by the barycentric coordinates C0, C1 and C2 of a triangle T that contains it in the mesh of the source frame, the triangle being defined by its three vertices V0, V1 and V2.

[0102] In step 420, the positions PV0', PV1' and PV2' of the target frame are calculated and associated with the three vertices V0, V1 and V2 of the source frame, respectively, and are calculated by the VTransfer function defined above: PV0' = VTransfer(V0,BR 01 ) PV1' = VTransfer(V1,BR 01 ) PV2' = VTransfer(V2,BR 01 ).

[0103] In step 430, the barycentric coordinates of a position Q in a triangle whose vertices are located by the three positions PV0', PV1', and PV2' are calculated by applying to these three positions the barycentric coordinates C0, C1, and C2 of a position P that defines a position Q corresponding to P in the target frame F1: PV0' * C0+PV1' * C1+PV2' * C2, shown in FIG. 3B(b) and having the same notation as used above.

[0104] Position Q generally does not lie on the surface of mesh M1 of object frame F1.

[0105] In step 440, similar to step 230 of method 200, as shown in (c) of FIG. 3B, the position on the surface of the mesh of the target frame F1 that is closest to Q and is the desired position P' is determined from the point located in the plane defined by positions PV1', PV2' and PV3'.

[0106] In step 450, similar to step 240 of method 200, barycentric coordinates C0', C1', and C2' are calculated, representing the position P' within the triangle T' of frame M1 in which it is located.

[0107] In doing so, a position P' associated with a position P in the source frame F0 is expressed in a reference frame specific to the target frame F1, and this position takes into account the displacement over time of the entire volume defined by the mesh.

[0108] The application of the PTransfer function can be repeated for every frame, allowing tracking of the position of a scene point or element throughout the entire 3D video sequence.

[0109] Combination bridge For a given 3D video sequence, it is possible to perform tracking on two directly successive frames Fp and Fq, as explained in detail above, but it is also possible to perform tracking between two non-successive frames, such as Fp and Fr, with frame Fq being inserted between these two frames.

[0110] If there are bridges Bpq and Bqr that clarify tracking between frames Fp and Fq, and between frames Fq and Fr, respectively, it is possible to perform a combination between these bridges according to the rules set out below to obtain a bridge Bpr that enables tracking between non-consecutive frames Fp and Fr.

[0111] Combining an air bridge with any delta bridge or fixed bridge results in an air bridge.

[0112] Combining a delta bridge with a delta bridge gives you a delta bridge.

[0113] Combining a delta bridge with a fixed bridge results in this same fixed bridge.

[0114] The three combination rules above are commutative in the algebraic sense, ie, the combination of bridge Ba with bridge Bb in this order is strictly equivalent to the combination of bridge Bb with bridge Ba in this order.

[0115] A combination of bridges BRpq and BRqr as defined above in this order, where the two bridges are fixed bridges, i.e. BRpq * BRqr gives the bridge Bpr as computed below, and this rule is not commutative.

[0116] In a reference frame specific to the target frame Fr, it is sought to represent the position that a vertex V of the mesh of the source frame Fp would have if it were moved to follow the global movement of the scene represented by the mesh cells of the frame.

[0117] For this purpose, the above-mentioned function VTransfer(V,BRpq) is applied to this vertex V of the mesh of frame Fp to obtain a position V' on the surface of the mesh of frame Fq, and then the above-mentioned function PTransfer(V',BRqr) is applied to this position V' to obtain a position V'' on the surface, as shown in FIG. 4B.

[0118] By repeating this operation for each vertex of the mesh of the frame Fp, the bridge BRpr can be constructed as part of the bridges BRpq and BRqr.

[0119] One application of the bridge combination is to change a 60 frames per second sequence to 30 frames per second (or fps), an operation in which every other frame is removed.

[0120] In this case, the calculation of a new set of bridges for a sequence at 30 fps is easily done from the bridges already calculated for the 60 fps sequence: from two bridges calculated for tracking over three consecutive frames (BRpq and BRqr), a single bridge (BRpr) is calculated for tracking between the first and third frames, and the intermediate frames are removed exactly according to the combination procedure described above.

[0121] The above mentioned bridge combination operator results in a new bridge, and since this operator is associative, it is possible to optimize the tracking of one or several points over the entire sequence, thanks to known approaches based on pre-computation, e.g. parallel addition or another similar approach.

[0122] It will also be appreciated that once the required bridges have been calculated, by following an optimization approach such as that described above, it is possible to track in real time the point where most of the required calculations have already been performed.

[0123] Specific application In general, any application that requires tracking the positions of points on the surface of a mesh within a 3D video sequence can benefit from the bridges and functions detailed above.

[0124] This is the case, for example, of any graphic processing of a frame of a sequence that is intended to be applied to other frames of this sequence.

[0125] A first example of a tracking application is that of audio tracking, where it is desired to correlate the 3D position of an audio source in a 3D video scene over time, such as in the case of an actor talking while moving in the scene.

[0126] In the context of virtual reality applications, it is important that this audio source is coherently positioned in space and follows the movements of the actor.

[0127] In this example, the operator selects frames in which the position of the audio source is manually defined at the actor's mouth, and then tracking of this position over time is performed automatically by recursively applying the PTransfer function from this manually defined position frame by frame.

[0128] More specifically, data representing a 3D video scene (or three-dimensional volumetric scene), including a sequence of successive frames each associated with a respective mesh representing the scene at a given moment, is stored in the data storage and processing system 140.

[0129] The tracking method shown in diagram 500 of FIG. 5 is performed by the data storage and processing system 140 and includes step 510 of accessing data in the data storage system (140) and step 520 of specifying, by a human operator, a first frame in a sequence of consecutive frames that is associated with a first mesh.

[0130] In this first frame, the human operator specifies, in a first mesh, during a specifying step 530, a first position that corresponds to the mouth of an actor in the scene.

[0131] In step 540, the system 140 determines, by conventional computational means performed by the data storage and processing system 140, which also performs the computations of the following operations, the first facet of the first mesh in which the first position is located.

[0132] The first position may be expressed in barycentric coordinates within a reference frame formed by the vertices of the first facet.

[0133] Prior to the start of the tracking method, a list of associations between the position of each vertex in each frame and positions in adjacent frames is defined, these associations representing the evolution in space of the scene elements over time, these associations can be formed by the bridges defined above, and are stored in the data storage and processing system 140.

[0134] In step 550, this list of pre-established associations is accessed, indicating, in particular, the associations between the positions of the vertices of the first facet and their respective associated positions represented in a reference frame specific to the second mesh of the second frame of the sequence.

[0135] From each associated position, in step 560, a position located in the second mesh is determined, which is associated with the specified position in the first mesh and can be represented by the barycentric coordinates of the facet of the second mesh to which it belongs.

[0136] In the particular case of audio tracking, the sound source is further associated in step 570 with a position in a second mesh that is associated with the position of the first mesh.

[0137] The two steps 550 and 560 can for example be implemented by the Ptransfer function defined above.

[0138] By repeating steps 540-570 frame by frame, starting for each iteration from the position determined in step 560 of the previous iteration instead of the first position, a step 580 can be performed to automatically track the position manually defined by a human operator and relate it to the 3D position of the sound source.

[0139] It should be noted that the first frame can be arbitrarily selected at the beginning, middle or end of a video sequence, and the second frame can be located in time before or after the first frame, with the pre-established association making it possible to associate positions independently of the order of the frames considered.

[0140] Furthermore, the first and second frames may be consecutive or discontinuous, i.e. separated by one or more other frames, and the association between the position of each vertex in each frame and the position in the adjacent frame makes it possible to establish an association between the discontinuous frames, for example by a combination of the bridges described above.

[0141] A second application example is a geometric anti-shake filter, where the position of a point in the scene is tracked in a given sequence as described above, then compared and the mesh of the frame is adjusted around the averaged position to eliminate or reduce vibrations from this point.

[0142] Unlike the first example, the step 530 of specifying points in the image is automated and supported by the data storage and processing system 140, which can be configured, for example, to sequentially specify a set of vertices of a mesh for a given frame.

[0143] Of course, the application of the embodiments disclosed herein is not limited to the examples cited above, and those skilled in the art will immediately appreciate that any processing of 3D video sequences involving tracking of points of a scene in space and over time can benefit from the present invention.

[0144] To simplify the description of the method according to the invention, the example used processes a sequence of consecutive frames such as frames F0, F1, and F2, however the invention is not limited to consecutive frames, particularly during the definition of a bridge, but extends to the definition of a bridge between two distinct frames of one or more other frames in a video sequence.

[0145] It goes without saying that the invention is not limited to the embodiments disclosed above, which can be modified without departing from the scope of the invention.

Claims

1. A method (500) for digitally processing data representing a three-dimensional volumetric scene and including a 3D video sequence derived from a videogrammetric capture of a three-dimensional scene (125) and having frames (F0, F1, F2) each associated with a respective mesh comprising facets and representing volumetric data of said three-dimensional scene at a particular time, said method being implemented by a computer and comprising: - accessing (510) said data representing a three-dimensional volumetric scene (125) in a data storage and processing system (140); - designating (520) in said sequence of frames a first frame associated with a first mesh (M0) of said respective meshes; - specifying (530) a first position (P) in said first mesh (M0), The method comprises: - determining (540) a first facet (T) of said first mesh (M0) in which said first location is located; - accessing (550) in said system (140) a list of pre-established associations (fixes) between original positions of vertices (V0, V1, V2) of said first facet (T) and their respective associated positions (PV0', PV1', PV2') expressed in a reference frame specific to a second mesh (M1) of a second frame of said sequence of frames, said associations representing the evolution of said three-dimensional volumetric scene over time, said first mesh (M0) and said second mesh (M1) having different topologies; - determining (560) from each associated position a position (P') located in the second mesh (M1) associated with the specified position in the first mesh.

2. The method of claim 1 , wherein each of said respective associated locations is defined by the belonging facet of said second mesh in which it is located and barycentric coordinates associated with said belonging facet.

3. The method of claim 1 , wherein the first frame and the second frame are two consecutive frames.

4. The method of claim 1 , further comprising the step of associating (570) a location of a sound source with the location located within the second mesh.

5. 2. The method of claim 1, wherein the frames (F0, F1, F2) belong to at least two blocks (BL0, BL1) of a frame (F) having different topologies, the list comprises pre-established associations between the position of each vertex of each frame and positions in adjacent frames, these associations representing the evolution in space of elements of the three-dimensional volumetric scene over time, and wherein when the two frames (F1, F2) considered belong to the same block of frames (BL1), the pre-established associations are defined by an identity operator (delta).

6. The method further comprises a preliminary step of generating a list of said pre-established associations (fixes) between said original positions of said vertices (V0, V1, V2) of said first facet (T) and said respective associated positions (PV0', PV1', PV2') expressed in said reference frame specific to a second mesh, said preliminary step comprising: - a step (210) of designating a current vertex (Vi) of said first mesh (M0); - a step (220) of deforming said first mesh (M0) so as to approximate said second mesh (M1); the position (PVi) of the deformed first mesh corresponding to the current vertex (Vi); ｄｅｆ ) calculating (230) the approximate position (PVi') closest to the facet (Tj) of said second mesh; - identifying (240) the facet in which the approximate position (PVi') is located; - calculating (250) barycentric coordinates (C0, C1, C2) defining said approximate position (PVi') in said reference frame specific to said second mesh (M1).

7. A data processing system comprising means for carrying out the steps of the method according to any one of claims 1 to 6.

8. A computer program comprising instructions which, when said program is executed by a computer, enable said computer to carry out the steps of the method according to any one of claims 1 to 6.

9. A computer readable medium comprising instructions which, when executed by a computer, enable the computer to perform the steps of the method according to any one of claims 1 to 6.

Images