Method and apparatus for compressing data representing volumetric 3D scenes using views for real-time decompression for online viewing - Patents.com

Abstract

A method (300) for compressing a volumetric video stream (172) including a plurality of frames, wherein groups of frames are defined, each frame including a key frame and an inter frame having a mesh defined relative to a mesh of a corresponding key frame, comprising: compressing information representing a texture, the step including, for each group of frames, block-compressing (302) the information representing the texture of each of the frames, thereby defining blocks of pixels; determining (304) a first block that can form a synthetic texture of the key frames and a second block that can modify the synthetic texture through iterative substitution to form an approximation inter frame texture; and encoding (310) the inter frame texture based on the synthetic texture of the key frames and the second block.

Description

[Technical field]

[0001] The present invention relates to a method and apparatus for compressing data representing a volumetric three-dimensional scene for real-time decoding for online viewing of the volumetric video by an end-user. [Background technology]

[0002] An object of the present invention is to compress volumetric data representing a three-dimensional scene for "online" viewing by an end user.

[0003] A method aimed at this goal is described in the paper “High-quality streamable free-viewpoint video”, ACM Trans. Graphics (SIGGRAPH), 34 (4), 2015.

[0004] According to the method, volumetric data are obtained from the capture of a three-dimensional scene by photogrammetry, i.e. using a set of cameras, in this case 106 cameras, each capturing images of the scene from a specific angle with a frequency of around 30-60 images per second.

[0005] The images are used to recreate the action of the scene by computing a point cloud for each instant of the capture of the scene or frame.

[0006] The point cloud represents the surfaces of the elements of the scene as seen by the camera and serves as the basis for modeling the scene by a mesh of continuously connected triangles, on top of which textures are applied in a first step independently for each frame.

[0007] In this step, the number of triangles used and their connectivity varies from frame to frame.

[0008] In a second step, in order to favor the compression of the resulting data, a geometric tracking of the mesh is performed over time, which consists in converging the meshes that constitute the frame by deformation of a reference mesh belonging to a frame called a "key frame", so that the mesh of the subsequent frame has the same number of triangles and the same connectivity as the reference mesh.

[0009] If deformation of the mesh in the reference frame does not allow the mesh to be acceptably approximated, a new mesh is defined for the new reference frame and the operation is repeated until the meshes in all frames have been processed.

[0010] A frame whose mesh is defined based on the mesh of a keyframe and which lies between two keyframes is called an "interframe".

[0011] Compression of mesh-related data benefits from temporal redundancy between adjacent frames where the mesh is based on the mesh of the same keyframe.

[0012] The meshes of the key frames are fully encoded, but only the mesh variations are encoded for interframes, which is more economical in terms of the amount of resulting data and the computational effort for their decoding, as well as the full encoding and decoding of the meshes for each frame.

[0013] However, the encoding of the mesh data can also be improved.

[0014] For textures compressed onto meshes, a complete atlas of textures corresponds to each frame, and this atlas must be fully encoded and then decoded during compression and decompression, respectively, according to the principles of the MPEG and H.264 compression standards.

[0015] The processing and transmission of such texture atlas data represents a high bandwidth and computational cost that is desirable to reduce.

[0016] Furthermore, the methods used were originally developed for creating videos with "traditional" images, such as film images, and not for creating videos with textures, which are images consisting of patches on a solid background.

[0017] A better method for compressing texture atlases is needed.

[0018] In the field of audio / video compression, the following documents are known: -”[V-PCC][EE2.6-related]Proposition of an anchor and a test model for coding animated meshes”by Jean-Eudes Marvie et al.,132.MPEG Conference, 20201012-20201016 online;(MOTION PICTURE EXPERT GROUP or ISO / IEC JTC1 / SC29 / WG11), no.m55327, 5 October 2020 (2020-10-05), XP030292836; Faramarzi Esmaeil et al., "Mesh Coding Extensions to MPEG-I V-PCC," 2020 IEEE 22nd INTERNATIONAL WORKSHOP ON MULTIMEDIA SIGNAL PROCESSING (MMSP), [Online], September 21, 2020 (2020-09-21), pages 1-5, XP055837185, and -Tang Danhang et al., "Real-time compression and streaming of 4D performances," ACM TRANSACTIONS ON GRAPHICS, ACM, NY, US, vol. 37, no. 6, 4 December 2018 (2018-12-04), pages 1-11, XP058464802.

[0019] These documents deal with compression methods, typically MPEG, JPEG, HEVC (High Efficiency Video Coding) or AVC type (Advanced Video Coding) methods such as h.264.

[0020] This type of method of temporal video compression, in turn, relies on the definition of blocks by cutting the image, a comparison between the blocks of two adjacent frames, and then encoding / compressing the difference between the two blocks; during decoding, this difference is applied to each pixel individually.

[0021] The objective of this type of method is to provide the lowest possible transmission speed over the computer network, without taking into account the transfer of images between the computer's processor and the graphics card that processes these image data streams, or the utilization of the graphics card's memory.

[0022] Therefore, this type of method does not respond to the problem addressed by the present patent application. Summary of the Invention

[0023] The object of the invention is, firstly, to improve the compression of information relating to the meshes of a frame of a volumetric video stream, and secondly, to improve the compression of information relating to the textures associated with said same frames.

[0024] The invention more particularly relates to a method for compressing a volumetric video stream of a three-dimensional action scene represented by a plurality of frames, the method being implemented by a computer, in which for each frame of the scene a mesh and a texture are generated, groups of frames are defined, each comprising a key frame and an inter frame, the mesh being defined relative to the mesh of a corresponding key frame, the method comprising a step of compressing information representing the texture, the texture compression step compressing, for each group of frames, the information representing the texture for each frame of the group according to a block compression method capable of forming blocks directly usable by a conventional graphics processing unit according to a standard algorithm such as DXT, ATTC or PVRTC, thus defining blocks of pixels directly usable by a conventional graphics processing unit. , a step of determining from the blocks associated with the key frames and the blocks associated with each of the inter frames, on the one hand, first blocks from which a synthetic texture of the key frames can be formed and, on the other hand, second blocks from which the synthetic texture of the key frames can be modified by iterative replacement of blocks so as to form an approximation texture of the inter frames; a step of forming a synthetic texture of the key frames of a group of frames from the first blocks associated with the key frames and the first blocks associated with the inter frames and encoding the synthetic texture; and a step of encoding a texture of the inter frames based on the synthetic texture of the key frames and the second blocks.

[0025] The compression of information representing texture according to the invention exploits in particular the temporal correspondence that exists between the textures of successive frames forming the volumetric video stream, thereby avoiding the encoding, transmission and subsequent decoding of redundant information.

[0026] Decompression of a video stream encoded by the compression method according to the invention has low computational cost, the compression level can be easily selected by simple parameters, making it possible to adapt a compromise between compression level and ease of decompression to the user's requirements, thus making it possible to obtain a volumetric video stream that is particularly suitable for reading in real time.

[0027] The method for compressing a volumetric video stream according to the invention has the following characteristics: The step of encoding the keyframe synthetic texture and the interframe texture may comprise compression by entropy coding, the step of determining the first and second blocks may comprise a step of evaluating a quantized graphic difference of the blocks of the group of frames at a given position relative to each other, said step of evaluating being dependent on a calculation such as a PSNR (Peak Signal to Noise Ratio) calculation between the two considered blocks and may result in a number each representative of the graphic variation between the two considered blocks, For each of the blocks of the group of frames, a graphical difference is evaluated relative to all of the other blocks of the group of frames; The step of determining the first and second blocks may further comprise the steps of constructing a graph with nodes interconnected along rows and columns, the quantized graphical difference being assigned a cost for a displacement along the row between two nodes and a cost for a displacement along the column; and determining a minimum cost path in the graph, each path starting in a column associated with one of the first blocks and each displacement following a column associated with one of the second blocks, the minimum cost being the sum of the cost of a displacement between two nodes in the same row and the cost of a displacement between two nodes in one same column, the path being composed of displacements between two nodes in the same row and displacements between two nodes in the same column; The step of determining the route may implement a Dijkstra algorithm. The method may further include the steps of compressing information representing the mesh, compressing the information representing the mesh by quantization, and compressing the quantized information representing the mesh according to a compression method, compressing information representing triangles of the mesh of a key frame according to an Edgebreaker algorithm, compressing information representing points of the mesh of a key frame according to a prediction algorithm, compressing information representing texture coordinates of the key frame according to a linear selective prediction algorithm, compressing information representing points of the mesh of an interframe mesh by differential encoding, and compressing the compressed information representing the points, the triangles and the texture coordinates of the mesh of a key frame and the information representing the points of the mesh of an interframe mesh by an entropy calculation algorithm.

[0028] The present invention can be expanded as follows: - a data processing system including means for performing the steps of the method; a computer program comprising instructions which, when executed by a computer, cause the computer to perform steps of a method; A computer-readable medium containing instructions that, when executed by a computer, cause the computer to perform the steps of a method. [Brief description of the drawings]

[0029] The invention will be better understood and other advantages will become apparent on reading the detailed description of embodiments thereof taken as non-limiting examples and illustrated by the accompanying drawings, in which:

[0030] [Figure 1A] FIG. 1 illustrates a videogrammetry studio including a camera. [Figure 1B] FIG. 1B is a diagram of the image capture and data processing system of the camera of FIG. 1A. [Figure 1C]FIG. 1 illustrates a method for generating a volumetric video of a scene. [Figure 1D] FIG. 1D illustrates a point cloud defined by the method of FIG. 1C. [Figure 1E] FIG. 1E shows a mesh corresponding to the point cloud of FIG. 1D. [Figure 2A] A series of frames is shown. [Figure 2B] FIG. 2B shows the distribution of the frames of FIG. 2A in a group. [Figure 2C] FIG. [Figure 2D] FIG. 2 shows (a) a complete texture of a key frame and (b) a partial texture of an inter frame according to the present invention. [Figure 3A] FIG. 2 is a diagram of a method according to the invention. [Figure 3B] 1 is a diagram of certain steps of the method according to the invention; [Figure 4A] 1 is a table of error values. [Figure 4B] 4B is a graph constructed based on the table of FIG. 4A. [Figure 5A] 4C is a particular case of the graph of FIG. 4B after processing according to the invention. [Figure 5B] 5B is a table summarizing the results of FIG. 5A. [Figure 5C] FIG. 2 illustrates uncompressed and compressed textures for a series of frames. [Figure 6] FIG. 2 illustrates decompression of a compressed volumetric video data stream according to the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] To reconstruct a volumetric action scene, i.e. a scene that takes place in three directions in space over time, the scene is played out by an actor 105 in a videogrammetry studio 100 with a green background.

[0032] Such a studio consists of a structure 110 surrounding a stage 120, this structure serving to support a set of cameras 130 observing a scene 125 according to different viewpoints.

[0033] The camera 130 is connected to a data storage and processing system 140 and to a user interface 150 .

[0034] A method for generating volumetric video using studio 100 includes the following steps of diagram 155 of FIG. 1C.

[0035] In step 160, the cameras capture the scene in a synchronized manner, for example at a frequency of 30-60 images per second, each from its own viewpoint, making it possible to reconstruct the entire scene.

[0036] In step 162, the image captured by the camera is optionally processed for calibration, to correct for bias or other errors, and to subtract background therefrom, according to conventional methods.

[0037] In step 164, these reprocessed images are fed to an algorithm, known to those skilled in the art, implemented by the data processing system 140 to generate a point cloud C, as shown by FIG. 1D, representing the objects that make up the scene by comparison of images of the same physical surface captured by different cameras at different viewing angles, establishing a depth map of the visible surfaces of said objects according to the stereoscopic photography principle.

[0038] In step 166, a mesh M is generated from the resulting point cloud, as shown by FIG. 1E, and textures are intended to be applied to it, respectively and independently for each frame, according to conventional methods, for example by use of Poisson surface reconstruction and the UVAtlas source code, copyright® Microsoft Corporation.

[0039] At this stage, the scene is represented by a series of frames F, each frame associated independently for each frame with information representative of a respective texture T on the one hand and with information representative of a respective mesh M on the other hand.

[0040] FIG. 2A thus shows five frames F1 to F5 in succession, each associated with a mesh M1 to M5 and a texture T1 to T5, respectively.

[0041] Each mesh consists of a set of points interconnected by triangles that reproduce the surface of an object in the scene, and each triangle is associated with texture coordinates by a texture coordinate table so that textures can be correctly applied to the triangles.

[0042] The triangles are used as primitive surfaces that make up the surface of the object as seen by the camera.

[0043] Other types of elementary surfaces may be used, but for clarity of explanation we will continue with the exemplary implementation of the invention that is based on triangles.

[0044] A geometric and photometric tracking step 168 is performed to distribute the frame F into groups of frames FGr, each of which contains a key frame KF and a number of inter-frames IF.

[0045] Thus, FIG. 2B shows 18 frames F1 to F18 distributed into three groups of frames FGr1 to FGr3, each group containing a key frame KF and a variable number of Inter frames depending on the group, and the frames are obtained in stream 172 in an output step 170.

[0046] A reference mesh is associated with each keyframe, interframe meshes are defined by a transformation of the reference mesh, i.e. by information representing the transformation of the mesh associated with the corresponding keyframe, and a texture is associated with each frame.

[0047] Each texture may be represented by an image containing patches on a uniform background, as shown in FIG. 2C.

[0048] In the case of a keyframe, the information representing the mesh consists of information representing the points of the mesh, i.e., data for positioning those points in three-dimensional space, information representing the triangles of the mesh, i.e., surface data each of which is contained between three points of the mesh, and information representing the texture associated with each triangle of the mesh, i.e., texture coordinate data.

[0049] The above steps are known to those skilled in the art and are described, for example, in the publication “High-quality streamable free-viewpoint video”, ACM Trans. Graphics (SIGGRAPH), 34(4), 2015.

[0050] The following steps provide improvements over conventional volumetric video compression methods, with regard to the compression of information representing textures on the one hand and meshes on the other hand, taking into account the subsequent decompression steps, as detailed below using diagrams 300 and 304 in Figures 3A and 3B, respectively. Texture Compression

[0051] At the end of the preceding steps, each frame is associated with its own texture in the stream of video data 172, which includes texture data T and mesh data M, each texture being a digital image.

[0052] In a test step 301 it is determined whether the input data is mesh data M or texture data T.

[0053] In step 302, block compression is applied to each texture associated with a frame of the stream of frames, and the textures are looked up in the texture data T such that the texture of each frame is associated with a set of blocks.

[0054] Block compression is a conventional compression method that allows reducing the amount of memory required to store color data, where blocks of pixels (such as 4x4 or 8x8 squares) are compressed according to standard algorithms such as DXT, ASTC or PVRTC, taking into account that the variation within one and the same block is very low.

[0055] In this specification, the term "block" refers to a block obtained by an algorithm of the type described in the previous paragraph and that can be directly used by a conventional graphics processing unit or GPU, i.e., without requiring decompression or further processing operations.

[0056] The "blocks" considered in the method according to the invention are therefore different from the "blocks" used in compression methods such as the JPEG and MPEG encoding compression methods AVC (Advanced Video Coding) mentioned above in the "Prior Art" section.

[0057] In fact, the blocks used in AVC type methods result directly from cuts in the image and are therefore each a group of pixels.

[0058] Their properties are therefore different from those of the blocks of the method according to the invention, which are the result of a compression operation applied to the pixel in question, and therefore their properties depend on the compression principle used in the compression operation, but not on the compression principle of a group of pixels.

[0059] One feature of AVC type method blocks is that, unlike the method blocks according to the invention, they are not typically considered to be directly usable by a graphics processing unit.

[0060] In fact, block compression according to the present invention results in data that can be directly decoded by a conventional graphics processing unit.

[0061] Thus, during decoding, the central processing unit of the data processing unit, i.e. the CPU, does not need to decompress the block-compressed texture before sending it to the graphics processing unit, i.e. the GPU, which makes it possible to limit the calculations performed on the CPU and significantly reduce the amount of data sent to the GPU.

[0062] In the context of real-time readout of volumetric video streams, this feature represents a decisive advantage by reducing the amount of data to be transferred and the computational costs of decompressing the transferred data.

[0063] Therefore, a strong advantage of the method according to the invention compared to conventional methods is that it exploits the power of the GPU to perform decompression operations on blocks in order to facilitate the processing of the resulting video stream.

[0064] Furthermore, block compression within the meaning of the invention is particularly suitable for compressing textures, in particular atlas type textures, and is much more suitable than image compression such as those used in JPEG and MPEG encoding.

[0065] However, blocks that are directly interpretable by a GPU, as considered in the present invention, are difficult to compress due to the fact that it is not possible to modify the blocks by differentials.

[0066] Therefore, conventional techniques that rely on differential compression are not applicable.

[0067] For each group of frames, a texture reduction method is applied during step 304 according to the following principle, called partial texture reduction.

[0068] In this step, there is a first set of first blocks (210) from which the keyframe synthetic texture can be formed, from among the blocks associated with the keyframe and the blocks associated with the interframes of the group of frames, and a second set of second blocks (220) from which the keyframe synthetic texture can be modified to form the interframe approximation texture.

[0069] Thus, instead of fully encoding all of the blocks of each texture associated with a frame, only the reconstructed texture blocks of each keyframe are fully encoded, but for interframes, it is only the encoded blocks that provide significantly new information.

[0070] This feature allows for a significant reduction in the amount of texture data that is encoded during compression and then transmitted and decoded when the video is viewed online.

[0071] The blocks coded for the interframes are used to modify the keyframe's synthetic texture by iteration for each new frame if necessary to maintain acceptable visual quality.

[0072] FIG. 2D illustrates this situation, with the complete texture of a keyframe formed by a set of 256 blocks 210, and a set of 92 blocks 220 forming the partial texture of the interframe corresponding to that keyframe, where the missing blocks are considered to be sufficiently close to the missing blocks of the keyframe so that they do not need to be encoded again.

[0073] Instead of an uncoded block, an already coded block is used that is graphically close to the uncoded block, as long as the degradation in terms of visual quality remains tolerable.

[0074] The already coded blocks are either blocks of a keyframe or blocks of partial texture that already exist as replacements for blocks of a keyframe, so that a texture close to the interframe texture is obtained by modifying the keyframe texture by successive iterations.

[0075] This principle can be used according to a specific embodiment detailed below with reference to Figures 3B and 4A-5C.

[0076] The data from the stream of frames consists of groups of frames, each group containing a key frame followed by an inter frame in temporal succession.

[0077] Textures associated with frames of the same group appear naturally as a result of the group definition itself by geometric and photometric tracking, and therefore have many blocks in common that do not need to be re-encoded multiple times; initial encoding of the reconstructed texture of key frames and specific blocks selected for inter frames is sufficient.

[0078] The problem of partial texture reduction is to determine which blocks should be used for the texture of a key frame and which blocks need to be updated within each group of frames, thereby minimizing the number of blocks to be coded while at the same time maintaining an acceptable visual quality of the final video.

[0079] For a given block position, the graphical differences of the blocks of each frame relative to each other are evaluated during sub-step 304-1, either directly block-by-block or with uncompressed textures corresponding from block to block, as described below.

[0080] If a group of Nf frames is considered, then there are potentially Nf possible variations for each pixel block location in the texture image.

[0081] In practice, Nf different blocks B1 to BNf follow one another at the same position during the frames F1 to FNf, respectively, and can be considered as the temporal variation of a given block during the Nf frames F1 to FNf considered.

[0082] For illustration purposes, refer to FIG. 5C, which represents a group of a sequence of Nf frames, each associated with an uncompressed texture Tnc and a compressed texture Tc, where blocks 131 to BNf correspond to the same position Pos in the respective compressed texture Tc, and images (or sets of pixels) I1 to INf correspond to the same position in the respective uncompressed texture Tnc.

[0083] The blocks B1 to BnF and the images I1 to INf consist of sets of pixels with the same position, the same geometric shape and the same dimensions.

[0084] To assess the graphical variation between blocks, PSNR, which stands for Peak Signal to Noise Ratio, is commonly used in digital imaging to assess the quality of the reconstruction of a compressed image relative to the original image.

[0085] Other methods of assessing the graphical variability of the blocks may also be used, with PSNR representing only one possibility.

[0086] For a given block position of the compressed texture Tc, the PSNR is calculated between blocks B1 to BNf of respective frames F1 to FNf and images I1 to INF of the same position of the uncompressed texture of the corresponding frame of the same group, which gives NfxNf PSNR calculations to be performed.

[0087] In this way, for each of the blocks of the group of frames considered, a graphic difference is evaluated against all of the other blocks of the group of frames, in this case against the uncompressed image of the block considered.

[0088] Here we refer to uncompressed textures to ensure the fidelity of the encoded video to the original image, but it is also possible to calculate the PSNR between each block of a given frame and the corresponding block of a compressed texture in another frame.

[0089] The PSNR constitutes an indication of the similarity between two images, here a first image which is an image of blocks defined during the block compression step, and a second image which is an image formed by groups of pixels which correspond to blocks in the uncompressed image.

[0090] Furthermore, PSNR is expressed on a logarithmic scale, and a large value of PSNR indicates significant similarity between two images.

[0091] For each NfxNf PSNR calculated in step 304-1, an error value ErrVal is calculated in step 304-2 to numerically represent, on a linear scale, the amplitude of the difference between blocks from one frame to another, and thus over time.

[0092] Step 304-3 consists in constructing, for each block position in the group of Nf frames, a square table 400 containing NfxNf entries, each consisting of one of the error values ​​ErrVal calculated in step 304-2.

[0093] Each row of table 400 is dedicated to the evaluation of the temporal variation of a given position block across Nf frames F1 to FNf, with reference here to images I1 to InF corresponding to the block in the corresponding uncompressed texture.

[0094] Each column of table 400 is dedicated to a frame, which are sorted by order of appearance in the volumetric video stream and identified as F1 through FN.

[0095] The table entries are filled by inserting therein the error values ​​ErrVal calculated in step 304-2 as follows.

[0096] The entry with coordinates (p;q) in the pth row and qth column of table 400 corresponds to an error value ErrVal(Bp / Iq) calculated between the pth block at a given position of the compressed texture in the pth frame and an image Iq of a set of pixels at said given position of the uncompressed texture in the qth frame Fq, and converts the amplitude of the graphic difference between said pth block and said set of pixels.

[0097] For example, the second entry in the first row of the table corresponds to an error value ErrVal(B1 / I2) calculated between a first block B1 for a first frame F1 and a set of pixels 12 corresponding to that block in the uncompressed texture in the second frame F2 before compression by the block, and converts the amplitude of the graphic difference between that first block B1 and that set of corresponding pixels 12.

[0098] The diagonal entries of the table (B1;F1)~(BNf / FNf) convert the amplitude of the graphic difference between the block and the corresponding group of pixels of the uncompressed texture of one and the same frame and indicate the degradation of video quality introduced by the block compression step.

[0099] A possible approach defined by the inventors to minimize the amount of data to be encoded and then decoded when reading a video stream consists in minimizing the number of blocks to be encoded by making a selection among the blocks according to a table 400 indicating the errors introduced by the block compression of step 302, not only within a frame but also between frames of the same group for a given position of a block.

[0100] This selection corresponds to determining the path with the lowest possible cost for moving in the table from any entry in the first column to the left of the table corresponding to the first frame of the group considered, to any entry in the last column to the right corresponding to the last frame of the group considered, moving to the right or vertically (up or down), by introducing a cost for each horizontal displacement, a cost for each vertical displacement, and by trying to minimize the overall cost caused by the path traced.

[0101] During horizontal displacement, a block located at the start of the displacement to a corresponding frame, called the previous frame, is selected as the coding block, and this block is retained in the displacement destination, called the subsequent frame, after each successive horizontal displacement, without coding any intermediate blocks in the generated video stream.

[0102] When a block is used to code successive images, the visual quality of the coded images tends to deteriorate as they move away in time from the image used to generate the block.

[0103] Therefore, the error value ErrVal tends to increase with the length of the horizontal displacement.

[0104] It is therefore necessary to limit the amplitude of the horizontal displacement by imposing a vertical displacement corresponding to the coding of new blocks that are more suitable for coding the images of the new frame, which is favorable from the point of view of visual quality but unfavorable from the point of view of the data compression ratio during the visualization of the video stream, its transmission and its decompression.

[0105] One way to optimize a path is to assign a first cost to the degradation in visual quality caused by reusing the same blocks to code images for several frames, and a second cost to the computational intensity corresponding to the increase in the amount of data to be coded, and then determine the path that minimizes the total cost, where the total cost represents the sum of the first and second costs.

[0106] The optimal path can be determined by conventional graph theory methods, for example by the graph 450 of FIG. 4B plotted from table 400, the graph consisting of nodes and possible transitions between the nodes, by determining the path as a function of parameters defined by the practitioner according to their objectives and priorities, here the costs mentioned above.

[0107] The central node of graph 450 is U p,q and corresponds to the entry in the pth column and qth row of table 400.

[0108] Node U in the graph p , q corresponds to the use of blocks from the block position considered in the texture of the pth frame to code the texture of the qth frame.

[0109] The nodes are arranged in Nf rows and Nf columns, similar to the table entries, and a dummy start node Ud is connected along the row to each of the nodes corresponding to the entries ErrVal(Bl / II) to ErrVal(BFn / II) in the first column of the table, where in this example the number of frames Nf is equal to 4.

[0110] In graph 450, each ellipse represents a node, and each arrow represents a possible transition between two nodes.

[0111] The possible displacements in the graph are horizontal within the same row and vertical within the same column.

[0112] A possible horizontal displacement is performed from a given node to its immediate adjacent node to the right, for a cost of visual degradation that corresponds to the value ErrVal of the table entry that corresponds to the end node of the considered displacement, as represented by a solid arrow in graph 400.

[0113] The possible vertical displacements are above and below, between two directly adjacent or non-adjacent nodes, as represented by the dotted arrows, for a user-defined calculation cost Ccalc.

[0114] Here, for example, node U 1,3 From Node U 1,4 The cost to go to node U is ErrVal(B1 / I4). 1,3 From Node U 3,3 or Unode 3,4 The cost to go to is Ccalc.

[0115] Once the shape of the graph and the displacement rules therein are defined, the optimal path to be explored can be determined by, among other conventional methods, an implementation of the Dijkstra algorithm, which is commonly used to determine the shortest path between two points in a graph consisting of multiple interconnected nodes, which is the case for graph 450.

[0116] The optimal path with the lowest overall cost is considered to be the path that allows minimizing the amount of data to be encoded while still meeting an acceptable video quality.

[0117] The video quality is determined by the implementer by selecting a Ccalc cost according to a priority-dependent criterion, with a high Ccalc cost promoting a high compression ratio, a low Ccalc cost resulting in high visual quality, and an intermediate Ccalc cost resulting in a compromise between compression ratio and visual quality.

[0118] In conventional compression operations, the Ccalc cost is preferably around 0.0001, for example between 0.00001 and 0.001, or may be initialized, for example, with the average value of the error value ErrVal.

[0119] For example, FIG. 5A shows the node U 2,1 , U 2,3 , U 4,3 , and U 4,5 Therefore, node U has an optimal path Popt that passes through 2,3 and U4,3 4 illustrates the application of graph 450 to the specific case of a group of five frames for a given block position, including row changes between

[0120] For a given block position, block B2 of the second frame is encoded and used for the first and second frames F1 and F2 during video encoding, i.e. for the key frame and the first inter frame synthetic texture.

[0121] This block B2 does not need to be re-encoded for the second frame since it has already been encoded for the first frame.

[0122] Block B4 of the fourth frame is coded, replacing block B2 in frame 3 and subsequent frames F3, F4 and F5.

[0123] Block B4 is coded only once, but is used for three frames.

[0124] From a practical standpoint, a practitioner can expect to process groups of frames containing up to several tens of frames.

[0125] Step 304 described above is only applied to a given block location within the inter-frame texture of each frame group.

[0126] This step 304 is then repeated as applied to each block position to determine the block that will be used to code the entire texture, as indicated by loop L in diagram 300 .

[0127] In a test step 306 it is determined whether the frame is a key frame KF or an interframe IF.

[0128] In step 308, the set of blocks required to encode the complete texture associated with a key frame and determined in step 304 for each block location is reconstructed so that a synthetic texture is formed for that key frame, and a conventional compression method, such as an entropy compression method, is applied to the synthetic texture, for example by Huffman coding.

[0129] Thus, for each keyframe, the complete synthetic texture is encoded, as described above, by using blocks of different textures associated with each frame of the group of frames of the keyframe considered.

[0130] This represents a first difference with respect to conventional texture coding, where the texture associated with a key frame is coded exclusively with the block resulting from its block-by-block compression, independent of the texture of adjacent frames.

[0131] In step 310, the newly encoded blocks for each Inter frame determined in step 304 are reconstructed for that Inter frame.

[0132] Thus, for each inter frame, only a partial texture is coded, consisting of blocks that are coded for that frame, and optionally for subsequent frames.

[0133] In this way, instead of the complete texture for a key frame, only a portion of this interframe texture block is coded.

[0134] It may also be advantageous to compress the blocks by conventional entropy coding methods before transmitting the blocks, and possibly streaming them within a video stream.

[0135] Thus, as described above and shown by FIG. 2D, only the blocks of the reconstructed texture of each keyframe are fully coded, whereas for interframes, the only blocks that are coded are those that provide significantly new information.

[0136] These features result in significant improvements in compression ratios and computational savings during video read, since fewer blocks have to be encoded, then transmitted and decoded during read of the video stream. Mesh Compression

[0137] At the end of the preceding steps, each frame is associated with its own mesh in the stream of volumetric video data 172, which includes mesh data T and mesh data M, a reference mesh is associated with each key frame, and information regarding the modification of the reference mesh for each inter frame within a given frame group.

[0138] In a test step 301 it is determined whether the input data is mesh data M or texture data T.

[0139] In step 350, a conventional compression method is applied to each mesh found in the mesh data M, consisting of discretizing the coordinates of the points defining the mesh and the coordinates of the texture associated with a subset of values ​​in order to reduce the memory occupied.

[0140] In a test step 352 it is determined whether the given mesh is associated with a keyframe or an interframe.

[0141] In step 354, a compression method is applied to the information representing the keyframe mesh, and includes sub-steps 354-1, 354-2, and 354-3, each adapted to a particular type of data that as a whole defines the keyframe mesh.

[0142] In step 354-1, information representing the triangles of the keyframe is compressed using, for example, the "Edgebreaker" algorithm known from J. Rossignac, "Edgebreaker; Connectivity compression for triangle meshes", IEEE Transactions on Visualization and Computer Graphics, Vol. 5, No. 1, pp. 47-61.

[0143] In step 354-2, the information representing the points or vertices of the keyframe is compressed by applying a predictive algorithm according to known methods, for example as described in the reference C. Touma and C. Gotsman, Triangle Mesh Compression, Proceedings Graphics Interface 98, pp. 26-34, 1998.

[0144] In step 354-3, information representing the texture coordinates of the triangles of the keyframe is compressed using a selective linear prediction algorithm, as described, for example, in Isenburg, M and Snoyeink, J., “Compressing texture coordinates with selective linear predictions”, Proceedings of Computer Graphics International 2003.

[0145] The "Linear Selective Prediction" algorithm encodes texture coordinates per point (vertex), whereas texture coordinates are typically encoded per triangle, which reduces their number and allows for better compression as well as computational optimization during decoding.

[0146] In step 356, a compression method is applied to the information representing the inter-frame points or vertices by differential encoding.

[0147] Within a group of frames, the inter-frame triangle and texture coordinates are the same as those of the corresponding keyframe, so their difference is zero within each group of frames and this information does not need to be re-encoded.

[0148] According to the principle of differential coding, only information about the positions of points that can vary from frame to frame needs to be coded; only the changes between frames are coded.

[0149] Differential encoding, also called delta or delta coding, is a lossless data compression technique that consists of transforming data by a series of successive data differences. In this case, it is a particularly effective technique.

[0150] In step 358, the information compressed during steps 354-1 through 354-3 and 356 is compressed again, this time by an entropy calculation algorithm.

[0151] In particular, it is preferable to perform entropy calculations using arithmetic compression such as FSE (Finite State Entropy), which can provide an excellent compression rate for the type of data being processed.

[0152] It should be noted that even if steps 354-1 to 354-3, 356 and 358 are performed individually and are already known to those skilled in the art, their combination as described herein is novel and provides superior results to known combinations of conventional methods in the field of optics for obtaining compressed video streams that can be viewed online.

[0153] Indeed, video data associated with a mesh, when compressed in accordance with the present invention, can be readily decoded in real time by conventional computing systems.

[0154] In step 360, the data streams resulting from steps 308, 310 and 358 are combined into a video file that can be decoded in streaming mode for viewing over a computer network. Decryption

[0155] During streaming of the video stored in the form of a video file in step 360, information representing the frames is transmitted over a computer network and sequentially decoded as it arrives at the user's computer system in the form of a compressed data stream 602, according to the diagram of FIG.

[0156] The data representing each frame arrives sequentially according to the order of the respective frames in the video stream and is decoded in that order, with the texture data T and mesh data M being separated.

[0157] The frames include key frames and inter frames, identified in the figure as KF and IF respectively, which represent the information items M and T to be decoded for information representing meshes and textures respectively.

[0158] During a test step 604, the data representing the mesh and the texture are each identified and then decoded separately.

[0159] In step 610, data representing the texture T is entropy decoded.

[0160] During a test step 612, the decoded data of step 610 which correspond respectively to key frames KF and inter frames IF are identified.

[0161] Since the data representing the texture of the keyframe KF is always in a compressed format, no additional processing is required, being block compressed in step 302, and the product can be processed directly by a conventional graphics processing unit.

[0162] However, during step 614, data representing the texture of the interframe IF which depends on data of other frames are processed to modify them by again obtaining the data of the preceding frame and replacing with the corresponding new block the data from the block in which the new block was coded following step 310.

[0163] This is a computationally straightforward and very fast operation.

[0164] In step 620, the data representing the mesh M first undergoes entropy coding decoding by the FSE, which is performed according to conventional algorithms for very fast execution.

[0165] In a test step 622, data corresponding respectively to key frames KF and inter frames IF are identified.

[0166] During step 624, the data M representing the mesh of the key frame KF obtained following steps 354-1, 354-2 and 354-3 are decoded independently of the data of other frames in a conventional manner, quickly with respect to the type of compression used on said data (predictive or Edgebreaker).

[0167] During step 626, the data representing the mesh M of the inter-frame IF obtained after step 354 and dependent on previously decoded data of the respective dependent keyframe KF are decoded according to the conventional very fast method, since they are only delta compressed with respect to the vertex positions.

[0168] In step 628, dequantization of the mesh data from steps 624 and 626 is performed according to conventional methods.

[0169] Finally, following operations 610, 614 and 628 the resulting data is processed by a data processing unit and / or a graphics processing unit in a conventional manner to facilitate display 630 of the video.

[0170] The particular method used to encode the video data allows for substantial improvements in data weight and decoding speed, and the decoding can be performed by conventional computing units.

[0171] It goes without saying that the invention is not limited to the embodiments disclosed above, which can be modified and combined without departing from the scope of the invention.

Claims

1. A method (300) for compressing a volumetric video stream (172) of a three-dimensional action scene (125) represented by a plurality of frames (F), the method being computer-implemented, according to which for each frame of the scene (125) a mesh (M) and a texture (T) are generated, and each group of frames (FGr1, FGr2, FGr3) consisting of key frames (KF) and inter frames (IF) are defined, the mesh of which is defined relative to the mesh of a corresponding key frame, the method comprising the steps of compressing information representative of the texture, the step of compressing the information representative of the texture comprising the steps of: - compressing (302) the information representing the texture of each of the frames of the group according to a block compression method capable of generating an encoding of the texture by blocks of pixels of the same size, each encoded block being independent of the other blocks, and forming blocks directly usable by a graphics processing unit according to standard algorithms including DXT, ASTC or PVRTC, thus defining blocks of pixels (B1, B2, B3, BNf) directly usable by the graphics processing unit, including a block associated with the key frame and a block associated respectively with each of the inter frames; - determining (304) from the blocks associated with the keyframe and the blocks associated with the interframe, on the one hand, first blocks (210) capable of forming the synthetic texture of the keyframe, and on the other hand, second blocks (220) capable of modifying the synthetic texture of the keyframe by copying said second blocks (220) into the synthetic texture so as to form an approximation texture of the interframe, by iterative substitution of blocks; - forming (308) a synthetic texture for said key frame of said group of frames from a first block associated with said key frame and a first block associated with said inter frame, and encoding said synthetic texture; and encoding (310) the texture of the inter frame based on the synthetic texture of the key frame and the second block.

2. The method of claim 1 , wherein the step of encoding the synthetic texture of the key frames and the texture of the inter frames comprises compression by entropy coding.

3. 3. The method according to claim 1 or 2, wherein the step of determining the first block and the second block comprises a step (304-1, 304-2) of evaluating a quantized graphic difference (ErrVal(B1 / I1) to ErrVal(BNf / INf)) based on the luminance or chrominance difference of the blocks of the group of frames in a given position relative to each other, the evaluating step being based on a calculation such as a calculation of a PSNR (Peak Signal to Noise Ratio) between two considered blocks of the group of frames, each of the blocks belonging to a respective frame, and wherein for each of the blocks of the group of frames, this calculation is repeated for the other blocks of the group of frames to obtain a number each representative of the graphic difference between the two considered blocks.

4. The method of claim 3 , wherein for each of the blocks of the group of frames, the graphical difference is evaluated relative to all of the other blocks of the group of frames.

5. The step of determining the first block and the second block comprises: According to the pth and qth columns, U p、q (U 1、1 From U 4、4 ), wherein each row is dedicated to an evaluation of the amplitude of the quantized graphic difference based on the luminance or chrominance difference between blocks of a given position in the frames of the group of frames, and thus, over time, each column is dedicated to a frame of the group of frames, the frames being sorted in the order of their appearance in the volumetric video stream, each node corresponding to the use of a block of the position of the considered block in the texture of the pth frame for coding the texture of the qth frame, the quantized graphic difference being determined by two nodes (U 1、1 , U 1、2 ), where the displacement cost represents the quantized graphic difference between two blocks of a group of frames at a given position; and a computation cost (Ccalc), defined by the user, for each row (U 1、3 , U 2、3 , U 3、3 , U 4、3 , U 5、3 ), a computational cost (Ccalc) of the method is assigned to the displacement according to the step of 5. The method according to claim 3, further comprising the step of determining a minimum cost path (Popt) in this graph, each path starting in a column associated with one of the first blocks (B2) and each displacement following a column associated with one of the second blocks (B4), the minimum cost being equal to the sum of the cost of each displacement between two nodes on the same row (ErrVal(B1 / I2)) and the cost of each displacement between two nodes on one same column (Ccalc), said path being composed of each said displacement between two nodes on the same row and each said displacement between two nodes on the same column.

6. The method of claim 5 , wherein the step of determining a route implements a Dijkstra algorithm.

7. compressing the information representing the mesh, - compressing (350) said information representing said mesh by quantization; - compressing (354) said information representing said quantized mesh according to a compression method, comprising: - compressing (354-1) the information representing the triangles of the mesh of the keyframes according to the Edgebreaker algorithm; - compressing (354-2) the information representative of the points of said mesh of said key frames according to a prediction algorithm; - compressing (354-3) information representative of the texture coordinates of said keyframes according to a linear selective prediction algorithm; - compressing (356) the information representative of the points of the interframe mesh by differential encoding; and compressing (358) the compressed information representing the points, the triangles, and the texture coordinates of the mesh of the key frames and the information representing the points of the mesh of the inter frames by an entropy calculation algorithm.

8. A data processing system comprising means for carrying out the steps of the method according to any one of claims 1 to 7.

9. A computer program comprising instructions which, when said program is executed by a computer, cause said computer to carry out the steps of the method according to any one of claims 1 to 7.

10. A computer readable medium containing instructions which, when executed by a computer, cause the computer to perform the steps of the method of any one of claims 1 to 7.

Images