A method and apparatus for decoding a three-dimensional scene

Abstract

A method for generating an image from an encoding of a source image of a projection onto a part of a three-dimensional scene is disclosed. The pixels of the source image comprise depth and color attributes. The pixels of the source image are unprojected into a colored point cloud. The unprojected points in the 3D space have the color attributes of the pixels that were unprojected. Furthermore, a score is attributed to the points according to the local depth gradient and / or the local color gradient of the pixels from which the generated points come, the lower the gradient, the higher the score. In a second step, the generated point cloud is captured by a virtual camera for rendering on a display device. The point cloud is projected onto a viewport image by blending the colors of the points projected onto the same pixel, the blending being weighted by the scores of these points.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to decoding a three-dimensional scene comprising a three-dimensional (3D) object. In particular, but not exclusively, the technical field of the present disclosure relates to decoding depth and texture image data obtained by projecting points of a 3D scene for representing geometry and colors of a three-dimensional scene. BACKGROUND

[0002] This section is intended to introduce the reader to various aspects of art that can be related to various aspects of the present disclosure that are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present disclosure. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0003] A point cloud is a set of points, usually used to represent the outer surface of a 3D object, but also more complex geometries such as hair or fur, which can not be efficiently represented by other data formats such as meshes. Each point in a point cloud is usually defined by a 3D spatial position (X, Y and Z coordinates in an orthogonal reference frame of the 3D space or angles p, f and distance d in a radial reference frame of the 3D space) and possibly other related attributes such as color represented in an RGB or YUV color space, transparency, reflectance, normal vector, etc. A point cloud can be static or dynamic, depending on whether the cloud evolves over time or not. It should be noted that in the case of a dynamic point cloud, the number of points is not constant but, on the contrary, usually evolves over time. A dynamic point cloud is thus a time-sequential sequence of sets of points.

[0004] Virtual Reality (VR) and immersive worlds have recently become a hot topic and are foreseen by many as the future of two-dimensional (2D) flat video. The basic idea is to immerse the viewer in the environment around him, as opposed to standard TV where he can only look at a virtual world in front of him. Immersion has several levels, depending on the viewer’s degree of freedom in the environment. Point clouds, especially colored point clouds, are a good candidate format for distributing VR worlds and volumetric video. They can be static or dynamic, usually of average size (a few million points at a time).

[0005] A well-known approach projects a colored point cloud representing the geometry and colors of a 3D object onto a surface in a 3D environment, for example onto the faces of a cube containing the 3D object, to obtain a video for texture and depth, and encodes the texture and depth video using a conventional encoder such as 3D-HEVC.

[0006] At the decoder, the pictures and depth images (a video is a sequence of images) are back-projected to regenerate a color point cloud (or a sequence of color point clouds). The back-projection is performed from information representative of the projection mapping associated with the surface onto which the original color point cloud has been projected. Due to unavoidable discretization due to image rasterization and non-lossy compression methods, representing the geometry and colors of a point cloud with projected images introduces inaccuracies in the coordinates and colors of the points. When regenerating a color point cloud from these images, the same point in the original color point cloud can be back-projected several times at different coordinates and / or with different colors. These errors and inaccuracies often result in ghosting effects in the viewport images when the reconstructed color point cloud is captured by a virtual camera not at the center of projection of the source images. The present disclosure solves this problem by exploiting the fact that the same point of the original color point cloud can be projected onto several images. SUMMARY

[0007] The following presents a simplified summary of the present disclosure to provide a basic understanding of some aspects of the disclosure. This summary is not an extensive overview of the present disclosure. It is not intended to identify key or critical elements of the present disclosure. The following summary merely presents some aspects of the present disclosure in a simplified form as a prelude to the more detailed description provided below.

[0008] The present disclosure relates to a method for generating an image from a source image encoding a projection of a part of a three-dimensional scene, the pixels of the source image comprising depth and color attributes. The method comprises:

[0009] - unprojecting the pixels of the source image into a color point cloud. From the depth attribute of a pixel, the pixel is unprojected into a point in the 3D space. This point has the color attribute of the pixel and has a score attributed to the local depth gradient and / or the local color gradient of the pixel from which this point is generated, the lower the gradient, the higher the score;

[0010] - projecting the built color point cloud onto the image to be generated by mixing the color attributes of the points projected onto the same pixel. This mixing is a function weighted by the score attribute of these points.

[0011] According to one embodiment, the source image is decoded from a data stream. The source image can be associated in the data stream with data representative of a projection of a part of a three-dimensional scene from which the generated image is derived. These data are subsequently used to unproject the pixels of the source image.

[0012] According to another embodiment, the generated image is a viewport image rendered from a position and orientation of a virtual camera in said 3D space. This pose (i.e. position and orientation) of the virtual camera can be controlled by sensors of a head-mounted display device.

[0013] The present disclosure also relates to a device comprising a processor configured to implement the above-described method. The present disclosure also relates to a non-transitory processor-readable medium having stored therein instructions for causing a processor to perform the method. BRIEF DESCRIPTION OF DRAWINGS

[0014] The present disclosure will be better understood by reading the following description, with reference to the attached drawings, in which:

[0015] - Figure 1 shows points of a three-dimensional (3D) model of an object and of a point cloud corresponding to the 3D model, according to non-limiting embodiments of the present disclosure;

[0016] - Figure 2 shows non-limiting examples of encoding, transmitting and decoding data representative of a sequence of 3D scenes, according to non-limiting embodiments of the present disclosure;

[0017] - Figure 3 shows an example architecture of a device that can be configured to implement the methods described in connection with Figure 7 the present disclosure, according to non-limiting embodiments of the present disclosure;

[0018] - Figure 4 shows an example of the syntax of a stream when data is sent through a packet-based transmission protocol, according to non-limiting embodiments of the present disclosure;

[0019] - Figure 5 shows projections of 3D models representative of a plurality of 3D objects on a first image corresponding to a first projection surface and on a second image corresponding to a second projection surface, according to non-limiting embodiments of the present disclosure;

[0020] - Figure 6 shows ghosting effects resulting from the compression of images encoding a 3D scene, according to non-limiting embodiments of the present disclosure;

[0021] - Figure 7 schematically shows a method 70 for generating an image (e.g. a viewport image) from a source image encoding a projection of a portion of a three-dimensional scene, according to non-limiting embodiments of the present disclosure. DETAILED DESCRIPTION

[0022] The disclosure will be described more fully hereinafter with reference to the accompanying drawings, in which examples of the disclosure are shown. The disclosure may, however, be embodied in many alternate forms and should not be construed as limited to the examples set forth herein. Accordingly, while the disclosure is susceptible to various modifications and alternative forms, specific examples thereof are shown by way of example in the drawings and will be described herein in detail. It should be understood, however, that there is no intent to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the disclosure as defined by the claims.

[0023] The terminology used herein is for the purpose of describing particular examples only and is not intended to be limiting of the disclosure. As used herein, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes" and / or "including," when used herein, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Additionally, when an element is referred to as being "responsive" or "connected" to another element, it can be directly responsive or connected to the other element, or indirectly responsive or connected to the other element through one or more other elements. In contrast, when an element is referred to as being "directly responsive" or "directly connected" to another element, there are no intervening elements. As used herein the term "and / or" includes any and all combinations of one or more of the associated listed items and can be abbreviated as " / ".

[0024] It should be understood that, although terms such as first, second, etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element without departing from the teachings of the present disclosure.

[0025] Although some of the diagrams include arrows on communication paths to show the primary direction of communication, it is to be understood that communication can occur in the opposite direction to the depicted arrows.

[0026] Some examples are described with respect to block and operational flow diagrams in which each block represents circuit elements, modules or portions of code that include one or more executable instructions for implementing the specified logical function of a part. It should also be noted that in other implementations, the functions described in the blocks can not occur in the order described. For example, according to the functions involved, two blocks shown in succession can in fact be executed substantially concurrently, or sometimes the blocks can be executed in the reverse order.

[0027] The phrase "according to an example" or "in an example" as used herein means that a particular feature, structure, or characteristic described in connection with the example is included in at least one implementation of the disclosure. The appearances of the phrase "according to an example" or "in an example" in various places in the specification are not necessarily all referring to the same example, nor are they necessarily mutually exclusive, or alternative examples to one another, to the other examples described herein.

[0028] The reference signs appearing in the claims are merely intended to illustrate the application and shall not be construed to limit the scope of the claims. Although not explicitly described, the present examples and variants can be implemented in any combination or sub-combination.

[0029] The disclosure describes encoding / decoding point clouds, but extends to encoding / decoding sequences of point clouds, as each colored point cloud in the sequence is sequentially encoded / decoded as described hereafter. The point clouds can be colored.

[0030] In the following, an image or picture contains one or more arrays of samples (pixel values) of a specific image / video format that specifies all information related to the pixel values of the image (or video) and all information that can be used by a display and / or any other device to visualize and / or decode the image (or video). An image comprises at least one component in the shape of a first array of samples, typically a luma component or a depth component, and possibly at least one other component in the shape of at least one other array of samples, typically a color component. Or, equivalently, the same information can also be represented by a set of color sample arrays, e.g. a three primary color RGB representation.

[0031] Figure 1 A three-dimensional (3D) model 10 of an object is shown as well as points of a point cloud 11 corresponding to the 3D model 10. The 3D model 10 and the point cloud 11 can for example correspond to a possible 3D representation of an object in a 3D scene comprising other objects. The model 10 can be a 3D mesh representation, while the points of the point cloud 11 can be the vertices of the mesh. The points of the point cloud 11 can also be points scattered on the surface of the faces of the mesh. The model 10 can also be represented as an unfolded version of the point cloud 11, the surface of the model 10 being created by unfolding the points of the point cloud 11. The model 10 can be represented by many different representations, e.g. voxels or splines. Figure 1It is shown that a point cloud can be defined with a surface representation of a 3D object, and that a surface representation of a 3D object can be generated from a point cloud. As used herein, projecting a point of a 3D object (by an extended point of a 3D scene) onto an image is equivalent to projecting any representation of that 3D object, e.g. a point cloud, a mesh, a spline model or a voxel model.

[0032] A point cloud can be represented in memory as, for example, a vector-based structure, where each point has its own coordinates in the reference frame of the viewpoint (e.g. three-dimensional coordinates XYZ, or a stereographic angle and distance from / to the viewpoint (also called depth)) and one or more attributes, also called components. One example of a component is a color component, which can be represented in various color spaces, such as RGB (red, green, and blue) or YUV (Y is the luminance component, UV are two chrominance components). A point cloud is a representation of a 3D scene comprising an object. The 3D scene can be viewed from a given viewpoint or range of viewpoints. A point cloud can be acquired in a variety of ways, for example:

[0033] • from a capture of a real object by a set of cameras, optionally complemented by depth active sensing devices;

[0034] • from a capture of a virtual / synthetic object by a set of virtual cameras in a modeling tool;

[0035] • from a mix of real and virtual objects.

[0036] Figure 2 Non-limiting examples of encoding, transmitting and decoding data representing a sequence of 3D scenes are shown. For example, an encoding format can be compatible with 3DoF, 3DoF+ and 6DoF decoding at the same time. It should be understood that in the present text, a sequence of items can be a unique item, e.g. a sequence of 3D scenes can be one 3D scene or a sequence of images can be one image.

[0037] A sequence of 3D scenes 20 is acquired. Just as a sequence of pictures is a 2D video, a sequence of 3D scenes is a 3D (also called stereoscopic) video. The sequence of 3D scenes can be provided to a stereoscopic video rendering device for 3DoF, 3DoF+ and 6DoF rendering and display.

[0038] The sequence of 3D scenes 20 is provided to an encoder 21. The encoder 21 takes as input one 3D scene or a sequence of 3D scenes and provides a bitstream representing the input. The bitstream can be stored in a memory 22 and / or on an electronic data medium, and can be transmitted on a network 22. The bitstream representing the sequence of 3D scenes can be read from the memory 22 and / or received from the network 22 by a decoder 23. The decoder 23 is input with said bitstream and provides a sequence of 3D scenes in, for example, a point cloud format.

[0039] Encoder 21 may include circuitry implementing several steps. In a first step, encoder 21 projects each 3D scene onto at least one 2D image. 3D projection is any method of mapping three-dimensional points onto a two-dimensional plane. Since most current methods of displaying graphical data are based on a two-dimensional medium (pixel information from some bit plane), this type of projection is widely used, particularly in computer graphics, engineering, and drafting. Projection circuitry 211 provides at least one two-dimensional frame 2111 for the 3D scene sequence 20. Frame 2111 includes color and depth information representing the 3D scene projected onto frame 2111. In a variant, the color and depth information are encoded in two separate frames 2111 and 2112.

[0040] Metadata 212 is used and updated by projection circuit 211. (For example, regarding...) Figures 5 to 7 The metadata 212 described includes information about projection operations (e.g., projection parameters) and how color and depth information is organized within frames 2111 and 2112.

[0041] The video encoding circuit 213 encodes the sequence of frames 2111 and 2112 into video. The video encoder 213 encodes images (or sequences of images of the 3D scene) of 2111 and 2112 into the stream. Then, the data encapsulation circuit 214 encapsulates the video data and metadata 212 into the data stream.

[0042] For example, encoder 213 is an encoder that conforms to the following:

[0043] -JPEG, standard ISO / CEI 10918-1UIT-T recommended T.81, https: / / www.itu.int / rec / T-REC-T.81 / en;

[0044] -AVC, also known as MPEG-4 AVC or h264, is specified in both UIT-T H.264 and ISO / CEI MPEG-4 Part 10 (ISO / CEI 14496-10), http: / / www.itu.int / rec / T-REC-H.264 / en, HEVC (its specification can be found on the ITU website -T Recommendation -H Series - h265). http: / / www.itu.int / rec / T-REC-H.265-201612-I / en );

[0045] -3D-HEVC (an extension of HEVC, whose specification can be found on the ITU website -T Recommendation-H Series-h265, http: / / www.itu.int / rec / T-REC-H.265-201612-I / en annex G and I);

[0046] - VP9 developed by Google; or

[0047] - AV1 (Alliance for Open Media Video 1) developed by the Alliance for Open Media.

[0048] The data stream is stored in a memory accessible by the decoder 23 through the network 22, for example. The decoder 23 comprises different circuits implementing different decoding steps. The decoder 23 takes as input the data stream generated by the encoder 21 and provides a sequence of 3D scenes 24 to be rendered and displayed by a stereoscopic video display device, like a head-mounted device (HMD). The decoder 23 gets the stream from a source 22. The source 22 belongs to a set comprising, for example:

[0049] - a local memory, for example a video memory or a RAM (or Random Access Memory), a flash memory, a ROM (or Read Only Memory), a hard disk;

[0050] - a storage interface, for example an interface of a mass memory, a RAM, a flash memory, a ROM, an optical or magnetic support;

[0051] - a communication interface, for example a wired interface (for example a bus interface, a wide area network interface, a local area network interface) or a wireless interface (for example an IEEE 802.11 interface or a Bluetooth interface); and

[0052] - a user interface, for example a graphical user interface, enabling a user to input data.

[0053] The decoder 23 comprises a circuit 234 for extracting the data encoded in the data stream. The circuit 234 takes as input the data stream and provides metadata 232 corresponding to the metadata 212 encoded in the stream and a two-dimensional video. The video is decoded by a video decoder 233 providing a sequence of frames. The decoded frames comprise color and depth information. In one variant, the video decoder 233 provides two sequences of frames, one comprising color information and the other comprising depth information. The circuit 231 uses the metadata 232 to de-project the color and depth information from the decoded frames, in turn providing a sequence of 3D scenes 24. The sequence of 3D scenes 24 corresponds to the sequence of 3D scenes 20, possibly with a loss of precision related to the encoding as a 2D video and the video compression.

[0054] The video rendering device 23 displays a part of the received 3D scene in its viewport according to a viewpoint determined by a controller, for example an inertial measurement unit (IMU) of the HMD.

[0055] Figure 3 An example architecture of a device 30 that can be configured to implement the method described in connection with Figure 7 is shown. Figure 2The encoder 21 and / or the decoder 23 can implement such an architecture. Alternatively, each circuit of the encoder 21 and / or the decoder 23 can be a device according to the architecture of Figure 3 linked together, for example via their buses 31 and / or via the I / O interface 36.

[0056] The device 30 comprises the following elements linked together by data and address buses 31 :

[0057] - a microprocessor 32 (or CPU), which is for example a DSP (or Digital Signal Processor);

[0058] - a ROM (Read Only Memory) 33;

[0059] - a RAM (Random Access Memory);

[0060] - a storage interface 35;

[0061] - an I / O interface 36 for receiving data to be transmitted from an application; and

[0062] - a power supply, for example a battery.

[0063] According to one example, the power supply is external to the device. In each of the mentioned memories, the word "register" used in the present description can correspond to an area of small capacity (a few bits) or to a very large area (for example, the entire program or a large amount of received or decoded data). The ROM 33 comprises at least the program and the parameters. The ROM 33 can store algorithms and instructions to perform the techniques according to the present disclosure. When switched on, the CPU 32 loads the program into the RAM and executes the corresponding instructions.

[0064] The RAM 34 comprises the program in registers executed by the CPU 32 and uploaded after switching on of the device 30, the input data in registers, the intermediate data of the different states of the method in registers, and other variables used to perform the method in registers.

[0065] The implementations described herein can be implemented in, for example, a method or a flow, an apparatus, a computer program product, data streams, or signals. The implementation of the described features can be implemented in other forms, for example, a program, even though described in the context of only a single implementation form (for example, only as a method or device). Apparatuses can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, an apparatus such as a processor, which is typically a processing device that includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, for example, a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate the communication of information between end users.

[0066] According to the example, device 30 is configured to implement bonding. Figure 7 The described method belongs to a set including the following:

[0067] -mobile device;

[0068] - Communication equipment;

[0069] -Gaming devices;

[0070] - Tablet (or tablet PC);

[0071] - Laptop;

[0072] -Still image camera;

[0073] - Video camera;

[0074] - Encoding chip;

[0075] - Servers (such as broadcast servers, video-on-demand servers, or web servers).

[0076] Figure 4 An example of an embodiment of the syntax of a stream is shown when data is sent via a packet-based transport protocol. Figure 4 Example structure 4 for a stereoscopic video stream is shown. This structure is contained within a container that organizes the stream by independent syntax elements. The structure may include a header 41, which is a set of data common to each syntax element of the stream. For example, the header includes metadata about the syntax elements, describing the properties and function of each element. The header may also include... Figure 2 This is part of the metadata 212, such as the coordinates of the center viewpoint used to project points of the 3D scene onto frames 2111 and 2112. The structure includes a payload containing syntax element 42 and at least one syntax element 43. Syntax element 42 includes data representing color and depth frames. The image may have been compressed according to a video compression method.

[0077] Syntax element 43 is part of the payload of the data stream and may include metadata about how the frames in syntax element 42 are encoded, such as parameters used for point projection of the 3D scene and packing onto the frames. This metadata may be associated with each frame of the video or a group of frames (also known as a group of pictures (GoP) in video compression standards).

[0078] Figure 5A 3D model 50 representing a plurality of 3D objects is shown projected onto a first surface 51 to obtain a first image 53 and onto a second surface 52 to obtain a second image 54. For the sake of clarity, surfaces 51 and 52 are planar rectangles associated with an orthographic projection mapping. The projection surfaces can be of any shape and associated with a respective projection mapping. For example, the projection surfaces can be a spherical crown associated with a spherical projection mapping such as equirectangular projection mapping or longitude-latitude projection mapping. In the example of Fig. 1, Figure 5 The 3D point cloud 50 comprises a three-textured (i.e. colorized) Moai statue. The pixels of images 53 and 54 store depth and color information, for example in RGB-D format. In Figure 5 For the sake of clarity, only color information is shown. In another embodiment, depth and color are stored in a pair of images, one for color information, for example in RGB or YUV format, and one for depth information, for example as a grayscale image. Without loss of generality, the present disclosure applies to this embodiment.

[0079] A first portion of points of the point cloud is projected onto surface 51 to obtain image 53, also called first image. A second portion of points of the point cloud is projected onto surface 52 to obtain image 54, also called second image. The first and second portions have points in common, the second portion comprises points not belonging to the first portion, and the union of the first and second portions can not contain the entire point cloud. Encoding the union of the first and second portions can be sufficient for a rendering in which the viewer can only observe the 3D model from a limited stereoscopic view of the viewpoints, also called 3DoF+ rendering scene. There is no reason to encode points if the end user is not allowed to observe the back of the statue. In a six degrees of freedom rendering scene in which the viewer is allowed to observe the encoded 3D scene 30 from each viewpoint, also called 6DoF, more projection images would be required, otherwise, parts of the model would be lost.

[0080] As Figure 2 shown, images 53 and 54 representing a portion of the 3D scene are encoded, compressed and sent to a decoder. The decoder receives images 53 and 54 associated with data describing the conditions and parameters of projections 51 and 52, for example the parametric surfaces and the mapping format. Images 53 and 54 are de-projected according to these data. In Figure 5 the example of Fig. 2, the pixels of image 53 are mapped on surface 51 and the points of the 3D scene under reconstruction are generated by orthogonally de-projecting the pixels at a distance corresponding to the depth attribute of the pixels of image 53. The color attribute of a pixel is attributed to the point generated from this pixel. The same operation is performed on image 54 and the portion of the original 3D scene encoded via the two images 53 and 54 is reconstructed.

[0081] At the decoder, once the 3D scene is reconstructed, for example, a viewport image is captured via a virtual camera. In a 3DoF+ scene, the camera can be moved by the user, for example, by a user wearing a head-mounted display (HMD), with the HMD's inertial measurement unit controlling the virtual camera's position and orientation. The reconstructed 3D scene is rendered by projecting it onto the plane of the viewport image, for example, using pinhole projection. Figure 5 In the example, the 3DoF+ space in which the virtual camera can move is typically located between the two centers of surfaces 51 and 52.

[0082] Due to the nature of video codecs (such as the HEVC codec), global low-pass filtering of the color and depth attributes of the compressed image may occur after compression (especially at low bit rates), resulting in undesirable smoothing of the color and depth atlas after decoding. The color and depth values ​​of the decoded image are slightly modified. Therefore, the 3D points deprojected from the pixels of such an image are not in the same location and / or have different colors than their corresponding points in the original 3D scene. This defect arises once strong gradients appear in the color and / or depth attributes of the image. The associated pixel values ​​are then very sensitive to low-pass filtering and produce artificial intermediate values ​​after encoding and decoding, which causes undesirable anti-aliasing around contours. This artifact is particularly important for the transition between foreground and background objects in a 3D scene. Anti-aliasing around color and / or depth contours tends to associate the foreground color with the background and / or the background color with the foreground. These artifacts produce a ghosting effect, which is especially noticeable when the virtual camera's position and / or orientation captures exposed portions of the 3D scene, i.e., when points seen from the virtual camera's viewpoint (i.e., points projected onto the viewport image) have been deprojected from multiple source images.

[0083] Figure 6 This illustrates the ghosting effect resulting from compressing an image of an encoded 3D scene. Figure 6 In the example, a point cloud is reconstructed by deprojecting a set of images representing a portion of an original 3D scene projected onto a dark background, which includes light-colored cubes in front of a dark background. When these images are rasterized, quantized, and compressed, the deprojection of their pixels includes inaccuracies and errors. Points in the reconstructed 3D scene may be located in different positions and / or have different colors than their corresponding original points. When a virtual camera capturing the scene for rendering in the viewport image of a display device is positioned close to the projection and oriented 63 (e.g., like...), the point cloud is reconstructed. Figure 5The ghosting effect is similar to the jaggies effect shown on the corresponding viewport image 65. However, when the camera is in a position and / or orientation from which it captures a point generated from a plurality of images similar to the point of view 64, the errors in the position and color of the point create a visible artifact called ghosting effect 66, as shown by the viewport image 65 corresponding to the point of view 64. This artifact is particularly important for transitions between foreground and background objects. The antialiasing that occurs around color and / or depth contours tends to associate foreground colors with background and / or background colors with foreground. It creates a residual ghost contour of the foreground object on the background exposed part.

[0084] Figure 7 A method 70 for generating an image, for example a viewport image, from source images encoding projections of a part of a three-dimensional scene is schematically illustrated. In step 71, images are obtained from the sources described in connection with Figure 2 and 3 The method 70 produces a technical effect of reducing the ghosting effect even with only one source image. Since it takes advantage of the redundancy of information, the more source images the more effective the method is. The source images can be encapsulated in a data stream and in this data stream are associated data representing the projections used to generate the image. In one variant, the decoder knows these data, which are stored in a memory accessible to the decoder. In another variant, these data are obtained from a source different from the source images. The source images have depth and color properties. In another embodiment, the depth and color are encoded in a pair of source images, one for the depth and one for the color.

[0085] At step 72, the pixels of the source image are de-projected according to the data associated with the image. The pixels of the source image are de-projected as points at a distance relative to their depth attribute and the color attribute of the pixel is obtained. The generated points can have other attributes from the pixel, such as a normal vector or a luminance attribute. According to the disclosure, the generated points have a score attribute determined from the local depth and / or color gradient of the pixel. Since each source image is generated by various projections (with different centers and / or parametric surfaces), it is likely that redundant parts of the original 3D scene, especially contours, are projected on the first source image with grazing incidence angles and on the second source image with more orthogonal incidence angles. In the latter case, the depth gradient and the color gradient are lower. Computing the local gradient of a pixel attribute is a well-known operation. For example, the local depth gradient is computed for a N x M rectangular (or square, i.e. N = M) pixel area around the considered pixel. In a variant, the gradient is computed for an elliptical area centered on the considered pixel. According to the disclosure, the depth and color gradients of the pixel to be de-projected are computed and the score is determined as a function of these two gradients, the lower the depth and / or color gradient of the pixel, the higher the score attributed to the generated point. For example, the score is determined according to equation 1, where a and b are constant coefficients:

[0086]

[0087] At step 73, an image is generated for rendering the decoded color point cloud onto a viewport of a display device according to the position and orientation of the virtual camera in the 3D space of the point cloud. For example, the projection used to generate the viewport image is a pinhole projection. Other projections can be used, like a fisheye projection, an equirectangular projection or a cubemap projection. According to the disclosure, the color attributed to a pixel is a mix of the colors of the 3D points projected onto this pixel, this mix being weighted by the scores of these 3D points determined at step 72. For example, the color of the generated point is determined according to equation 2.

[0088]

[0089] In another embodiment, the color attributed to a pixel is the color of the 3D point projected onto this pixel with the highest score determined at step 72.

[0090] The implementations described herein can be implemented, for example, in methods or processes, apparatus, computer program products, data streams, or signals. Even if discussed only in the context of a single implementation (e.g., only as a method or apparatus), the implementation of the features in question can be implemented in other forms (e.g., a program). Apparatus can be implemented, for example, with appropriate hardware, software, and firmware. These methods can be implemented in apparatuses such as processors, which generally refer to processing devices including, for example, computers, microprocessors, integrated circuits, or programmable logic devices. Processors also include communication devices such as smartphones, tablets, computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0091] The various processes and features described herein can be implemented in a wide variety of devices or applications, particularly those associated with data encoding, data decoding, view generation, texture processing, and other processing of image and related texture and / or depth information. Examples of such devices include encoders, decoders, post-processors that process the output of decoders, pre-processors that provide input to encoders, video encoders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, cellular phones, PDAs, and other communication devices. It should be clear that the devices can be mobile and may even be installed in mobile vehicles.

[0092] Furthermore, these methods can be implemented by instructions executed by a processor, and such instructions (and / or data values ​​generated by the implementation) can be stored on a processor-readable medium, such as an integrated circuit, software carrier, or other storage device, such as a hard disk, optical disc (“CD”), optical disc (e.g., DVD, often referred to as a digital multifunction disc or digital video disc), random access memory (“RAM”), or read-only memory (“ROM”). Instructions can form an application tangibly contained on the processor-readable medium. Instructions can be, for example, in hardware, firmware, software, or a combination thereof. Instructions can be found, for example, in an operating system, a standalone application, or a combination of both. Thus, a processor can be characterized, for example, as a device configured to execute a process and a device comprising a processor-readable medium (e.g., a storage device) having instructions for executing the process. Furthermore, in addition to or instead of instructions, the processor-readable medium can store data values ​​generated by the implementation.

[0093] As will be evident to one of skill in the art, implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal can be formatted to carry as data the rules for writing or reading the syntax of a described embodiment, or to carry as data the actual syntax values written by a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (for example, using a radio frequency portion of spectrum) or a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The signal that carries the information can be transmitted in, for example, a communications network or a computer bus. The signal can be stored on a processor-readable medium.

[0094] A number of implementations have been described. Nevertheless, it will be understood that various modifications can be made. For example, elements of different implementations can be combined, supplemented, modified, or removed to produce other implementations. Additionally, one of ordinary skill will understand that other structures and processes can be substituted for those disclosed and that the resulting implementations will perform at least substantially the same function(s), in at least substantially the same way(s), to achieve at least substantially the same result(s) as the implementations disclosed. Accordingly, these and other implementations are contemplated by this application.

Claims

1. A method comprising: - de-projecting pixels of a source image into a colored point cloud, the pixels of the source image comprising depth and color attributes, wherein the pixels of the source image are de-projected into points having the color attributes of the pixels, and wherein a point has a score attributed according to a local depth gradient and / or a local color gradient of the pixel, the lower the gradient, the higher the score; - projecting the colored point cloud onto an image to generate the image by blending the color attributes of points projected on the same pixel, the blending being weighted by the score attribute of the points.

2. The method of claim 1, wherein the color of a pixel of the image is the color of the point projected onto the pixel having the highest score.

3. The method of claim 1 or 2, wherein the score of a point is a weighted sum of the inverse of the depth gradient and the inverse of the local color gradient of the de-projected pixel.

4. The method of claim 1, wherein the method comprises decoding the source image from a data stream.

5. The method of claim 4, wherein the method comprises: - decoding from the data stream data representative of a projection mode used to generate the source image; and - using the data to de-project the pixels of the source image into a colored point cloud.

6. The method of one of claims 1-5, wherein the generated image is a viewport image rendered according to a position and orientation of a virtual camera in a 3D space.

7. The method of claim 6, wherein the position and orientation of the virtual camera are controlled by sensors of a head-mounted display device.

8. A device comprising a processor configured to: - de-project pixels of a source image into a colored point cloud, the pixels of the source image comprising depth and color attributes, wherein the pixels of the source image are de-projected into points having the color attributes of the pixels, and wherein a point has a score attributed according to a local depth gradient and / or a local color gradient of the pixel, the lower the gradient, the higher the score; - project the colored point cloud onto an image to generate the image by blending the color attributes of points projected on the same pixel, the blending being weighted by the score attribute of the points.

9. The device of claim 8, wherein the processor is configured to determine the color of a pixel of the image is the color of the point projected onto the pixel having the highest score.

10. The device of claim 8, wherein the processor is configured to determine the score of a point is a weighted sum of the inverse of the depth gradient and the inverse of the local color gradient of the de-projected pixel.

11. The device of claim 8, wherein the processor is configured to decode the source image from a data stream.

12. The device of claim 11, wherein the processor is configured to: - decode from the data stream data representative of a projection mode used to generate the source image; and - use the data to de-project the pixels of the source image into a colored point cloud. ​ 13. The device of one of claims 8 to 12, wherein the generated image is a viewport image rendered according to a position and orientation of a virtual camera in 3D space.

14. The device of claim 13, wherein the position and orientation of the virtual camera are controlled by sensors of a head-mounted display device.

15. A non-transitory processor-readable medium having stored therein instructions for causing a processor to perform the method of one of claims 1 to 7.

Images