Method, device and computer program for projecting a first portion of a three-dimensional (3D) mesh frame onto a two-dimensional (2D) occupancy map - Patents.com
By projecting 3D mesh frames onto 2D occupancy maps using optimized packing techniques, the method addresses inefficiencies in 3D mesh representation, achieving reduced data intensity and improved resource utilization.
Patent Information
- Application Number
- JP2023565409
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2022-10-27
- Filing Date
- 2022-11-02
- Publication Date
- 2024-12-09
- Estimated Expiration
- 2042-11-02
AI Technical Summary
Existing techniques for representing and compressing 3D meshes into 2D space are inefficient, leading to resource-intensive data storage and transmission requirements for immersive applications.
A method for projecting a portion of a 3D mesh frame onto a 2D occupancy map using a correlation matrix calculated via convolution, optimizing packing positions to minimize overlap and reduce data intensity.
This approach enables more efficient data representation and compression of 3D meshes, reducing storage and transmission needs while maintaining immersive quality.
Smart Images

Figure 0007596019000002 
Figure 0007596019000003 
Figure 0007596019000004
Abstract
Description
[Technical field]
[0001] This application claims the benefit of priority to U.S. Non-provisional Application No. 18 / 050,372, entitled "Convolutional Approach to Fast and Compact Packing of 3D Mesh into 2D Maps," filed on October 27, 2022, and U.S. Provisional Application No. 63 / 323,853, entitled "Using Image Filtering for Fast and Compact Packing of Charts," filed on March 25, 2022, which are incorporated by reference in their entireties herein.
[0002] This disclosure relates generally to coding and decoding three-dimensional (3D) meshes, and more particularly to fast and efficient packing of portions of a 3D mesh into a map in two-dimensional (2D) space. [Background technology]
[0003] The discussion of the background art provided herein is intended to generally present the context of the present disclosure. The inventors' work is not admitted expressly or implicitly as prior art to the present disclosure, to the extent that that work is described in this background section, together with aspects of the description that may not otherwise be admitted as prior art at the time of filing of this application.
[0004] Various techniques have been developed to capture, represent, and simulate real-world objects, environments, and the like in 3D space. A 3D representation of the world can enable more immersive interactive communication. Exemplary 3D representations of objects and environments include, but are not limited to, point clouds and meshes. A series of 3D representations of objects and environments may form a video sequence. Redundancy and correlation within a sequence of 3D representations of objects and environments may be exploited to compress and code such video sequences into a more compact digital format. Summary of the Invention [Means for solving the problem]
[0005] This disclosure relates generally to coding and decoding of 3D meshes, and more particularly to fast and efficient packing of portions of a 3D mesh into a UV map om 2D.
[0006] In one example implementation, a method for projecting a first portion of a three-dimensional (3D) mesh frame onto a two-dimensional (2D) occupancy map is disclosed. The method may include obtaining an initial 2D occupancy map, where the initial 2D occupancy map may include zero or more pixels having a first predefined value indicating that the zero or more pixels are mapped to at least one other portion of the 3D mesh frame, where the at least one other portion does not overlap with the first portion of the 3D mesh frame, and remaining pixels having a second predefined value indicating that the remaining pixels are not mapped to the 3D mesh frame. The method may further include the steps of generating a 2D occupancy matrix for a first portion of the 3D mesh frame, where a first predefined value corresponds to an occupied pixel and a second predefined value corresponds to an unoccupied pixel; calculating a correlation matrix between the initial 2D occupancy map and the 2D occupancy matrix; identifying a zero-valued target element in the correlation matrix; and generating an updated 2D occupancy map by replacing pixels having the second predefined value in a patch in the initial 2D occupancy map having the same size as the 2D occupancy matrix with a corresponding element in the 2D occupancy matrix, where the patch is located in the initial 2D occupancy map at a packing position that coincides with the position of the zero-valued target element in the correlation matrix.
[0007] In the above implementation, the correlation matrix is computed as a convolution between the 2D occupancy matrix as the convolution kernel and the initial 2D occupancy map.
[0008] In any one of the above implementations, the convolution is calculated using a Fast Fourier Transform.
[0009] In any one of the above implementations, identifying a zero-valued target element in the correlation matrix may include identifying a plurality of zero-valued elements in the correlation matrix, and selecting one of the plurality of zero-valued elements in the correlation matrix as the zero-valued target element.
[0010] In some of the above implementations, selecting one of the plurality of zero-valued elements in the correlation matrix as the zero-valued target element may include selecting a zero-valued element of the plurality of zero-valued elements in the correlation matrix that has a smallest index in a width dimension.
[0011] In some of the above implementations, selecting one of the plurality of zero-valued elements in the correlation matrix as the zero-valued target element may include selecting a zero-valued element of the plurality of zero-valued elements in the correlation matrix that has a smallest index in a height dimension.
[0012] In some of the above implementations, selecting one of the plurality of zero-valued elements in the correlation matrix as the zero-valued target element may include selecting a zero-valued element of the plurality of zero-valued elements in the correlation matrix that has a minimum sum of a width index and a height index.
[0013] In some of the above implementations, selecting one of the plurality of zero-valued elements in the correlation matrix as the zero-valued target element may include selecting the zero-valued element of the plurality of zero-valued elements based on an optimization function of a height index and a width index.
[0014] In the above implementation, the optimization function is p Includes norm functions.
[0015] In some of the above implementations, selecting one of the plurality of zero-valued elements in the correlation matrix as the zero-valued target element may include determining a preferred packing location of the 2D occupancy matrix into an initial 2D occupancy map, and selecting the zero-valued target element from the plurality of zero-valued elements in the plurality of zero-valued elements in the correlation matrix according to the preferred packing location.
[0016] In some of the above implementations, selecting a zero-valued target element from a plurality of zero-valued elements in a plurality of zero-valued elements in the correlation matrix according to a preferred packing position may include selecting, if possible, as the zero-valued target element from a plurality of zero-valued elements that coincide with the preferred packing position, and otherwise selecting from a plurality of zero-valued elements in a vicinity of the preferred packing position based on an optimization function of the preferred packing position and the positions of the zero-valued elements in the vicinity of the preferred packing position.
[0017] In the above implementation, the optimization function is a function of the position difference between the preferred packing position and the position of a zero-valued element among a plurality of zero-valued elements. p It may also include a norm function.
[0018] In any one of the above implementations, identifying the zero-valued target element in the correlation matrix may include determining a first identified zero-valued element of the correlation matrix as the zero-valued target element, and the method further includes terminating calculation of the correlation matrix upon finding the first identified zero-valued element of the correlation matrix.
[0019] In some examples, the steps of any one of the above implementations may be repeated until all portions of the 3D mesh frame have been mapped to the updated 2D occupancy map.
[0020] In the above implementation, the method may further include compressing the updated 2D occupancy map into a bitstream using a video encoder and compressing information of the sizes and packing positions of the 2D occupancy matrices of all parts of the 3D mesh frame.
[0021] In any one of the above implementations, the initial 2D occupancy map and the 2D occupancy matrix include binary values that include a first predefined value and a second predefined value.
[0022] Aspects of the present disclosure also provide an electronic device or apparatus that includes circuitry configured to perform any of the above method implementations.
[0023] Aspects of the present disclosure also provide a non-transitory computer-readable medium for storing computer instructions that, when executed by a computer for 3D mesh processing, cause the computer to perform any one of the above method implementations.
[0024] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]
[0025] [Figure 1] FIG. 1 is a schematic diagram of a simplified block diagram of an exemplary communication system according to one embodiment of the present disclosure. [Diagram 2] FIG. 1 is a schematic diagram of a simplified block diagram of an exemplary streaming system according to one embodiment of the present disclosure. [Diagram 3] 1 illustrates a data flow for encoding and decoding a 3D mesh or point cloud frame according to some embodiments of the present disclosure. [Figure 4] 1 shows a block diagram of an encoder for encoding a 3D mesh frame or a point cloud frame according to some embodiments of the present disclosure. [Diagram 5]1 shows a block diagram of a decoder for decoding compressed bitstreams corresponding to 3D mesh frames and point cloud frames in accordance with some embodiments of the present disclosure. [Figure 6] FIG. 2 is a schematic diagram of a simplified block diagram of a video decoder according to one embodiment of the present disclosure. [Figure 7] FIG. 2 is a schematic diagram of a simplified block diagram of a video encoder according to one embodiment of the present disclosure. [Figure 8] 1 illustrates an example implementation for packing multiple charts of a 3D mesh into a 2D map, according to some embodiments of the present disclosure. [Figure 9] 1 illustrates another example implementation for packing multiple charts of a 3D mesh into a 2D map, according to some embodiments of the present disclosure. [Figure 10] 1 illustrates an example of efficient and compact packing of a chart of a 3D mesh into a 2D map, according to some embodiments of the present disclosure. [Figure 11] 1 illustrates an example 2D occupancy map for a chart of a 3D mesh, according to some embodiments of the present disclosure. [Figure 12] 1 shows an example flow chart illustrating an example process according to some embodiments of the present disclosure. [Figure 13] FIG. 1 is a schematic diagram of an exemplary computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] Throughout this specification and the claims, terms may have subtle meanings that are suggested or implied in the context beyond the meaning explicitly stated. The phrases "in one embodiment" or "in some embodiments" used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" used herein do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in other implementations" used herein do not necessarily refer to different implementations. For example, the claimed subject matter is intended to include all or part of the exemplary embodiments / implementations.
[0027] Generally, terms may be understood at least in part from their usage in context. For example, terms such as "and", "or", or "and / or" as used herein may include various meanings that may depend at least in part on the context in which such terms are used. Typically, "or" when used to relate a list such as A, B, or C is intended to mean A, B, and C, which are used herein in an inclusive sense, as well as A, B, or C, which are used herein in an exclusive sense. In addition, the terms "one or more" or "at least one" as used herein may be used to describe any feature, structure, or characteristic in a singular sense, or may be used to describe a combination of features, structures, or characteristics in a plural sense, depending at least in part on the context. Similarly, terms such as "a", "an", or "the" may also be understood to convey a singular usage or a plural usage, depending at least in part on the context. In addition, the terms "based on" or "determined by" may be understood as not necessarily intended to convey an exclusive set of factors, but instead may allow for the existence of additional factors not necessarily expressly recited, also depending at least in part on the context.
[0028] Technological developments in 3D media processing, such as advances in 3D capture, 3D modeling, and 3D rendering, have facilitated the ubiquitous creation of 3D content across several platforms and devices. Such 3D content includes, for example, information that can be processed to generate various forms of media to provide immersive viewing / rendering and interactive experiences. Applications of 3D content are abundant and include, but are not limited to, virtual reality, augmented reality, metaverse interaction, gaming, immersive video conferencing, robotics, computer-aided design (CAD), and the like. According to one aspect of the present disclosure, to improve the immersive experience, 3D models are becoming more and more sophisticated, and the creation and consumption of 3D models demands a significant amount of data resources, such as data storage, data transmission resources, data processing resources, and the like.
[0029] Compared to traditional two-dimensional (2D) content (such as images) that are typically represented by datasets in the form of 2D pixel arrays, 3D content with three-dimensional full resolution pixelation may be prohibitively resource intensive, but is unnecessary in many, if not most, practical applications. In most 3D immersive applications, less data-intensive representations of 3D content may be employed according to some aspects of the present disclosure. For example, most applications may require only topographical information, not volumetric information, of objects in a 3D scene (either a real-world scene captured by a sensor such as a LIDAR device or an animated 3D scene generated by a software tool). Thus, more efficient forms of datasets may be used to represent 3D objects and 3D scenes. For example, 3D meshes may be used as a type of 3D model to represent immersive 3D content, such as 3D objects in a 3D scene.
[0030] A mesh (or alternatively called a mesh model) of one or more objects may include a collection of vertices. The vertices may connect together to form edges. The edges may further connect to form faces. The faces may further form polygons. 3D surfaces of various objects may be decomposed, for example, into faces and polygons. Each of the vertices, edges, faces, polygons, or surfaces may be associated with various attributes, such as color, surface normal, texture, and the like. The normal of a surface may be referred to as a surface normal, and / or the normal of a vertex may be referred to as a vertex normal. Information about how vertices are connected into an edge, face, or polygon may be referred to as connectivity information. The same set of vertices may form different faces, surfaces, and polygons, so connectivity information is important to uniquely define components of a mesh. In general, the location of a vertex in 3D space may be represented by its 3D coordinates. A face may be represented by a set of sequentially connected vertices, each associated with a set of 3D coordinates. Similarly, an edge may be represented by two vertices, each associated with its 3D coordinates. The vertices, edges, and faces can be indexed in the 3D mesh data set.
[0031] A mesh may be defined and described by a collection of one or more of these basic element types. However, not all of the above types of elements are required to fully describe a mesh. For example, a mesh may be fully described by using only vertices and vertex connectivity. In another example, a mesh may be fully described by using only faces and a list of their common vertices. Thus, a mesh may be of a variety of alternative types described by alternative dataset configurations and formats. Examples of mesh types include, but are not limited to, face-vertex meshes, winged edge meshes, half edge meshes, quad edge meshes, corner table meshes, vertex-vertex meshes, etc. Correspondingly, the mesh dataset may be stored with information conforming to alternative file formats having file extensions including, but not limited to, .raw, .blend, .fbx, .3ds, .dae, .dng, 3dm, .dsf, .dwg, .obj, .ply, .pmd, .stl, amf, .wrl, .wrz, .x3d, .x3db, .x3dv, .x3dz, .x3dbz, .x3dvz, .c4d, .lwo, .smb, .msh, .mesh, .veg, .z3d, .vtk, .l4d, etc. Attributes of these elements, such as color, surface normals, textures, etc., may be included in the mesh dataset in a variety of ways.
[0032] In some implementations, the vertices of a mesh may be mapped to a pixelated 2D space called UV space. Thus, each vertex of a mesh may be mapped to a pixel in UV space. In some implementations, a vertex may be mapped to multiple pixels in UV space, for example, a boundary vertex may be mapped to two or three pixels in UV space. Similarly, a face or surface in a mesh may be sampled to multiple 3D points that may or may not be among the recorded vertices in the mesh, and these multiple 3D points may also be mapped to pixels in the two-dimensional UV space. The mapping of the vertices and sampled 3D points of a face or surface in a mesh to UV space, and subsequent data analysis and processing in UV space, may facilitate data storage, compression, and coding of a 3D dataset of a mesh or sequence of meshes, as described in more detail below. The mapped UV space dataset may be referred to as a UV image, or a 2D map, or a 2D image of the mesh.
[0033] Following mapping of the vertices and sampled surface points in the 3D mesh to the 2D UV space, some pixels may be mapped to the vertices and sampled surface points of the 3D mesh, while other pixels may not (or may not) be mapped. Each mapped pixel in the 2D image of the mesh may be associated with information of the corresponding mapped vertex or surface point in the 3D mesh. Depending on the type of information contained for a pixel in UV space, various 2D images or maps of the mesh may be constructed. A collection of multiple 2D maps may be used as alternative and / or combined representations of the mesh.
[0034] For example, the simplest 2D map of a mesh may be configured as an occupancy map. The occupancy map may indicate which pixels in UV space are mapped to the 3D vertices or sampled surface points of the mesh. The indication of occupancy may be represented by a binary indicator in each of the 2D pixels, for example with a binary value of "1" indicating mapping or occupancy and a binary value of "0" indicating non-mapping or non-occupied (not mapped). Thus, the occupancy map may be configured as a 2D image. While a normal 2D image includes an array of three channels (RGB, YUV, YCrCb, etc.) with, for example, 8-bit color depth, such a 2D occupancy map of a mesh only requires a single-bit binary channel. A pixel in the 2D image with a value of "1" is mapped and occupied, otherwise the pixel is not mapped and unoccupied.
[0035] In another example, a 2D geometry map may be constructed for a mesh that, instead of containing a single binary channel, becomes a full three-channel image in which the three color channels at each occupied pixel correspond to the three 3D coordinates of the corresponding mapped vertex or sampled 3D point in the mesh.
[0036] In some implementations, other 2D maps may be constructed for the mesh. For example, a set of attributes for each of the vertices and sampled 3D points of the mesh may be extracted from the mesh dataset and coded into three color channels of a 2D map image. Such a 2D map may be referred to as an attribute map of the mesh. A particular attribute map may include three channels of color for each of the occupied pixels in UV space. In another example, texture attributes associated with each mapped vertex or sampled 3D point of the mesh may be parameterized into a three-channel value and coded into a 2D attribute map. In another example, regular attributes associated with each mapped vertex or sampled 3D point of the mesh may be parameterized into a three-channel value and coded into a 2D attribute map. In some example implementations, multiple 2D attribute maps may be constructed to hold all the necessary attribute information for the vertices and sampled surface points of the mesh.
[0037] The above 2D maps are merely examples. Other types of 2D maps for a mesh may be constructed. In addition, other data sets may be extracted from the 3D mesh to represent the original 3D mesh together with the above 2D maps. For example, the connections or connectivity information between vertices may be grouped and organized separately from the 2D map in the form of a list, table, or the like. The connectivity information may, for example, reference the vertices using a vertex index. The vertex index may be mapped to a corresponding pixel location in the 2D map. In another example, surface texture, color, normal, displacement, and other information may be extracted and organized separately from the 2D map, rather than as a 2D map. Other metadata may be further extracted from the 3D mesh to represent the 3D mesh together with the 2D map and the other data sets described above.
[0038] Although the above example implementations focus on static meshes, according to one aspect of the present disclosure, the 3D mesh may be dynamic. A dynamic mesh may refer to a mesh in which at least one of its components (geometry information, connectivity information, mapping information, vertex attributes, and attribute maps) changes over time, for example. Thus, a dynamic mesh may be described by a sequence of meshes or meshes (also called mesh frames), similar to a timed sequence of 2D image frames that form a video.
[0039] In some example implementations, the dynamic mesh may have constant connectivity information, time-varying geometry, and time-varying vertex attributes. In some other examples, the dynamic mesh may have time-varying connectivity information. In some examples, digital 3D content creation tools may be used to generate the dynamic mesh with time-varying attribute maps and time-varying connectivity information. In some other examples, volume acquisition / detection / sensing techniques are used to generate the dynamic mesh. The volume acquisition techniques can generate the dynamic mesh with time-varying connectivity information, especially under real-time constraints.
[0040] Dynamic meshes may require large amounts of data since dynamic meshes may contain a significant amount of information that changes over time. However, compression may be performed to exploit redundancy within mesh frames (intra-compression) and between mesh frames (inter-compression). Various mesh compression processes may be implemented to allow efficient storage and transmission of media content in mesh representations, particularly for mesh sequences.
[0041] Aspects of the present disclosure provide example architectures and techniques for mesh compression. These techniques may be used for various mesh compression including, but not limited to, static mesh compression, dynamic mesh compression, compression of dynamic meshes with constant connectivity information, compression of dynamic meshes with time-varying connectivity information, compression of dynamic meshes with time-varying attribute maps, etc. These techniques may be used for lossy and lossless compression for various applications such as real-time immersive communication, storage, free viewpoint video, augmented reality (AR), virtual reality (VR), etc. These applications may include features such as random access and scalable / progressive coding.
[0042] Although this disclosure explicitly describes techniques and implementations applicable to 3D meshes, the principles underlying the various implementations described herein are applicable to other types of 3D data structures, including but not limited to Point Cloud (PC) data structures. For simplicity, references below to 3D meshes are intended to be general and include other types of 3D representations, such as point clouds and other 3D volumetric datasets.
[0043] Turning first to an exemplary architecture-level implementation, FIG. 1 illustrates a simplified block diagram of a communication system (100) according to an exemplary embodiment of the present disclosure. The communication system (100) may include multiple terminal devices that can communicate with each other, for example, via a communication network (150) (alternatively referred to as a network). For example, the communication system (100) may include a pair of terminal devices (110) and (120) interconnected via the network (150). In the example of FIG. 1, a first pair of terminal devices (110) and (120) may perform unidirectional transmission of a 3D mesh. For example, the terminal device (110) may compress a 3D mesh or a sequence of 3D meshes that may be generated by the terminal device (110), retrieved from storage, or captured by a 3D sensor (105) connected to the terminal device (110). The compressed 3D mesh or sequence of 3D meshes may be transmitted to the other terminal device (120) via the network (150), for example in the form of a bitstream (also called a coded bitstream). The terminal device (120) may receive the compressed 3D mesh or sequence of 3D meshes from the network (150), decompress the bitstream to reconstruct the original 3D mesh or sequence of 3D meshes, and process the reconstructed 3D mesh or sequence of 3D meshes appropriately for display or other purposes / uses. Unidirectional data transmission may be common, such as in media serving applications.
[0044] In the example of FIG. 1, either or both of the terminal devices (110) and (120) may be implemented as servers, fixed or mobile personal computers, laptop computers, tablet computers, smart phones, gaming consoles, media players, and / or dedicated three-dimensional (3D) appliances, etc., although the principles of the present disclosure are not so limited. The network (150) may represent any type of network or combination of networks that transmits the compressed 3D mesh between the terminal devices (110) and (120). The network (150) may include, for example, wired (wired connection) and / or wireless communication networks. The network (150) may exchange data over circuit-switched and / or packet-switched channels. Representative networks include long-haul telecommunications networks, local area networks, wide area networks, cellular networks, and / or the Internet. For purposes of the present disclosure, the architecture and topology of the network (150) may not be important to the operation of the present disclosure, unless otherwise described herein below.
[0045] 2 illustrates an exemplary simplified block diagram of a streaming system (200) according to one embodiment of the present disclosure. FIG. 2 illustrates an exemplary application for implementations of the disclosure related to 3D meshes and compressed 3D meshes. The disclosed subject matter may be equally applicable to other 3D mesh or point cloud enabled applications, such as 3D telepresence applications, virtual reality applications, etc.
[0046] The streaming system (200) may include a capture or storage subsystem (213). The capture or storage subsystem (213) may include a 3D mesh generator or storage medium (201), e.g., a 3D mesh or point cloud generation tool / software, a graphics generation component, or a point cloud sensor, such as a light detection and ranging (LIDAR) system, a 3D camera, a 3D scanner, a 3D mesh store, etc., that generates or provides an uncompressed 3D mesh (202) or point cloud (202). In some example implementations, the 3D mesh (202) includes the vertices of a 3D mesh or the 3D points of a point cloud (both referred to as a 3D mesh). The 3D mesh (202) is illustrated with a bold line to emphasize the high amount of data compared to the compressed 3D mesh (204) (compressed 3D mesh bitstream). The compressed 3D mesh (204) may be generated by an electronic device (220) including an encoder (203) coupled to the 3D mesh (202). The encoder (203) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The compressed 3D mesh (204) (or a bitstream of the compressed 3D mesh (204)), illustrated as a thin line to emphasize its low data volume compared to the stream of the uncompressed 3D mesh (202), may be stored in a streaming server (205) for future use. One or more streaming client subsystems, such as the client subsystems (206) and (208) of FIG. 2, may access the streaming server (205) to retrieve copies (207) and (209) of the compressed 3D mesh (204). The client subsystem (206) may include, for example, a decoder (210) in the electronic device (230).The decoder (210) may be configured to decode an input compressed copy of the 3D mesh (207) and create an output stream of reconstructed 3D meshes (211) that can be rendered on a rendering device (212) or for other uses.
[0047] It should be noted that the electronic devices (220) and (230) may include other components (not shown). For example, the electronic device (220) may also include a decoder (not shown), and the electronic device (230) may also include an encoder (not shown).
[0048] In some streaming systems, the compressed 3D meshes (204), (207), and (209) (e.g., bitstreams of the compressed 3D meshes) may be compressed according to a particular standard. In some examples, as described in more detail below, a video coding standard is used to exploit redundancy and correlation in compressing the 3D mesh after the 3D mesh is first projected and mapped to a 2D representation suitable for video compression. Non-limiting examples of such standards include High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), etc., as described in more detail below.
[0049] A compressed 3D mesh or sequence of 3D meshes may be generated by an encoder, while a decoder may be configured to decompress the compressed or coded 3D meshes. FIG. 3 illustrates a high-level exemplary data flow of 3D meshes in such an encoder (301) and decoder (303). As illustrated in FIG. 3, a raw input 3D mesh or sequence of 3D meshes (302) may be pre-processed by track remeshing, parameterization, and / or voxelization to generate input data to a mapping unit (304) for mapping the 3D mesh into a 2D UV space, which may in some implementations include a mesh with a UV atlas. The 3D mesh may be sampled to include 3D surface points that may not be among the vertices, and these sampled 3D surface points may be added to the UV space in the mapping. Various 2D maps may be generated in the encoder 301, including, but not limited to, an occupancy map (310), a geometry map (312), and an attribute map (314). These image type maps may be compressed by the encoder 301 using, for example, video coding / compression techniques. For example, a video coder may assist in compressing 3D mesh frames using intra prediction techniques and inter prediction with other 3D mesh reference frames. Other non-image or non-map data or metadata (316) may also be coded in various ways to remove redundancy via entropy coding to generate compressed non-map data, as a non-limiting example. The encoder 301 may then combine or multiplex the compressed 2D map and non-map data and further code the combined data to generate an encoded bitstream (alternatively referred to as a coded bitstream). The encoded bitstream may then be stored or transmitted for use by the decoder 303.The decoder may be configured to decode the bitstream, demultiplex the decoded bitstream to obtain compressed 2D maps and non-mapped data, and perform decompression to generate a decoded occupancy map (320), a decoded geometry map (322), a decoded attribute map (324), and decoded non-mapped data and metadata (326). The decoder 303 may then be further configured to reconstruct a 3D mesh or a sequence of 3D meshes (330) from the decoded 2D maps (320, 322, 324) and the decoded non-mapped data (326).
[0050] In more detail, Figure 4 shows a block diagram of an example 3D mesh encoder (400) for encoding 3D mesh frames in accordance with some embodiments of the present disclosure. In some example implementations, the mesh encoder (400) may be used in the communication system (100) and the streaming system (200). For example, the encoder (203) may be configured and operate similarly to the mesh encoder (400).
[0051] The mesh encoder (400) may receive a 3D mesh frame as an uncompressed input and generate a bitstream corresponding to the compressed 3D mesh frame. In some example implementations, the mesh encoder (400) may receive the 3D mesh frame from any source, such as the mesh or point cloud source (201) of FIG. 2.
[0052] In the example of FIG. 4, the mesh encoder (400) may include a patch generation module (406) (alternatively referred to as a chart generation module), a patch packing module (408), a geometry image generation module (410), a texture image generation module (412), a patch information module (404), an occupancy map module (414), a smoothing module (436), image padding modules (416) and (418), a group expansion module (420), video compression modules (422), (423), and (432), an auxiliary patch information compression module (438), an entropy compression module (434), and a multiplexer (424).
[0053] In various embodiments of the present disclosure, a module may refer to a software module, a hardware module, or a combination thereof. A software module may include a computer program or a portion of a computer program that has a predefined function and cooperates with other related parts to achieve a predefined goal, such as the function described in this disclosure. A hardware module may be implemented using a processing circuit and / or memory configured to perform the function described in this disclosure. Each module may be implemented using one or more processors (or a processor and memory). Similarly, one or more modules may be implemented using a processor (or a processor and memory). Furthermore, each module may be a part of an entire module that includes the functionality of the module. The description herein may also apply to the term module and other equivalent terms (e.g., unit).
[0054] According to one aspect of the disclosure, as described above, the mesh encoder (400) converts the 3D mesh frame into an image-based representation (e.g., a 2D map) along with any non-map metadata (e.g., patch or chart information) used to aid in converting the compressed 3D mesh back to a decompressed 3D mesh. In some examples, the mesh encoder (400) may convert the 3D mesh frame into a 2D geometry map or image, texture map or image, and occupancy map or image, and then encode the geometry image, texture image, and occupancy map along with metadata and other compressed non-map data into a bitstream using video coding techniques. In general, as described above, a 2D geometry image is a 2D image having 2D pixels filled with geometry values associated with 3D points projected (the term "projected" is used to mean "mapped") onto the 2D pixels, where the 2D pixels filled with geometry values may be referred to as geometry samples. A texture image is a 2D image having pixels filled with texture values associated with 3D points projected onto the 2D pixels, where the 2D pixels filled with texture values may be referred to as texture samples. An occupancy map is a 2D image with 2D pixels filled with values that indicate occupation or non-occupancy by 3D points.
[0055] The patch generation module (406) segments the 3D mesh into a set of overlapping or non-overlapping charts or patches (e.g., a patch is defined as a contiguous subset of a surface described by a 3D mesh or point cloud), such that each patch can be described by a depth field relative to a plane in 2D space (e.g., by flattening the surface so that deeper 3D points on the surface are further away from the center of the corresponding 2D map). In some embodiments, the patch generation module (406) aims to decompose the 3D mesh into a minimum number of patches with smooth boundaries while also minimizing the reconstruction error.
[0056] The patch information module (404) may collect patch information indicating the size and shape of the patch. In some examples, the patch information may be packed into a data frame and then encoded by the auxiliary patch information compression module (438) to generate compressed auxiliary patch information. The auxiliary patch compression may be implemented in a variety of forms, including, but not limited to, various types of arithmetic coding.
[0057] The patch or chart packing module (408) may be configured to map the extracted patches onto a 2D grid in UV space while minimizing unused space. In some example implementations, pixels in the 2D UV space may be granulated into blocks of pixels for mapping of patches or charts. The block size may be predefined. For example, the block size may be M×M (e.g., 16×16). With such granularity, it may be guaranteed that every M×M block of the 2D UV grid is associated with a unique patch. In other words, each patch is mapped into the 2D UV space with a 2D granularity of M×M. Efficient patch packing can directly impact compression efficiency by either minimizing unused space or ensuring temporal consistency. An example implementation of packing patches or charts into 2D UV space is given in further detail below.
[0058] The geometry image generation module (410) can generate a 2D geometry image associated with the geometry of the 3D mesh at a given patch location in the 2D grid. The texture image generation module (412) can generate a 2D texture image associated with the texture of the 3D mesh at a given patch location in the 2D grid. The geometry image generation module (410) and the texture image generation module (412) essentially store the geometry and texture of the 3D mesh as a 2D image using the 3D to 2D mapping calculated during the packing process described above, as described above. In some implementations, the 2D images can be layered to better handle cases where multiple points are projected onto the same sample (e.g., patches overlap in the 3D space of the mesh). In other words, each patch can be projected onto, for example, two images, called layers, such that multiple points can be projected onto the same point in different layers.
[0059] In some example implementations, the geometry image may be represented by a monochromatic frame of width x height (W x H). Thus, three geometry images of three luma or chroma channels may be used to represent the 3D coordinates. In some example implementations, the geometry image may be represented by a 2D image with three channels (RGB, YUV, YCrCb, etc.) with a particular color depth (e.g., 8 bits, 12 bits, 16 bits, etc.). Thus, one geometry image with three color channels may be used to represent the 3D coordinates.
[0060] To generate a texture image, the texture generation procedure utilizes the reconstructed / smoothed geometry to calculate colors to be associated with sampled points from the original 3D mesh (e.g., see "Sampling" in Figure 3, which will generate 3D surface points that are not among the vertices of the original 3D mesh).
[0061] The occupancy map module (414) may be configured to generate an occupancy map that describes the padding information in each unit. For example, as described above, the occupancy image may include a binary map that indicates, for each cell of a 2D grid, whether the cell belongs to an empty space or to a 3D mesh. In some example implementations, the occupancy map may use binary information to describe, on a pixel-by-pixel basis, whether the pixel is padded or not. In some other example implementations, the occupancy map may use binary information to describe, on a block-by-block basis, whether the block-of-pixels is padded or not (e.g., each M×M block).
[0062] The occupancy map generated by the occupancy map module (414) may be compressed using lossless or lossy coding. If lossless coding is used, an entropy compression module (434) may be used to compress the occupancy map. If lossy coding is used, a video compression module (432) may be used to compress the occupancy map.
[0063] Note that the patch packing module (408) may leave some empty space between the packed 2D patches in the image frame. The image padding modules (416) and (418) may fill the empty space (called padding) to generate an image frame that may be suitable for 2D video and image codecs. Image padding, also known as background filling, may fill unused space with redundant information. In some examples, well-implemented background filling minimally increases bitrate while avoiding introducing significant coding artifacts around patch boundaries.
[0064] The video compression modules (422), (423), and (432) can encode 2D images, such as padded geometry images, padded texture images, occupancy maps, etc., based on an appropriate video coding standard, such as HEVC, VVC, etc. In some example implementations, the video compression modules (422), (423), and (432) are individual components that operate separately. It should be noted that the video compression modules (422), (423), and (432) can be implemented as a single component in some other example implementations.
[0065] In some example implementations, the smoothing module (436) may be configured to generate a smoothed image of the reconstructed geometry image. The smoothed image may be provided to the texture image generation (412). The texture image generation (412) may then adjust the generation of the texture image based on the reconstructed geometry image. For example, if the patch shape (e.g., geometry) is slightly distorted during encoding and decoding, the distortion may be taken into account when generating the texture image to correct for the distortion in the patch shape.
[0066] In some embodiments, the group expansion (420) is configured to pad pixels around object boundaries with redundant low-frequency content to improve the coding gain as well as the visual quality of the reconstructed 3D mesh.
[0067] A multiplexer (424) may be configured to multiplex the compressed geometry images, the compressed texture images, the compressed occupancy maps, and the compressed auxiliary patch information into a compressed bitstream.
[0068] 5 shows a block diagram of an example mesh decoder (500) for decoding a compressed bitstream corresponding to a 3D mesh frame according to some embodiments of the present disclosure. In some example implementations, the mesh decoder (500) can be used in the communication system (100) and the streaming system (200). For example, the decoder (210) can be configured to operate similarly to the mesh decoder (500). The mesh decoder (500) receives a compressed bitstream and generates a reconstructed 3D mesh based on the compressed bitstream, including, for example, compressed geometry images, compressed texture images, compressed occupancy maps, and compressed auxiliary patch information.
[0069] In the example of FIG. 5, the mesh decoder (500) may include a demultiplexer (532), video decompression modules (534) and (536), an occupancy map decompression module (538), an auxiliary patch information decompression module (542), a geometry reconstruction module (544), a smoothing module (546), a texture reconstruction module (548), and a color smoothing module (552).
[0070] A demultiplexer (532) may receive the compressed bitstream and separate it into compressed texture images, compressed geometry images, compressed occupancy maps, and compressed auxiliary patch information.
[0071] The video decompression modules (534) and (536) may decode compressed images according to an appropriate standard (e.g., HEVC, VVC, etc.) and output decompressed images. For example, the video decompression module (534) may decode compressed texture images and output decompressed texture images. The video decompression module (536) may further decode compressed geometry images and output decompressed geometry images.
[0072] The occupancy map decompression module (538) may be configured to decode the compressed occupancy map according to an appropriate standard (eg, HEVC, VVC, etc.) and output a decompressed occupancy map.
[0073] The auxiliary patch information decompression module (542) may be configured to decode the compressed auxiliary patch information according to a suitable decoding algorithm and output decompressed auxiliary patch information.
[0074] A geometry reconstruction module (544) may be configured to receive the decompressed geometry image and generate a reconstructed 3D mesh geometry based on the decompressed occupancy map and the decompressed auxiliary patch information.
[0075] The smoothing module (546) may be configured to smooth discontinuities at the edges of the patches. The smoothing procedure may aim to mitigate potential discontinuities that may occur at the patch boundaries due to compression artifacts. In some example implementations, a smoothing filter may be applied to pixels located on the patch boundaries to reduce distortions that may be caused by compression / decompression.
[0076] A texture reconstruction module (548) may be configured to determine texture information for points in the 3D mesh based on the decompressed texture image and the smoothed geometry.
[0077] The color smoothing module (552) may be configured to smooth out color inconsistencies. Patches that are not neighbors in 3D space are often packed adjacent to each other in 2D video. In some instances, pixel values from non-neighboring patches may be confused by a block-based video codec. The goal of color smoothing may be to reduce visible artifacts that appear at patch boundaries.
[0078] 6 shows a block diagram of an exemplary video decoder (610) according to one embodiment of the present disclosure. The video decoder (610) may be used in the mesh decoder (500). For example, the video decompression modules (534) and (536) and the occupancy map decompression module (538) may be configured similarly to the video decoder (610).
[0079] The video decoder (610) may include a parser (620) for reconstructing symbols (621) from a compressed image, such as a coded video sequence. The categories of those symbols may include information used to manage the operation of the video decoder (610). The parser (620) may parse / entropy decode a received coded video sequence. The coding of the coded video sequence may be according to a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context dependency, etc. The parser (620) may extract from the coded video sequence a set of at least one subgroup parameter for a subgroup of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (620) may also extract information from the coded video sequence, such as transform coefficients, quantization parameter values, motion vectors, and the like.
[0080] The parser (620) may perform an entropy decoding / parsing operation on the image sequence received from the buffer memory to produce symbols (621).
[0081] The reconstruction of the symbols (621) may require a number of different units depending on the type of coded video picture or portion thereof (inter-picture and intra-picture, inter-block and intra-block, etc.), as well as other factors. Which units are required and how may be controlled by subgroup control information parsed from the coded video sequence by the parser (620). The flow of such subgroup control information between the parser (620) and the following units is not shown for the sake of simplicity.
[0082] Besides the functional blocks already mentioned, the video decoder (610) can be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these units will interact closely with each other and may be, at least in part, integrated with each other. The following conceptual subdivision into functional units is made solely for purposes of illustrating the subject matter of the disclosure.
[0083] The video decoder (610) may include a scaler / inverse transform unit (651). The scaler / inverse transform unit (651) may receive quantized transform coefficients as well as control information including which transform to use, block size, quantization coefficients / parameters, quantization scaling matrices, etc. as symbol(s) (621) from the parser (620). The scaler / inverse transform unit (651) may output blocks including sample values that can be input to an aggregator (655).
[0084] In some cases, the output samples of the scalar / inverse transform (651) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (652). In some cases, the intra-picture prediction unit (652) may generate a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from a current picture buffer (658). The current picture buffer (658) may, for example, buffer the partially reconstructed current picture and / or the fully reconstructed current picture. The aggregator (655) may, in some cases, add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (652) to the output sample information as provided by the scalar / inverse transform unit (651).
[0085] In other cases, the output samples of the scalar / inverse transform unit (651) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensated prediction unit (653) may access the reference picture memory (657) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (621) relating to the block, these samples may be added to the output of the scalar / inverse transform unit (651) by the aggregator (655) to generate output sample information (in this case referred to as residual samples or residual signals). The addresses in the reference picture memory (657) from which the motion compensated prediction unit (653) fetches prediction samples may be controlled by motion vectors available to the motion compensated prediction unit (653) in the form of symbols (621), which may have, for example, an X component, a Y component, and a reference picture component. Motion compensation may also include interpolation of sample values fetched from the reference picture memory (657) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.
[0086] The output samples of the aggregator (655) may be subjected to various loop filtering techniques in a loop filter unit (656). Video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also called the coded video bitstream) and provided to the loop filter unit (656) as symbols (621) from the parser (620), but may also be responsive to meta-information obtained during decoding of a coded picture or previous portion (in decoding order) of the coded video sequence, or may be responsive to previously reconstructed and loop filtered sample values.
[0087] The output of the loop filter unit (656) may be a sample stream that can be output to a rendering device or stored in a reference picture memory (657) for use in future inter-picture prediction.
[0088] Once a particular coded picture is fully reconstructed, it may be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (620)), the current picture buffer (658) may become part of the reference picture memory (657), and a new current picture buffer may be reallocated before beginning reconstruction of a subsequent coded picture.
[0089] The video decoder (610) may perform decoding operations according to a given video compression technique of a standard, such as ITU-T Recommendation H.265. The coded video sequence may comply with a syntax specified by the video compression technique or standard being used in the sense that the coded video sequence adheres to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, the profile may select a particular tool from all tools available in the video compression technique or standard as the limited tools available under that profile. It may also be required for compliance that the complexity of the coded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited in some cases by the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled within the coded video sequence.
[0090] 7 shows a block diagram of a video encoder (703) according to one embodiment of the disclosure. The video encoder (703) can be used in the mesh encoder (400) to compress a 3D mesh or point cloud. In some example implementations, the video compression modules (422) and (423) and the video compression module (432) are configured similarly to the encoder (703).
[0091] The video encoder (703) may receive 2D images, such as padded geometry images, padded texture images, etc., and generate compressed images.
[0092] According to an exemplary embodiment of the present disclosure, the video encoder (703) may code and compress pictures (images) of a source video sequence into a coded video sequence (compressed images) in real time or under any other time constraint required by the application. Enforcing an appropriate coding rate is one function of the controller (750). In some embodiments, the controller (750) controls and is operatively coupled to other functional units described below. Coupling is not shown for simplicity. Parameters set by the controller (750) may include rate control related parameters (picture skip, quantization, lambda value for rate distortion optimization techniques, ...), picture size, Group of Pictures (GOP) layout, maximum motion vector search range, etc. The controller (750) may be configured with other appropriate functions related to the video encoder (703) optimized for a particular system design.
[0093] In some example implementations, the video encoder (703) may be configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop may include a source coder (730) (e.g., responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder (733) embedded in the video encoder (703). The decoder (733) may reconstruct the symbols to create sample data in a manner similar to what a (remote) decoder would also create (since any compression between the symbols and the coded video bitstream is lossless in the video compression techniques contemplated by the disclosed subject matter). The reconstructed sample stream (sample data) may be input to a reference picture memory (734). Since decoding of the symbol stream results in bit-exact results regardless of the location of the decoder (local or remote), the contents in the reference picture memory (734) are also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values as the reference picture samples that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel error) is also used in several related technologies.
[0094] The operation of the "local" decoder (733) may be the same as the operation of a "remote" decoder, such as the video decoder (610), already described in detail above in conjunction with Figure 6. Referring also briefly to Figure 6, however, because symbols are available and the encoding / decoding of the symbol-pair coded video sequence by the entropy coder (745) and parser (620) may be lossless, the entropy decoding portion of the video decoder (610), including the parser (620), may not be fully implemented in the local decoder (733).
[0095] In various embodiments of the present disclosure, any decoder technique, other than parsing / entropy decoding, present in a decoder may necessarily need to be present in a corresponding encoder in substantially the same functional form. For this reason, the subject matter of the disclosure focuses on the decoder operation. Descriptions of the encoder techniques may be omitted, since they are the opposite of the decoder techniques described generically. Only in certain areas are more detailed descriptions required and are provided below.
[0096] In some examples, during operation, the source coder (730) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (732) may code differences between pixel blocks of the input picture and pixel blocks of the reference picture(s) that may be selected as the predictive reference(s) to the input picture.
[0097] The local video decoder (733) may decode the coded video data of pictures that may be designated as reference pictures based on symbols created by the source coder (730). The operation of the coding engine (732) may advantageously be a lossy process. If the coded video data may be decoded in a video decoder (not shown in FIG. 7), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (733) may replicate the decoding process that may be performed by the video decoder on the reference pictures and store the reconstructed reference pictures in a reference picture cache (734). In this way, the video encoder (703) may locally store copies of reconstructed reference pictures that have common content with the reconstructed reference pictures obtained by the far-end video decoder (without transmission errors).
[0098] The predictor (735) may perform a prediction search for the coding engine (732). That is, for a new picture to be coded, the predictor (735) may search the reference picture memory (734) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (735) may operate pixel block by pixel block to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (735), the input picture may have a prediction reference chosen from multiple reference pictures stored in the reference picture memory (734).
[0099] The controller (750) may manage the coding operations of the source coder (730), including, for example, setting the parameters and subgroup parameters used to encode the video data.
[0100] The output of all of the aforementioned functional units may be subjected to entropy coding in an entropy coder (745), which may convert the symbols produced by the various functional units into a coded video sequence by lossless compression of the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.
[0101] The controller (750) may manage the operation of the video encoder (703). The controller (750) may assign a particular coded picture type to each coded picture during coding, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:
[0102] An intra picture (I-picture) may be a picture that can be coded and decoded without using other pictures in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures, as well as their respective uses and characteristics.
[0103] A predictive picture (P picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which predicts sample values for each block using at most one motion vector and reference index.
[0104] A bidirectionally predicted picture (B-picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which predicts sample values for each block using at most two motion vectors and reference indexes. Similarly, a multi-predictive picture can use more than two reference pictures and associated metadata for the reconstruction of a single block.
[0105] A source picture is generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. A block may be predictively coded with reference to other (already coded) blocks as determined by a coding assignment applied to the block's respective picture. For example, a block of an I picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). A pixel block of a P picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. A block of a B picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.
[0106] The video encoder (703) may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Recommendation H.265. In its operations, the video encoder (703) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.
[0107] Video may be in the form of multiple source pictures (images) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits the (temporal or other) correlation between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.
[0108] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, both of which precede the current picture in the video in decoding order (but may be in the past and future, respectively, in display order), are used. A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be predicted by a combination of the first reference block and the second reference block.
[0109] Furthermore, merge mode techniques can be used to improve coding efficiency in inter-picture prediction.
[0110] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed on a block-by-block basis. For example, according to the HEVC standard, a picture in a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be partitioned into one CU of 64×64 pixels, or into four CUs of 32×32 pixels, or into 16 CUs of 16×16 pixels. In one example, each CU is analyzed to determine a prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) according to temporal predictability and / or spatial predictability. In general, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values (e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.
[0111] In various embodiments, the mesh encoder (400) and mesh decoder (500) described above can be implemented using hardware, software, or a combination thereof. For example, the mesh encoder (400) and mesh decoder (500) can be implemented using processing circuitry, such as one or more integrated circuits (ICs) that operate with or without software, such as application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), etc. In another example, the mesh encoder (400) and mesh decoder (500) can be implemented as software or firmware that includes instructions stored in a non-volatile (or non-transitory) computer-readable storage medium. The instructions, when executed by a processing circuitry, such as one or more processors, cause the processing circuitry to perform the functions of the mesh encoder (400) and / or mesh decoder (500).
[0112] Returning to patch generation 406 of Figure 4, in some example implementations for mapping vertices and sampled 3D points of a 3D mesh into 2D UV space, the 3D mesh may be divided into patches (alternatively referred to as partitions or portions). As described above, the patches may or may not overlap in the 3D mesh. Each patch may be separated and mapped into the 2D UV space.
[0113] In some example implementations, non-overlapping patches may be generated. Such patches may be referred to as charts of the 3D mesh. As mentioned above, in some example implementations, the patch generation module (406) of FIG. 4 may aim to decompose the 3D mesh into a minimum number of patches with smooth boundaries while also minimizing the reconstruction error.
[0114] For example, a mesh representing a 3D scene may contain multiple 3D objects. Each 3D object may be described by one or more surfaces. Each 3D object may be further divided into different surface patches. For example, a 3D model of a human character in a mesh may be decomposed into head, torso, arm, and leg parts. Thus, the 3D scene may be decomposed into surfaces or portions of surfaces that each form a chart of a 3D mesh. The mapping of the entire mesh into UV space may be decomposed into mappings of multiple charts of the mesh into UV space.
[0115] Returning further to patch packing 408 of FIG. 4, in some exemplary implementations, different charts of a 3D mesh may be mapped to regions (or 2D patches) in UV space. Each 2D region or patch may be described, for example, by a depth field relative to a plane in the 2D space (e.g., flattening the surface so that deeper 3D points on the surface are further away from the center of the corresponding 2D map). In some exemplary implementations, the 2D regions or patches corresponding to various charts or patches of a 3D mesh may not overlap, as shown in the example of FIG. 8. In FIG. 8, a 3D mesh 802 may be divided into non-overlapping charts (portions or partitions), including, for example, charts 804, 806, 808, and 809, which may be mapped to non-overlapping 2D regions or patches 824, 826, 828, and 829, respectively, of a 2D map 820 in 2D UV space. In each of regions 824, 826, 828, and 829 in the 2D map, a collection of pixels may be mapped to vertices or sampled 3D surface points in a corresponding chart of the 3D mesh. The remaining pixels in each of these regions may be unmapped (or unoccupied).
[0116] In some other example implementations, different charts of a mesh may not be mapped to regions of the 2D UV such that they do not overlap. In other words, different charts from a 3D mesh may overlap in the 2D UV space after 3D to 2D mapping by the patch packing function 408 of FIG. 4, as long as there is no overlap of occupied points in the 2D map. An example is shown in FIG. 9. FIG. 9 shows a 3D mesh 802 with charts identical to those in FIG. 8, except that charts 804 and 809 are mapped to separate non-overlapping regions 824 and 829 of a 2D map 820 in the 2D UV space, respectively, while charts 806 and 808 of the 3D mesh are mapped to regions 828 and 826 of the 2D map 820 that overlap. Such implementations may help achieve a more compact 2D map, as described above.
[0117] Such overlap packing of 3D charts of a mesh into a 2D map is further illustrated in Fig. 11. Specifically, as illustrated in Fig. 10, a 2D occupancy map that already contains one or more charts of a 3D mesh is illustrated as 1002. Additional charts of the 3D mesh may be mapped into 2x4 sub-2D maps 1004 that will be packed into the overall 3D map 1002. Packing the sub-2D maps 1004 into 1002 may not require a clean (or all-zero) 2x4 region in 1002. Instead, the sub-2D maps 1004 may be packed into 2x4 regions of 1002 as long as that region does not have an overlap "1" with the 2x4 sub-2D map 1004. In the example of FIG. 10, two 2×4 regions in the 2D occupancy map 1002, shown as 1006 and 1008, may be used to pack the sub-2D map 1004 because the positions in each of the regions 1006 and 1008 that are already occupied (with a binary value of “1”) do not overlap with occupied positions (positions having a binary value of “1”) in the sub-2D map 1004. One of the regions 1006 and 1008 may be selected to pack the sub-2D map 1004. For example, the region 1006 may be selected for packing. The combined 2D occupancy map after such packing is shown as 1010. Such packing, which allows for region overlap as shown in the example of FIG. 10, results in a more compact 2D map.
[0118] Exemplary implementations are described in further detail below for efficient and factual packing of patches or charts of a 3D mesh or point cloud into a 2D map or 2D image. These implementations may be applied to the patch packing function 408 of the mesh encoder 400 of FIG. 4. For simplicity, the following disclosure is described in the context of packing charts of a 3D mesh generated by the patch generation function 406 of the mesh encoder 400 of FIG. 4 into a 2D occupancy map or image. It should be noted that such packing is applicable to other 2D maps, including geometry maps / images, depth 2D images, texture maps / images, etc. These various exemplary implementations apply to static 3D meshes, where there is only one frame for each of the meshes, or where the mesh content does not change over time. Additionally, these exemplary implementations are also applicable to dynamic 3D meshes or mesh frames of a sequence of 3D meshes.
[0119] FIG. 11 illustrates an occupancy map 1102 corresponding to a chart of a 3D mesh frame. The chart may correspond to, for example, a portion of the 3D mesh frame. The chart may be among multiple charts generated by the patch generation module 406 of the mesh encoder 400 of FIG. 4. The occupancy map 1102 may be a binary map indicating which pixels in the map are occupied and therefore correspond to vertices or sampled surface points in the 3D mesh. For example, the binary values in the occupancy map 1102 may be "0" or "1", where a value "1" may be used to indicate occupied and a value "0" may be used to indicate unoccupied. A visual guide of the occupancy map is shown as 1104, indicating that the area of the map is occupied in black.
[0120] In some example implementations, the occupancy map for each of the multiple charts of an input 3D mesh frame provided by the patch generation module 406 of the mesh encoder 400 of FIG. 4 may be generated similar to the particular occupancy map 1102 of FIG. 11.
[0121] In some exemplary implementations, zeros may be padded along the edges to expand the occupancy map to any predefined or configured region. In the exemplary padded occupancy map 1106 of FIG. 4, the top, bottom, left, and right of the occupancy map 1102 are padded with three rows, one row, four columns, and three columns of zeros, respectively, resulting in an expanded occupancy map. These two occupancy maps 1102 and 1106 of FIG. 11 represent the same occupancy information for an exemplary projected chart of a 3D mesh. In some practical implementations, the patch generation module 406 of the mesh encoder 400 of FIG. 4 may be configured to generate a minimum size occupancy map for each chart. Extension to an expanded occupancy map, if necessary, is straightforward.
[0122] For packing, we create an occupancy map of the projected image, and pixel values of the occupancy map (which has the same image size as the projected image) are initialized to 0. The projected image hosts all the projected charts. Once a projected chart is packed into the projected image, the occupancy map of the projected image is updated, and pixel values corresponding to occupied pixels in the projected charts are updated to 1.
[0123] The occupancy maps of multiple charts of the 3D mesh as described above may then be packed into an overall occupancy map. As described above, for a more compact packing, the occupancy maps of the various charts may be allowed to overlap some areas of the overall occupancy map, as long as the occupied pixels of the occupancy map of a chart are not projected onto pixels of the overall occupancy map that correspond to the occupied pixels of the occupancy map of the other charts of the 3D mesh frame, as illustrated in FIG. 10 and described above.
[0124] 10, a 2D occupancy map that already contains one or more charts of a 3D mesh is shown as 1002. Additional charts of the 3D mesh may be mapped into 2×4 sub-2D maps 1004 that will be packed into the overall 3D map 1002. Packing the sub-2D maps 1004 into 1002 may not require a clean (or all-zero) 2×4 region in 1002. Instead, the sub-2D maps 1004 can be packed into a 2×4 region of 1002 as long as that region has no overlapping “1”s with the 2×4 sub-2D map 1004. In the example of FIG. 10, two 2×4 regions in the 2D occupancy map 1002, shown as 1006 and 1008, may be used to pack the sub-2D map 1004 because the positions in each of the regions 1006 and 1008 that are already occupied (with a binary value of “1”) do not overlap with occupied positions (positions having a binary value of “1”) in the sub-2D map 1004. One of the regions 1006 and 1008 may be selected to pack the sub-2D map 1004. For example, the region 1006 may be selected for packing. The combined 2D occupancy map after such packing is shown as 1010. Such packing, which allows for region overlap as shown in the example of FIG. 10, results in a more compact 2D map.
[0125] Identification of candidate packing positions or candidate packing regions 1006 and 1008 in the 2D occupancy map 1002 for packing the sub 2D map 1004 of the corresponding chart of the 3D mesh frame may be done in various ways. The criterion for such identification may be that the pixels in the candidate regions corresponding to the positions in the sub 2D map 1004 having a binary value of "1" should not be "1". In other words, the sum of the element-wise products between the sub 2D occupancy map 1004 of the projected chart and the candidate regions in the global 2D occupancy map 1002 should be zero. This may lead to a simple no overlap check algorithm to check the sum of the element-wise products of all the patches in the global occupancy map 1002 with the sub 2D occupancy map 1004 of the projected chart to be packed. If the sum is zero, the sub 2D occupancy map 1004 can be packed into the position without overlap, otherwise, if the sum is greater than zero, i.e., if both occupancy maps are occupied for a pixel, then there will be an overlap when packed into the position.
[0126] Some typical example implementations for identifying such candidate packing regions may iteratively compute the element-wise sum of products between the occupancy map of the projected chart and a patch of the global map filtered by a moving window until the sum reaches 0 at some locations. The following disclosure further describes example efficient methods for computing the sums at different locations using image filtering.
[0127] In some example implementations, such an image filtering filter process may be designed to calculate the correlation between a 2D matrix and a 2D kernel. Given a 2D matrix Mat(x,y) and a 2D kernel ker(x',y') of size M*N, the output correlation matrix Out(x,y) may be defined as follows:
[0128]
number
[0129] Thus, by setting the occupancy map of the projection image (total occupancy image) as a 2D matrix Mat(x,y) and the occupancy map of the projection chart as the 2D kernel ker(x',y'), Out(0,0) gives the element-wise sum of products between the occupancy map of the projection chart and one of the projection images at position (0,0), Out(0,1) gives the element-wise sum of products between the occupancy map of the projection chart and one of the projection images at position (0,1), and so on. Image filtering, on the other hand, can be calculated very efficiently, for example, by using a Fast Fourier Transform (FFT). After deriving the output correlation matrix Out(x,y), the element-wise sum of products at all possible positions (pixel positions in the total 2D occupancy image) is obtained. The above calculations based on image filtering are essentially convolution operations and can be efficiently performed by utilizing hardware support such as parallel processing provided by a GPU.
[0130] In particular, since spatial convolution corresponds to multiplication in the spatial frequency domain, the above convolution calculation between the kernel (the 2D map of the chart) and the overall 2D map can be done by Fast Fourier Transform to the frequency domain, followed by frequency domain multiplication cascaded by an inverse Fast Fourier Transform.
[0131] Thus, starting from multiple charts of the 3D mesh frame generated by the patch generation module 406 of the mesh encoder 400, the blank global 2D occupancy map may be iteratively packed by the charts according to the following exemplary procedure.
[0132] An initial global occupancy map of the projection image is generated, and pixel values of the global occupancy map are initialized to a binary value of all "0".
[0133] Take the projection chart C and generate a sub-occupancy map of the projection chart.
[0134] Image filtering is performed to calculate the correlation or convolution between the global occupancy map of the projected image and the occupancy map of the projected chart (kernel) to generate an output correlation or convolution matrix.
[0135] From the output correlation matrix, select one matrix element that has a value of 0.
[0136] A location in the total occupancy image for packing the projected chart C is determined according to the location of the selected matrix element in the output correlation matrix.
[0137] The global occupancy map is updated with the projected images by packing the projected chart C into the location within the global occupancy map determined above.
[0138] Repeat the above steps to pack all the charts one after another and iteratively update the global occupancy map.
[0139] The step of selecting a packing location may be required above because in the output correlation matrix, there may very often be multiple matrix elements with value 0. The selection of a packing location from multiple candidate locations may be performed in various ways.
[0140] In some example implementations, the zero-valued element in the output correlation matrix that has the smallest pixel coordinate in one of the 2D dimensions among all the zero-valued elements may be selected. For example, the zero-valued element that has the smallest pixel coordinate in the width direction may be selected. Similarly, the zero-valued element that has the smallest pixel coordinate in the height direction may be selected.
[0141] In some other example implementations, the zero-valued element in the output correlation matrix that has the minimum sum of pixel coordinates in two of the 2D dimensions among all the zero-valued elements may be selected. Specifically, the zero-valued element that has the minimum width-plus-height pixel value may be selected to determine a location in the global occupancy map for packing the current chart.
[0142] In some other example implementations, the zero-valued elements in the output correlation matrix may be selected from among all non-zero values according to an optimization function of the width and height positions of the zero-valued elements in the correlation matrix. Such an optimization function may be p These may include optimization functions such as, but are not limited to, norm functions.
[0143] In some other alternative implementations, the zero-valued elements of the output correlation matrix may be selected based on a time constraint. For example, if the projection chart C is (x p ,y p ), the search may be done through all zero-valued elements. p ,y p ) is a zero-valued element, then (x p ,y p ) can be selected. (x p ,y p ) is not a zero-valued element in the correlation matrix, e.g., x p , y p Based on the optimization function of , x, and y, the (x p ,y p ) can be selected. Such an optimization function can be, for example, p ) and (yy p )l p The function may include, but is not limited to, a norm function. For example, the minimum l p Norm (or preferred position (x p ,y p ) may be identified for packing.
[0144] In some other implementations, during the above image filtering based method for fast and compact chart packing, it may not be necessary to complete the process to derive all possible candidate locations in the global projected image for packing the chart. For example, once the first zero-valued location (where the output correlation matrix element is 0) is identified and available, the process can be terminated and the projected chart can be assigned to its earliest identified location.
[0145] FIG. 12 shows a flow chart of an exemplary process (1200) according to an embodiment of the present disclosure. The process (1200) starts with step (S1201). In step (S1210), an initial 2D occupancy map corresponding to a 3D mesh frame is obtained. The initial 2D occupancy map includes zero or more pixels having a first predefined value indicating that the zero or more pixels are mapped to at least one other portion of the 3D mesh frame, where the at least one other portion does not overlap with the first portion of the 3D mesh frame, and remaining pixels having a second predefined value indicating that the remaining pixels are not mapped to the 3D mesh frame. In step (S1220), a 2D occupancy matrix is generated for the first portion of the 3D mesh frame, where the first predefined value corresponds to the occupied pixels and the second predefined value corresponds to the unoccupied pixels. In step (S1230), a correlation matrix between the initial 2D occupancy map and the 2D occupancy matrix is calculated. In step (S1240), a zero-valued target element in the correlation matrix is identified. In step (S1250), an updated 2D occupancy map is generated by replacing pixels having a second predefined value in a patch in an initial 2D occupancy map having the same size as the 2D occupancy matrix with a corresponding element in the 2D occupancy matrix, the patch being located in the initial 2D occupancy map in a packing position that coincides with the position of the zero-valued target element in the correlation matrix. The procedure (1200) stops at (S1299). The procedure (1200) may be repeated until all charts or portions of the 3D mesh are packed into the 2D occupancy map.
[0146] Process (1200) may be adapted as appropriate. Step(s) of process (1200) may be modified and / or omitted. Additional step(s) may be added. Any suitable order of performance may be used.
[0147] The techniques disclosed in this disclosure may be used separately or combined in any order. Furthermore, each of the techniques (e.g., methods, embodiments), encoders, and decoders may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In some examples, the one or more processors execute a program stored on a non-transitory computer-readable medium.
[0148] The techniques described above can be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 13 illustrates a computer system (1300) suitable for implementing certain embodiments of the disclosed subject matter.
[0149] Computer software can be coded using any suitable machine code or computer language that can be subjected to mechanisms such as assembly, compilation, linking, etc. to produce code containing instructions that can be executed directly, or via interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.
[0150] The instructions may be executed on various types of computers or computer components including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, Internet of Things devices, and the like.
[0151] 13 for computer system (1300) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system (1300).
[0152] The computer system (1300) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, via tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), olfactory input (not shown). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (voice, music, ambient sounds, etc.), images (scanned images, photographic images still images captured from a camera, etc.), video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).
[0153] The input human interface devices may include one or more (only one of each shown) of a keyboard (1301), a mouse (1302), a trackpad (1303), a touch screen (1310), a data glove (not shown), a joystick (1305), a microphone (1306), a scanner (1307), and a camera (1308).
[0154] The computer system (1300) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, by haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1310), data gloves (not shown), or joystick (1305), although there may also be haptic feedback devices that do not function as input devices), audio output devices (speakers (1309), headphones (not shown), etc.), visual output devices (e.g., screens (1310), including CRT screens, LCD screens, plasma screens, OLED screens, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting output in more than three dimensions by means of two-dimensional visual output or stereoscopic output, etc.), and printers (not shown).
[0155] The computer system (1300) may also include human accessible storage devices and their associated media, such as optical media, including a CD / DVD ROM / RW (1320) with CD / DVD or similar media (1321), thumb drives (1322), removable hard drives or solid state drives (1323), legacy magnetic media such as tapes and floppy disks (not shown), and dedicated ROM / ASIC / PLD based devices such as security dongles (not shown).
[0156] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.
[0157] The computer system (1300) may also include an interface (1354) to one or more communication networks (1355). The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable television, satellite television and terrestrial television broadcast, vehicular and industrial including CANBus, etc. Certain networks generally require an external network interface adapter connected to a particular general-purpose data port or peripheral bus (1349) (e.g., a USB port of the computer system (1300)), while other networks are generally integrated into the core of the computer system (1300) by connection to a system bus (e.g., an Ethernet interface to a PC computer system, a cellular network interface to a smartphone computer system), as described below. Using any of these networks, the computer system (1300) can communicate with other entities. Such communications can be unidirectional, receive only (e.g., television broadcast), unidirectional, transmit only (e.g., CANbus to a particular CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks can be used in each of these networks and network interfaces, as described above.
[0158] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core (1340) of the computer system (1300).
[0159] The cores (1340) may include one or more central processing units (CPUs) (1341), graphics processing units (GPUs) (1342), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1343), hardware accelerators for specific tasks (1344), graphics adapters (1350), etc. These devices may be connected via a system bus (1348), along with read-only memory (ROM) (1345), random access memory (1346), internal mass storage such as internal non-user accessible hard drives, SSDs, etc. (1347). In some computer systems, the system bus (1348) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1348) or via a peripheral bus (1349). In one example, a screen (1310) may be connected to the graphics adapter (1350). Peripheral bus architectures include PCI, USB, and the like.
[0160] The CPU (1341), GPU (1342), FPGA (1343), and accelerator (1344) can combine to execute certain instructions that may constitute the aforementioned computer code. The computer code can be stored in ROM (1345) or RAM (1346). Persistent data can be stored, for example, in internal mass storage (1347), while temporary data can also be stored in RAM (1346). Rapid storage and retrieval in any of the memory devices can be made possible by the use of cache memory, which can be closely associated with one or more of the CPU (1341), GPU (1342), mass storage (1347), ROM (1345), RAM (1346), etc.
[0161] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the available kind well known to those skilled in the computer software arts.
[0162] By way of example and not limitation, the computer system (1300) having the architecture, and in particular the core (1340), may provide functionality as a result of a processor (or processors) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage as introduced above, as well as media associated with the particular storage of the core (1340) that is non-transitory in nature, such as mass storage (1347) internal to the core or ROM (1345). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (1340). The computer-readable media may include one or more memory devices or chips, depending on the particular need. The software can cause the core (1340), and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (1346) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system can provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerators (1344)) that can operate to execute certain processes or certain portions of certain processes described herein in place of or in conjunction with software. References to software can encompass logic, and vice versa, where appropriate. References to computer-readable media can encompass circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any appropriate combination of hardware and software.
[0163] While this disclosure describes several exemplary embodiments, there exist modifications, substitutions, and various alternative equivalents, which fall within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art can devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of this disclosure and are therefore within the spirit and scope of this disclosure. [Explanation of symbols]
[0164] 105 3D sensor, 100 communication system, 110 terminal device, 120 terminal device, 200 streaming system, 201 3D mesh generator or storage medium, 202 3D mesh, 203 encoder, 204 compressed 3D mesh, 205 streaming server, 206 client subsystem, 207 compressed 3D mesh copy, 208 client subsystem, 209 compressed 3D mesh copy, 210 decoder, 211 stream of reconstructed 3D mesh, 212 rendering device, 213 capture or storage subsystem, 301 encoder, 302 raw input 3D mesh or sequence of 3D meshes, 303 decoder, 304 mapping unit for mapping 3D mesh into 2D UV space, 310 occupancy map, 312 geometry map, 314 attribute map, 316 other non-image or non-map data or metadata, 320 decoded occupancy map, 322 decoded geometry map, 324 decoded attribute map, 326 decoded non-mapped data and metadata, 330 3D mesh or sequence of 3D meshes, 400 3D mesh encoder, 404 patch information module, 406 patch generation module, 408 patch packing module, 410 geometry image generation module, 412 texture image generation module, 414 occupancy map module, 416 image padding module, 418 image padding module, 420 group expansion module, 422 video compression module, 423 video compression module, 424 multiplexer, 432 video compression module, 434 entropy compression module, 436 smoothing module, 438 auxiliary patch information compression module, 500 mesh decoder, 532 demultiplexer, 534 video decompression module, 536 video decompression module, 538 occupancy map decompression module, 542 auxiliary patch information decompression module, 544 geometry reconstruction module, 546 Smoothing module, 548 Texture reconstruction module, 552 Color smoothing module, 610 Video decoder, 620 Parser, 621symbols, 651 scaler / inverse transform unit, 652 intra picture prediction unit, 653 motion compensated prediction unit, 655 aggregator, 656 loop filter unit, 657 reference picture memory, 658 current picture buffer, 703 video encoder, 730 source coder, 732 coding engine, 733 local decoder, 734 reference picture memory, 735 predictor, 745 entropy coder, 750 controller, 802 3D mesh, 804 chart, 806 chart, 808 chart, 809 chart, 820 2D map, 824 2D region, 826 2D region, 828 2D region, 829 2D region, 926 region, 1002 2D occupancy map, 1004 sub 2D map, 1006 2×4 region, 1008 2×4 region, 1010 combined 2D occupancy map, 1102 occupancy map, 1104 visual guide for occupancy map, 1106 padded occupancy map, 1200 process, 1300 computer system, 1301 keyboard, 1302 mouse, 1303 trackpad, 1305 joystick, 1306 microphone, 1307 scanner, 1308 camera, 1309 speaker, 1310 touch screen, 1320 CD / DVD ROM / RW, 1321 CD / DVD or similar medium, 1322 thumb drive, 1323 removable hard drive or solid state drive, 1340 core of computer system, 1341 central processing unit (CPU), 1342 graphics processing unit (GPU), 1343 field programmable gate area (FPGA), 1344 hardware accelerator, 1345 Read-only memory (ROM), 1346 Random access memory (RAM), 1347 Internal mass storage, 1348 System bus, 1349 Peripheral bus, 1350 Graphics adapter, 1354 Network interface, 1355 Cloud computing environment
Claims
1. 1. A method for projecting a first portion of a three-dimensional (3D) mesh frame onto a two-dimensional (2D) occupancy map, the method comprising: Obtaining an initial 2D occupancy map, said initial 2D occupancy map comprising: zero or more pixels having a first predefined value indicating that the zero or more pixels are mapped to at least one other portion of the 3D mesh frame, where the at least one other portion does not overlap the first portion of the 3D mesh frame; remaining pixels having a second predefined value indicating that the remaining pixels are not mapped to the 3D mesh frame; and generating a 2D occupancy matrix for the first portion of the 3D mesh frame, the first predefined values corresponding to occupied pixels and the second predefined values corresponding to unoccupied pixels; calculating a correlation matrix between the initial 2D occupancy map and the 2D occupancy matrix; identifying zero-valued target elements in the correlation matrix; generating an updated 2D occupancy map by replacing pixels having the second predefined value in a patch in the initial 2D occupancy map having the same size as the 2D occupancy matrix with a corresponding element in the 2D occupancy matrix, the patch being located in the initial 2D occupancy map in a packing position according to a position of the zero-valued target element in the correlation matrix.
2. The method of claim 1 , wherein the correlation matrix is calculated as a convolution between the 2D occupancy matrix and the initial 2D occupancy map as a convolution kernel.
3. The method of claim 2 , wherein the convolution is calculated using a Fast Fourier Transform.
4. The step of identifying the zero-valued target elements in the correlation matrix comprises: identifying a number of zero-valued elements in the correlation matrix; and selecting one of the plurality of zero-valued elements in the correlation matrix as the zero-valued target element.
5. 5. The method of claim 4, wherein selecting one of the plurality of zero-valued elements in the correlation matrix as the zero-valued target element comprises selecting a zero-valued element of the plurality of zero-valued elements in the correlation matrix having a smallest index in a width dimension.
6. 5. The method of claim 4, wherein selecting one of the plurality of zero-valued elements in the correlation matrix as the zero-valued target element comprises selecting a zero-valued element of the plurality of zero-valued elements in the correlation matrix having a minimum index in a height dimension.
7. 5. The method of claim 4, wherein selecting one of the plurality of zero-valued elements in the correlation matrix as the zero-valued target element comprises selecting a zero-valued element of the plurality of zero-valued elements in the correlation matrix having a minimum sum of a width index and a height index.
8. 5. The method of claim 4, wherein selecting one of the plurality of zero-valued elements in the correlation matrix as the zero-valued target element comprises selecting a zero-valued element of the plurality of zero-valued elements based on an optimization function of a height index and a width index.
9. The optimization function is p The method of claim 8 including a norm function.
10. selecting one of the plurality of zero-valued elements in the correlation matrix as the zero-valued target element, determining a preferred packing location for the 2D occupancy matrix into the initial 2D occupancy map; and selecting the zero-valued target element from the plurality of zero-valued elements in the correlation matrix according to the preferred packing position.
11. selecting the zero-valued target element from the plurality of zero-valued elements in the plurality of zero-valued elements in the correlation matrix according to the preferred packing position, selecting, if possible, as the zero-valued target element from the plurality of zero-valued elements that coincide with the preferred packing location; and otherwise, and selecting from the plurality of zero-valued elements in the vicinity of the preferred packing location based on an optimization function of the preferred packing location and locations of zero-valued elements in the vicinity of the preferred packing location.
12. The optimization function is a function of l of a position difference between the preferred packing position and the position of a zero-valued element among the plurality of zero-valued elements. p The method of claim 11 including a norm function.
13. 13. The method of any one of claims 10 to 12, wherein identifying the zero-valued target element in the correlation matrix comprises determining a first identified zero-valued element of the correlation matrix as the zero-valued target element, the method further comprising terminating calculation of the correlation matrix once the first identified zero-valued element of the correlation matrix is found.
14. A method according to any one of claims 10 to 12, wherein the steps of the method according to claim 1 are repeated until all parts of the 3D mesh frame have been mapped onto the updated 2D occupancy map.
15. The method of claim 14 , further comprising compressing the updated 2D occupancy map into a bitstream using a video encoder to compress information of sizes and packing positions of 2D occupancy matrices of all portions of the 3D mesh frame.
16. The method according to any one of claims 10 to 12, wherein the initial 2D occupancy map and the 2D occupancy matrix comprise binary values comprising the first predefined value and the second predefined value.
17. 13. A device comprising: a memory for storing computer instructions; and a processor, the processor executing the computer instructions for projecting a first portion of a three-dimensional (3D) mesh frame onto a two-dimensional (2D) occupancy map to cause the processor to perform the method of claim 1.
18. 13. A computer program comprising computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform the method of claim 1 to project a first portion of a three-dimensional (3D) mesh frame onto a two-dimensional (2D) occupancy map.
Citation Information
Patent Citations
Mesh compression via point cloud representation
JP2022542419A
Image processing device and method
WO2020145117A1
Mesh compression via point cloud representation
WO2021116838A1
