Effective patch rotation in point cloud coding

Abstract

A point cloud decoding (PCC) method implemented by a decoder is provided. The method includes: a receiver of a decoder receiving a code stream including a patch rotation enable flag for a two-dimensional (2D) patch and atlas information; a processor of the decoder determines that the 2D patch is rotatable based on the patch rotation enabling flag; the processor rotates the 2D patch; the processor reconstructs a three-dimensional (3D) image using the atlas information and the rotated 2D patch.

Description

This application is a divisional application. The application number of the original application is 202080009098.0, and the original application date is January 13, 2020. The entire contents of the original application are incorporated into this application by reference. CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U.S. Provisional Patent Application No. 62 / 792,259, filed by Vladyslav Zakharchenko et al. on January 14, 2019, entitled “Point Cloud Codestream Structure and Auxiliary Information Differential Decoding,” which is incorporated herein by reference. Technical Field

[0002] The present disclosure relates generally to point cloud coding, and more particularly to a high-level syntax for point cloud coding. Background Art

[0003] Point clouds are widely used in applications such as the entertainment industry, smart car navigation, geospatial detection, three-dimensional (3D) modeling of real-world objects, and visualization. Given the non-uniformly sampled geometry of point clouds, a compact representation for storing and transmitting such data is useful. Irregular point clouds are more general and applicable to a wider range of sensors and data acquisition strategies than other 3D representations. For example, when performing 3D rendering in a virtual reality world or remote rendering in a telepresence environment, the rendering of virtual graphics and real-time instructions is processed as a dense point cloud dataset. Summary of the invention

[0004] A first aspect relates to a point cloud coding (PCC) method performed by a decoder. The method includes: a receiver of the decoder receives a code stream, the code stream including a patch rotation enable flag and atlas information for a two-dimensional (2D) patch; a processor of the decoder determines that the 2D patch can be rotated based on the patch rotation enable flag; the processor rotates the 2D patch; and the processor reconstructs a three-dimensional (3D) image using the atlas information and the rotated 2D patch.

[0005] Using this flexible patch orientation scheme, patches can be packed more efficiently within a bounding box. Due to this more efficient packing, the area occupied by the patch within the bounding box can be reduced compared to techniques where patch rotation is not available or allowed, which leads to higher decoding efficiency (e.g., fewer bits are required to decode the codestream).

[0006] In a first implementation of the method according to the first aspect, the above-mentioned code stream includes a default patch rotation and a preferred patch rotation, and wherein the 2D patch is rotated according to the default patch rotation or the preferred patch rotation.

[0007] In a second implementation of the method according to the first aspect or any preceding implementation of the first aspect, the bitstream includes one of a plurality of available rotations, and wherein the 2D patch is rotated according to one of the plurality of available patch rotations.

[0008] In a third implementation of the method according to the first aspect or any preceding implementation of the first aspect, a three-bit flag in the code stream identifies one of the multiple available patch rotations.

[0009] In a fourth implementation of the method according to the first aspect or any preceding implementation of the first aspect, the code stream includes a limited rotation enabling flag.

[0010] In a fifth implementation of the method according to the first aspect or any preceding implementation of the first aspect, when the limited rotation flag has a first value, the 2D patch is rotated based on an exhaustive orientation mode, and when the limited rotation flag has a second value, the 2D patch is rotated based on a simple orientation mode.

[0011] In a sixth implementation of the method according to the first aspect or any preceding implementation of the first aspect, the bitstream includes a patch rotation existence flag, and wherein, when the patch rotation existence flag has a first value, the 2D patch is rotated according to a default patch rotation.

[0012] In a seventh implementation of the method according to the first aspect or any preceding implementation of the first aspect, the code stream includes a patch rotation existence flag, and wherein, when the patch rotation existence flag has a second value, the 2D patch is rotated according to a preferred patch rotation or one of multiple available patch rotations.

[0013] In an eighth implementation of the method according to the first aspect or any preceding implementation of the first aspect, it further includes, when the patch rotation enable flag has a second value, the limited rotation enable flag has a second value, and the patch rotation existence flag has a second value, rotating the above-mentioned 2D patch according to the preferred patch rotation.

[0014] In a ninth implementation of the method according to the first aspect or any preceding implementation of the first aspect, it further includes, when the patch rotation enable flag has a second value, the limited rotation enable flag has a second value, and the patch rotation existence flag has a first value, rotating the above-mentioned 2D patch according to a default patch rotation.

[0015] In a tenth implementation of the method according to the first aspect or any preceding implementation of the first aspect, it further includes, when the patch rotation enable flag has a second value, the limited rotation enable flag has a first value, and the patch rotation existence flag has a first value, rotating the above-mentioned 2D patch according to a default patch rotation.

[0016] In an eleventh implementation of the method according to the first aspect or any preceding implementation of the first aspect, it further includes, when the patch rotation enable flag has a second value, the limited rotation enable flag has a first value, and the patch rotation existence flag has a second value, rotating the above-mentioned 2D patch according to one of a plurality of available patch rotations.

[0017] A second aspect relates to a point cloud coding (PCC) method implemented by an encoder. The method includes: a receiver of the encoder obtains a three-dimensional (3D) image; a processor of the encoder uses a plurality of available patch rotations to determine a plurality of two-dimensional (2D) projections of the 3D image; the processor selects one of the plurality of 2D projections; the processor sets a plurality of flags according to the selected one of the plurality of 2D projections; the processor generates a bitstream containing the plurality of flags and atlas information for reconstructing the 3D image; and the bitstream is stored in a memory of the encoder for transmission to a decoder.

[0018] Using this flexible patch orientation scheme, patches can be packed more efficiently within a bounding box. Due to this more efficient packing, the area occupied by the patch within the bounding box can be reduced compared to techniques where patch rotation is not available or allowed, which leads to higher decoding efficiency (e.g., fewer bits are required to decode the codestream).

[0019] In a first implementation of the method according to the second aspect, the plurality of flags include a patch rotation enabled flag, a limited rotation enabled flag, and a patch rotation existing flag.

[0020] In a second implementation of the method according to the second aspect or any preceding implementation of the second aspect, at least one of the plurality of flags is set to signal a decoder to use a default patch rotation.

[0021] In a third implementation of the method according to the second aspect or any preceding implementation of the second aspect, at least one of the plurality of flags is set to signal a decoder to use a preferred patch rotation.

[0022] In a fourth implementation of the method according to the second aspect or any preceding implementation of the second aspect, at least one of the plurality of flags is set to signal a decoder to use one of a plurality of available patch rotations.

[0023] The third aspect relates to a decoding device, which includes: a receiver for receiving a decoded video stream, the decoded video stream including a patch rotation enable flag and atlas information for a two-dimensional (2D) patch; a memory coupled to the receiver, the memory storing instructions; a processor coupled to the memory, the processor executing the above instructions to enable the decoding device to: determine whether the 2D patch can be rotated based on the patch rotation enable flag; rotate the 2D patch; and reconstruct a three-dimensional (3D) image using the above atlas information and the rotated 2D patch.

[0024] Using this flexible patch orientation scheme, patches can be packed more efficiently within a bounding box. Due to this more efficient packing, the area occupied by the patch within the bounding box can be reduced compared to techniques where patch rotation is not available or allowed, which leads to higher decoding efficiency (e.g., fewer bits are required to decode the codestream).

[0025] In a first implementation of the decoding device according to the third aspect, the decoding device further comprises a display for displaying the above-mentioned 3D image.

[0026] A fourth aspect relates to an encoding device, comprising: a receiver for receiving a three-dimensional (3D) image; a memory coupled to the receiver, the memory containing instructions; a processor coupled to the memory, the processor for executing the above instructions to enable the encoding device to: determine multiple two-dimensional (2D) projections of the 3D image using multiple available patch rotations; select one of the multiple 2D projections; set multiple flags according to the selected one of the multiple 2D projections; generate a code stream containing the multiple flags and atlas information for reconstructing the 3D image; and store the code stream in the memory of the encoder for transmission to the decoder.

[0027] Using this flexible patch orientation scheme, patches can be packed more efficiently within a bounding box. Due to this more efficient packing, the area occupied by the patch within the bounding box can be reduced compared to techniques where patch rotation is not available or allowed, which leads to higher decoding efficiency (e.g., fewer bits are required to decode the codestream).

[0028] In a first implementation of the encoding device according to the fourth aspect, the encoding device further comprises a transmitter coupled to the processor, the transmitter being configured to send the code stream to a decoder.

[0029] A fifth aspect relates to a decoding device, which comprises: a receiver for receiving a volume image for encoding, or for receiving a code stream for decoding; a transmitter coupled to the receiver, the transmitter for sending the above-mentioned code stream to a decoder or sending the decoded volume image to a reconstruction device for reconstructing the decoded volume image, the reconstruction device for reconstructing the decoded volume image; a memory coupled to at least one of the receiver or the transmitter, the memory for storing instructions; and a processor coupled to the memory, the processor for executing the instructions stored in the memory to perform any method disclosed herein.

[0030] Using this flexible patch orientation scheme, patches can be packed more efficiently within a bounding box. Due to this more efficient packing, the area occupied by the patch within the bounding box can be reduced compared to techniques where patch rotation is not available or allowed, which leads to higher decoding efficiency (e.g., fewer bits are required to decode the codestream).

[0031] In a first implementation of the decoding device according to the fifth aspect, the decoding device further comprises a display for displaying a projection image based on the decoded volume image.

[0032] A sixth aspect relates to a system, comprising: an encoder; and a decoder communicating with the encoder, wherein the encoder or decoder comprises the encoding device, decoding device, or decoding apparatus described herein.

[0033] Using this flexible patch orientation scheme, patches can be packed more efficiently within a bounding box. Due to this more efficient packing, the area occupied by the patch within the bounding box can be reduced compared to techniques where patch rotation is not available or allowed, which leads to higher decoding efficiency (e.g., fewer bits are required to decode the codestream).

[0034] The seventh aspect relates to a decoding device, which comprises: a receiving device for receiving a volume image for encoding or for receiving a code stream for decoding, reconstruction, and projection; a sending device, coupled to the receiving device, the sending device for sending the above-mentioned code stream to a decoder or sending a decoded image to a display device; a storage device, coupled to at least one of the receiving device or the sending device, the storage device for storing instructions; and a processing device, coupled to the storage device, the processing device for executing the instructions stored in the storage device to perform any method disclosed herein.

[0035] For clarity, any of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create new embodiments within the scope of the present disclosure.

[0036] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] For a more complete understanding of the present disclosure, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0038] Figure 1 is a block diagram of an example coding system that may use context modeling techniques.

[0039] Figure 2 is a block diagram illustrating an example encoder in which context modeling techniques may be implemented.

[0040] Figure 3 is a block diagram illustrating an example decoder in which context modeling techniques may be implemented.

[0041] Figure 4 It is a representation of a group of frames code stream.

[0042] Figure 5 It is a representation of a three-dimensional (3D) point cloud.

[0043] Figure 6 yes Figure 5 Representation of a 3D point cloud projected onto a bounding box.

[0044] Figure 7 is with Figure 6 A representation of the occupancy map corresponding to the two-dimensional (2D) projection into the bounding box of .

[0045] Figure 8 is with Figure 6 A representation of the geometry map corresponding to the 2D projection into the bounding box of .

[0046] Fig. 9 is with Figure 6 Representation of the attribute map corresponding to the 2D projection in the bounding box of .

[0047] Fig.10 is an embodiment of a representation of a patch directional index.

[0048] Fig.11 It is an embodiment of the patch-directed decoding process.

[0049] Fig.12 is an embodiment of a point cloud decoding (PCC) method implemented by a decoder.

[0050] Fig.13 is an embodiment of a PCC method implemented by an encoder.

[0051] Fig.14 It is a schematic diagram of a decoding device.

[0052] Fig.15 is a schematic diagram of an embodiment of a decoding device. DETAILED DESCRIPTION

[0053] First, it should be understood that although exemplary implementations of one or more embodiments are provided below, any number of currently known or existing technologies may be used to implement the disclosed systems and / or methods. The present disclosure should not be limited to the exemplary implementations, drawings, and techniques shown below (including the exemplary designs and implementations shown and described herein), but may be modified within the scope of the appended claims and the full scope of their equivalents.

[0054] Video coding (including encoding and decoding) standards include: International Telecommunications Union Telecommunication Standardization Sector (ITU-T) H.261, International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) Moving Picture Experts Group (MPEG)-1 Part 2, ITU-T H.262 or ISO / IEC MPEG-2 Part 2, ITU-T H.263, ISO / IEC MPEG-4 Part 2, Advanced Video Coding (AVC), also known as ITU-TH.264 or ISO / IEC MPEG-4 Part 10, and High Efficiency Video Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVC includes its extensions, such as scalable video coding (SVC), multiview video coding (MVC), multiview video coding plus depth (MVC+D), and 3D AVC (3D-AVC). HEVC includes its extensions, such as scalable HEVC (SHVC), multiview HEVC (MV-HEVC), and 3DHEVC (3D-HEVC).

[0055] A point cloud is a set of data points in 3D space. Each data point includes multiple parameters that determine the position (e.g., X, Y, Z), color (e.g., R, G, B or Y, U, V), and possibly other properties such as transparency, reflectivity, acquisition time, etc. Typically, each point in the cloud has the same number of additional properties. Point clouds can be used for a variety of applications, such as real-time 3D immersive telepresence, virtual reality (VR) viewing of content with interactive parallax, 3D free-viewpoint sports replay broadcasting, geographic information systems, cultural heritage, autonomous navigation based on large 3D dynamic maps, and automotive applications.

[0056] The ISO / IEC Moving Picture Experts Group has begun developing a new point cloud decoder standard for point cloud decoding of lossless and lossy compressed point cloud data in 2016. The point cloud decoder standard has a fairly high decoding efficiency and robustness to network environments. By using this decoder standard, point clouds can be processed as a form of computer data and stored on various storage media, and can also be transmitted and received through existing and future networks and distributed on existing and future broadcast channels.

[0057] Recently, the Point Cloud Coding (PCC) work has been divided into three categories: PCC Category 1, PCC Category 2, and PCC Category 3, where two separate working drafts are being developed, one for PCC Category 2 (PCC Cat2) and the other for PCC Category 1 and 3 (PCC Cat13). The latest working draft (WD) of PCC Cat2 is included in MPEG output document N17534, and the latest WD of PCC Cat13 is included in MPEG output document N17533.

[0058] The main design concept behind the PCC Cat2 decoder in PCC Cat2 WD is to use the existing video decoder to compress the geometric and texture information of dynamic point clouds by compressing the point cloud data into a set of different video sequences. In particular, two video sequences can be generated and compressed by using the video decoder, one representing the geometric information of the point cloud data and the other representing the texture information. Additional metadata used to interpret the two video sequences, i.e., occupancy maps and auxiliary patch information, can also be generated and compressed separately.

[0059] However, the existing design of PCC has defects. For example, the data units belonging to a time instance (i.e., an access unit (AU)) are discontinuous in decoding order. In PCC Cat 2WD, the data units of texture information, geometric information, auxiliary information, and occupancy map of each AU are interleaved in units of frame groups. That is, the geometric data of all frames in the group are together. Similarly, the same is true for texture data, etc. In PCC Cat 13WD, the geometric data units and general attributes of each AU are interleaved at the level of the entire PCC code stream (for example, when only one frame group has the same length as the entire PCC code stream, it is the same as PCC Cat2 WD). The interleaving of data units belonging to an AU will result in a large end-to-end delay, which is at least equal to the length of the frame group within the presentation duration in the application system.

[0060] Another disadvantage concerns the bitstream format. The bitstream format allows start code mode emulation like 0x0003 and is therefore not suitable for transmission on MPEG-2 transport streams (TS) where start code emulation prevention is required. For PCC Cat2, when decoding geometry and texture components using HEVC or AVC, there is currently only start code emulation prevention on group_of_frames_geometry_video_payload() and group_of_frames_texture_video_payload(). For PCC Cat13, there is no start code emulation prevention anywhere in the bitstream.

[0061] In PCC Cat2 WD, some decoder information for geometry and texture streams (e.g. which decoder, decoder profile, level, etc.) is hidden in multiple instances of the structures group_of_frames_geometry_video_payload() and group_of_frames_texture_video_payload(). In addition, information such as profile and level, which indicate the ability to decode auxiliary information and occupancy map components and point cloud reconstruction, is missing.

[0062] A high-level syntax design is provided that addresses one or more of the above-mentioned issues associated with point cloud decoding. As explained more fully below, the present disclosure utilizes a type indicator in a data unit header (also referred to as a PCC network access layer (NAL) header) to specify the content type in the payload of a PCC NAL unit. In addition, the present disclosure utilizes a frame group header NAL unit to carry the frame group header parameters. The frame group header NAL unit can also be used to signal the profile and level of each geometry codestream or texture codestream.

[0063] Figure 1 is a block diagram illustrating an example decoding system 10 that may utilize PCC video decoding techniques. Figure 1As shown, the decoding system 10 includes a source device 12 that provides encoded video data that is subsequently decoded by a destination device 14. In particular, the source device 12 may provide the video data to the destination device 14 via a computer-readable medium 16. The source device 12 and the destination device 14 may include any of a variety of devices, including desktop computers, notebook computers (e.g., laptop computers), tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, so-called "smart" tablets, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, the source device 12 and the destination device 14 may be configured for wireless communication.

[0064] The destination device 14 may receive the encoded video data to be decoded via a computer-readable medium 16. The computer-readable medium 16 may include any type of medium or device capable of transferring the encoded video data from the source device 12 to the destination device 14. In one example, the computer-readable medium 16 may include a communication medium to enable the source device 12 to send the encoded video data directly to the destination device 14 in real time. The encoded video data may be modulated according to a communication standard such as a wireless communication protocol and sent to the destination device 14. The communication medium may include any wireless communication medium or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet network such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other device that facilitates communication from the source device 12 to the destination device 14.

[0065] In some examples, the encoded data can be output from the output interface 24 to a storage device. Similarly, the encoded data can be accessed from the storage device through the input interface. The storage device may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a digital video disk (DVD), a compact disc read-only memory (CD-ROM), a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device may correspond to a file server or another intermediate storage device that can store the encoded video generated by the source device 12. The destination device 14 can access the stored video data from the storage device by streaming or downloading. The file server can be any type of server that can store encoded video data and send the encoded video data to the destination device 14. Example file servers include network servers (e.g., for websites), file transfer protocol (FTP) servers, network attached storage (NAS) devices, or local disk drives. The destination device 14 can access the encoded video data through any standard data connection including an Internet connection. These data connections may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of wireless and wired connections suitable for accessing the encoded video data stored on the file server. The transmission of the encoded video data from the storage device may be streaming transmission, download transmission, or a combination of streaming transmission and download transmission.

[0066] The technology disclosed herein is not limited to wireless applications or settings. The technology can be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission, such as dynamic adaptive streaming over HTTP (DASH) based on HTTP (hyper text transfer protocol), digital video encoded on a data storage medium, decoding digital video stored on a data storage medium, or other applications. In some embodiments, the decoding system 10 can be used to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0067] exist Figure 1 In the example of , the source device 12 includes a video source 18 for providing a volumetric image, a projection device 20, a video encoder 22, and an output interface 24. The destination device 14 includes an input interface 26, a video decoder 28, a reconstruction device 30, and a display device 32. According to the present disclosure, the encoder 22 of the source device 12 and / or the decoder 28 of the destination device 14 can be used to apply video decoding techniques. In other examples, the source device and the destination device can include other components or arrangements. For example, the source device 12 can receive video data from an external video source such as an external camera. Similarly, the destination device 14 can be connected to an external display device without including an integrated display device.

[0068] Figure 1 The decoding system 10 shown is only an example. The techniques for video decoding can be performed by any digital video encoding and / or decoding device. Although the techniques of the present disclosure are generally performed by a decoding device, these techniques can also be performed by an encoder / decoder commonly referred to as a "coder (CODEC)". In addition, the techniques of the present disclosure can also be performed by a video preprocessor. The encoder and / or decoder can be a graphics processing unit (GPU) or a similar device.

[0069] Source device 12 and destination device 14 are merely examples of such decoding devices, where source device 12 generates encoded video data for transmission to destination device 14. In some examples, source device 12 and destination device 14 may operate in a substantially symmetrical manner, such that both source device 12 and destination device 14 include a video encoding portion and a decoding portion. Thus, decoding system 10 may support one-way or two-way video transmission between video devices 12, 14, such as for video streaming, video playback, video broadcasting, or video telephony.

[0070] The video source 18 of the source device 12 may include a video capture device (e.g., a camera), a video archive containing previously captured video, and / or a video feed interface for receiving volumetric images or video from a video content provider. As another alternative, the video source 18 may generate volumetric images or computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video.

[0071] In some cases, where video source 18 is a video camera, source device 12 and destination device 14 may form so-called camera phones or video phones. However, as described above, the techniques described in this disclosure may be generally applicable to video coding and may be applied to wireless and / or wired applications.

[0072] As described in more detail below, the projection device 20 is used to project a volumetric image onto a plane (eg, a bounding box). That is, the projection device 20 is used to convert a three-dimensional (3D) image into one or more two-dimensional (2D) images.

[0073] In any case, the volumetric image, captured video, pre-captured video, or computer-generated video may be encoded by encoder 22. The encoded video information may then be output by output interface 24 onto computer-readable medium 16.

[0074] Computer-readable medium 16 may include transient media such as wireless broadcast or wired network transmission, or storage media such as hard disk, flash drive, optical disk, digital video disk, Blu-ray disk, or other computer-readable media (i.e., non-transient storage media). In some embodiments, a network server (not shown) may receive encoded video data from source device 12 and provide the encoded video data to destination device 14, for example, via network transmission. Similarly, a computing device of a media production facility such as a disc stamping facility may receive encoded video data from source device 12 and produce a disc containing the encoded video data. Therefore, in different examples, computer-readable medium 16 may be understood to include one or more computer-readable media in different forms.

[0075] Input interface 26 of destination device 14 receives information from computer-readable medium 16. The information of computer-readable medium 16 may include syntax information defined by encoder 22 and used by decoder 28, including syntax elements that describe characteristics and / or processing of blocks and other coding units such as groups of pictures (GOPs).

[0076] As described in more detail below, the reconstruction device 30 is used to convert one or more planar images back into a volumetric image. That is, the reconstruction device 30 is used to convert one or more 2D images back into a 3D image.

[0077] The display device 32 displays the volumetric image or the decoded video data to the user, and the display device may include any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

[0078] The encoder 22 and decoder 28 may operate in accordance with a video coding standard, such as the High Efficiency Video Coding (HEVC) standard currently under development, and may conform to the HEVC Test Model (HM). Alternatively, the encoder 22 and decoder 28 may operate in accordance with other proprietary or industry standards, such as the H.264 standard of the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) (or H.265 / HEVC for Moving Picture Experts Group (MPEG)-4, Part 10, Advanced Video Coding (AVC)) or extensions of these standards. However, the techniques of the present disclosure are not limited to any particular coding standard. Other examples of video coding standards include MPEG-2 and ITU-T H.263. Although in Figure 1 2, but in some aspects, the encoder 22 and decoder 28 may be integrated with the audio encoder and decoder, respectively, and may include appropriate multiplexer-demultiplexer (MUX-DEMUX) units or other hardware and software to handle the encoding of audio and video in a common data stream or in separate data streams. Where applicable, the MUX-DEMUX unit may conform to the ITU H.223 multiplexer protocol, or other protocols such as the user datagram protocol (UDP).

[0079] The encoder 22 and the decoder 28 can be implemented as any of a variety of applicable encoder circuits, such as one or more microprocessors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA), discrete logic, software, hardware, firmware, or any combination thereof. When these techniques are partially implemented in software, the device can store the instructions of the software in an applicable non-temporary computer-readable medium, and use one or more processors to implement these instructions in hardware to perform the techniques of the present disclosure. The encoder 22 and the decoder 28 can both be included in one or more encoders or decoders, wherein any of the two can be integrated as part of a combined encoder / decoder (encoder / decoder, CODEC) in the corresponding device. The device including the encoder 22 and / or the decoder 28 can include an integrated circuit, a microprocessor, and / or a wireless communication device such as a cellular phone.

[0080] Figure 2is a block diagram showing an example of an encoder 22 that can implement video decoding techniques. The encoder 22 can perform intra-frame decoding and inter-frame decoding on video blocks within a video slice (slice, also known as a slice, slice). Intra-frame decoding relies on spatial prediction to reduce or eliminate spatial redundancy in video within a given video frame or picture. Inter-frame decoding relies on temporal prediction to reduce or eliminate temporal redundancy in video within adjacent frames or pictures of a video sequence. Intra-frame mode (I mode) can refer to any of several spatial-based decoding modes. Inter-frame modes, such as uni-directional prediction (also known as uni prediction) (P mode) or bi-prediction (also known as bi prediction) (B mode), can refer to any of several temporal-based decoding modes.

[0081] like Figure 2 As shown, encoder 22 receives a current video block within a video frame to be encoded. Figure 2 In the example of , the encoder 22 includes a mode selection unit 40, a reference frame memory 64, a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy decoding unit 56. The mode selection unit 40 further includes a motion compensation unit 44, a motion estimation unit 42, an intra-prediction (also known as intra prediction) unit 46, and a partitioning unit 48. For video block reconstruction, the encoder 22 also includes an inverse quantization unit 58, an inverse transform unit 60, and a summer 62. A deblocking filter ( Figure 2 The output of summer 50 is filtered by a deblocking filter (not shown) to remove blocking artifacts from the reconstructed video. If necessary, the output of summer 62 is typically filtered by a deblocking filter. In addition to the deblocking filter, additional filters (in-loop or post-loop) may also be used. For simplicity, such filters are not shown in the figure, but if necessary, the above filters can filter the output of summer 50 (as in-loop filters).

[0082] During the encoding process, encoder 22 receives a video frame or slice to be encoded. The frame or slice may be divided into a plurality of video blocks. Motion estimation unit 42 and motion compensation unit 44 perform inter-frame prediction decoding on the received video block relative to one or more blocks in one or more reference frames to provide temporal prediction. Optionally, intra-frame prediction unit 46 may perform intra-frame prediction decoding on the received video block relative to one or more neighboring blocks in the same frame or slice as the block to be encoded to provide spatial prediction. Encoder 22 may perform multiple decoding passes, for example, to select an appropriate decoding mode for each block of video data.

[0083] In addition, the partitioning unit 48 can divide the video data block into sub-blocks based on an evaluation of a previous partitioning scheme in a previous decoding channel. For example, the partitioning unit 48 can first divide the frame or slice into the largest coding unit (LCU), and divide each LCU into sub-coding units (sub-CU) based on rate-distortion analysis (e.g., rate-distortion optimization). The mode selection unit 40 can further generate a quadtree data structure that indicates the division of the LCU into sub-CUs. The leaf node CU of the quadtree may include one or more prediction units (PU) and one or more transform units (TU).

[0084] The present disclosure uses the term "block" to refer to any of CU, PU, ​​or TU in the case of HEVC, or similar data structures in the case of other standards (for example, macroblocks and their sub-blocks in H.264 / AVC). A CU includes a decoding node, a PU associated with a decoding node, and a TU. The size of the CU corresponds to the size of the decoding node and is square. The size of the CU can range from 8×8 pixels to a maximum tree block size of 64×64 pixels or larger. Each CU may contain one or more PUs and one or more TUs. The syntax data associated with the CU may describe, for example, the partitioning of the CU into one or more PUs. Whether the CU is skipped or the CU is encoded in direct mode, intra-prediction mode, or inter-prediction (also called inter prediction) mode, the partitioning mode used may be different. The PU may be partitioned into a non-square shape. The syntax data associated with the CU may also describe, for example, the partitioning of the CU into one or more TUs according to a quadtree. The shape of the TU may be square or non-square (for example, rectangular).

[0085] Mode selection unit 40 may select one of the coding modes (intra or inter), for example, based on the error result, and provide the resulting intra or inter coded block to summer 50 to generate residual block data, and to summer 62 to reconstruct the coded block for use as a reference frame. Mode selection unit 40 also provides syntax elements, such as motion vectors, intra mode indicators, partition information, and other such syntax information, to entropy coding unit 56.

[0086] The motion estimation unit 42 and the motion compensation unit 44 may be highly integrated, but are shown separately for conceptual purposes. The motion estimation performed by the motion estimation unit 42 is the process of generating motion vectors, which are used to estimate the motion of video blocks. For example, the motion vector may indicate the displacement of a PU of a video block within a current video frame or picture relative to a prediction block within a reference frame (or other coding unit), wherein the prediction block within the reference frame is relative to the current block being encoded within the current frame (or other coding unit). The prediction block is a block that is found to closely match the block to be encoded in terms of pixel difference, which may be determined by the sum of absolute difference (SAD), the sum of square difference (SSD), or other difference metrics. In some embodiments, the encoder 22 may calculate the values ​​of sub-integer pixel positions of the reference picture stored in the reference frame memory 64. For example, the encoder 22 may interpolate the values ​​of quarter pixel positions, eighth pixel positions, or other fractional pixel positions of the reference picture. Therefore, the motion estimation unit 42 may perform motion search relative to full pixel positions and fractional pixel positions, and output motion vectors with fractional pixel accuracy.

[0087] Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU to the position of a prediction block of a reference picture. The reference picture may be selected from a first reference picture list (list 0) or a second reference picture list (list 1), each list identifying one or more reference pictures stored in reference frame memory 64. Motion estimation unit 42 sends the calculated motion vector to entropy encoding unit 56 and motion compensation unit 44.

[0088] Motion compensation performed by motion compensation unit 44 may include obtaining or generating a prediction block based on the motion vector determined by motion estimation unit 42. Also, in some examples, motion estimation unit 42 and motion compensation unit 44 may be functionally integrated. Upon receiving the motion vector for the PU of the current video block, motion compensation unit 44 may locate the prediction block to which the motion vector points in one of the reference picture lists. Summer 50 forms a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded, thereby forming pixel difference values ​​as described below. Typically, motion estimation unit 42 performs motion estimation relative to the luma component, and motion compensation unit 44 uses the motion vector calculated based on the luma component for the chroma components and the luma component. Mode selection unit 40 may also generate syntax elements associated with video blocks and video slices for use by decoder 28 when decoding video blocks of the video slices.

[0089] Intra-prediction unit 46 may perform intra-prediction on the current block as an alternative to the inter-prediction performed by motion estimation unit 42 and motion compensation unit 44 as described above. In particular, intra-prediction unit 46 may determine an intra-prediction mode to use for encoding the current block. In some examples, intra-prediction unit 46 may encode the current block using different intra-prediction modes, for example during separate encoding passes, and intra-prediction unit 46 (or, in some examples, mode selection unit 40) may select an applicable intra-prediction mode to use from the tested modes.

[0090] For example, intra-prediction unit 46 may calculate rate-distortion values ​​using rate-distortion analysis for various tested intra-prediction modes, and select the intra-prediction mode with the best rate-distortion characteristics among the tested modes. Rate-distortion analysis is typically used to determine the amount of distortion (or error) between a coded block and an original uncoded block that was encoded to produce the coded block, as well as the bit rate (i.e., the number of bits) used to produce the coded block. Intra-prediction unit 46 may calculate a ratio based on the distortion and rate of the various coded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0091] In addition, the intra prediction unit 46 can be used to encode the depth block of the depth map using a depth modeling mode (DMM). The mode selection unit 40 can use rate-distortion optimization (RDO) to determine whether the available DMM mode produces better decoding results than the intra prediction mode and other DMM modes. Data of the texture image corresponding to the depth map can be stored in the reference frame memory 64. The motion estimation unit 42 and the motion compensation unit 44 can also be used to perform inter-frame prediction on the depth block of the depth map.

[0092] After selecting an intra-prediction mode for a block (e.g., one of the conventional intra-prediction modes or the DMM mode), the intra-prediction unit 46 may provide information indicating the selected intra-prediction mode for the block to the entropy decoding unit 56. The entropy decoding unit 56 may encode the information indicating the selected intra-prediction mode. The encoder 22 may include configuration data in the transmitted bitstream, which may include a plurality of intra-prediction mode index tables and a plurality of modified intra-prediction mode index tables (also referred to as codeword mapping tables), definitions of encoding contexts for various blocks, and indications of the most likely intra-prediction mode, the intra-prediction mode index table, and the modified intra-prediction mode index table for each context.

[0093] Encoder 22 forms a residual video block by subtracting the prediction data from mode select unit 40 from the original video block being encoded. Summer 50 represents one or more components that perform this subtraction operation.

[0094] Transform processing unit 52 applies a transform such as a discrete cosine transform (DCT) or a conceptually similar transform to the residual block, thereby generating a video block including residual transform coefficient values. Transform processing unit 52 may perform other transforms conceptually similar to DCT. That is, wavelet transforms, integer transforms, sub-band transforms, or other types of transforms may also be used.

[0095] The transform processing unit 52 applies the transform to the residual block, thereby generating a block with residual transform coefficients. The transform can convert the residual information from the pixel value domain to the transform domain, such as the frequency domain. The transform processing unit 52 can send the resulting transform coefficients to the quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process can reduce the bit depth associated with some or all coefficients. The degree of quantization can be modified by adjusting the quantization parameter. In some examples, the quantization unit 54 can then scan the matrix including the quantized transform coefficients. Optionally, the entropy coding unit 56 can perform the scan.

[0096] After quantization, the entropy decoding unit 56 performs entropy decoding on the quantized transform coefficients. For example, the entropy decoding unit 56 can perform context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) decoding, or another entropy decoding technique. In the case of context-based entropy decoding, the context can be based on adjacent blocks. After the entropy decoding unit 56 performs entropy decoding, the coded bitstream can be sent to another device (e.g., decoder 28), or archived for later transmission or retrieval.

[0097] Inverse quantization unit 58 and inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, for example, for subsequent use as a reference block. Motion compensation unit 44 may calculate a reference block by adding the residual block to a prediction block of one of the frames of reference frame memory 64. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values ​​for motion estimation. Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to produce a reconstructed video block for storage in reference frame memory 64. Motion estimation unit 42 and motion compensation unit 44 may use the reconstructed video block as a reference block to inter-code a block in a subsequent video frame.

[0098] Figure 3 is a block diagram illustrating an example of a decoder 28 that may implement video decoding techniques. Figure 3 In the example of , decoder 28 includes entropy decoding unit 70, motion compensation unit 72, intra prediction unit 74, inverse quantization unit 76, inverse transform unit 78, reference frame memory 82, and summer 80. In some examples, decoder 28 may perform operations generally related to encoder 22 ( Figure 2 ) is a decoding pass that is the opposite of the encoding pass described in . The motion compensation unit 72 may generate prediction data based on the motion vector received from the entropy decoding unit 70, and the intra-frame prediction unit 74 may generate prediction data based on the intra-frame prediction mode indicator received from the entropy decoding unit 70.

[0099] During the decoding process, the decoder 28 receives an encoded video bitstream representing video blocks of an encoded video slice and associated syntax elements from the video encoder 22. An entropy decoding unit 70 of the decoder 28 entropy decodes the bitstream to generate quantization coefficients, motion vectors, or intra-prediction mode indicators, and other syntax elements. The entropy decoding unit 70 forwards the motion vectors and other syntax elements to the motion compensation unit 72. The decoder 28 may receive syntax elements at the video slice level and / or at the video block level.

[0100] In the case where the video slice is encoded as an intra-coded (I) slice, the intra-prediction unit 74 may generate prediction data for the video block of the current video slice based on the signaled intra-prediction mode and data from previously decoded blocks of the current frame or picture. In the case where the video frame is encoded as an inter-coded (e.g., B, P, or GPB) slice, the motion compensation unit 72 generates a prediction block for the video block of the current video slice based on the motion vectors and other syntax elements received from the entropy decoding unit 70. The prediction block may be generated from one of the reference pictures in one of the reference picture lists. The decoder 28 may construct the reference frame lists, i.e., List 0 and List 1, using a default construction technique based on the reference pictures stored in the reference frame memory 82.

[0101] Motion compensation unit 72 parses the motion vectors and other syntax elements to determine prediction information for the video blocks of the current video slice, and uses the prediction information to generate prediction blocks for the current video block being decoded. For example, motion compensation unit 72 uses some of the received syntax elements to determine the prediction mode (e.g., intra-frame or inter-frame prediction) used to encode the video blocks of the video slice, the inter-frame prediction slice type (e.g., B slice, P slice, or GPB slice), construction information for one or more reference picture lists for the slice, motion vectors for each inter-frame coded video block of the slice, inter-frame prediction status for each inter-frame coded video block of the slice, and other information for decoding the video blocks in the current video slice.

[0102] Motion compensation unit 72 may also perform interpolation based on interpolation filters. Motion compensation unit 72 may calculate interpolated values ​​of sub-integer pixels of a reference block using interpolation filters used by encoder 22 during encoding of the video block. In this case, motion compensation unit 72 may determine the interpolation filters used by encoder 22 from received syntax elements and use the interpolation filters to produce a prediction block.

[0103] Data of the texture image corresponding to the depth map may be stored in the reference frame memory 82. The motion compensation unit 72 may also be used to perform inter-frame prediction on the depth block of the depth map.

[0104] Figure 4 4 is a representation of a frame group code stream 400. As shown, the frame group code stream 400 includes a first frame group 402 (GOF_0) and a second frame group 404 (GOF_1). For illustration purposes only, the first frame group 402 and the second frame group 404 are separated from each other by a dotted line. Figure 4 Two frame groups 402 and 404 are shown, but it should be understood that in actual applications, the frame group code stream 400 may include any number of frames.

[0105] The first frame group 402 and the second frame group 404 are each formed by a set of access units 406. The access unit 406 is configured to contain frames that contain all or part of a compressed image (eg, a point cloud). Figure 4 The access unit 406 in may include or may be referred to as an atlas frame. In an embodiment, an atlas frame is a frame that contains sufficient information for reconstructing a point cloud by mapping decoded components together, wherein the components include point geometry, point attributes, occupancy maps, patches, etc.

[0106] Disclosed herein is a decoding technique that allows for flexible patch orientation. As used herein, patch orientation takes into account rotation, mirroring, and axis swapping (collectively referred to as rotation) of a patch. Using this flexible patch orientation scheme, patches can be more efficiently packed within a bounding box. Due to this more efficient packing, the area occupied by the patch within the bounding box can be reduced compared to techniques where patch rotation is not available or not allowed, which results in higher decoding efficiency (e.g., fewer bits required to decode the codestream).

[0107] Figure 5 is a representation of point cloud 500. Point cloud 500 is a volume representation of space on a regular 3D grid. That is, point cloud 500 is three-dimensional (3D). Figure 5 As shown, point cloud 500 includes point cloud content 502 in 3D space 504. Point cloud content 502 is represented by a set of points (e.g., voxels) in 3D space 504. Voxels are volume elements that represent some numerical quantities (e.g., the color of a point in 3D space) for visualization and analysis of 3D data. Therefore, voxels in 3D can be considered equivalent to pixels in 2D images.

[0108] Figure 5 Each voxel in the point cloud 500 has coordinates (eg, x, y, z coordinates) and one or more attributes (eg, red / green / blue (RGB) color components, reflectivity, etc.). Figure 5 The point cloud content 502 in FIG. 5 shows a person, but in practical applications, the point cloud content 502 may be any other volumetric object or image.

[0109] Figure 6 yes Figure 5 The point cloud 500 is projected onto the bounding box 600. Figure 6 As shown, bounding box 600 contains patches 602 projected onto its two-dimensional (2D) surface or plane 604. Therefore, patches 602 are 2D representations of portions of the 3D image. Patches 602 collectively correspond to Figure 5 Point cloud content 502. Data representation in video-based point cloud coding (V-PCC) (also called point cloud compression) relies on this 3D to 2D conversion.

[0110] For example, using Figure 7 Occupancy map 710, Figure 8 The geometric diagram 810, and Fig. 9 910 , the data representation in V-PCC is described as a set of planar 2D images (eg, patch 602 ).

[0111] Figure 7 is with Figure 6 602). The occupancy map 710 is decoded in binary form. For example, "zero" indicates that a portion of the bounding box 700 is not occupied by one of the patches 702. These portions of the bounding box 700 represented by zero will not participate in the reconstruction of the volume representation (e.g., point cloud content 502). Conversely, "one" indicates that a portion of the bounding box 700 is occupied by one of the patches 702. These portions of the bounding box 700 represented by one will participate in the reconstruction of the volume representation (e.g., point cloud content 502).

[0112] Figure 8 is with Figure 6 802 in the bounding box 600. The geometric map 810 provides or illustrates the outline or topography of each patch 802. That is, the geometric map 810 indicates the distance of each point in the patch 802 from the plane of the bounding box 800 (e.g., the plane 604).

[0113] Fig. 9 is with Figure 6 600 (e.g., patch 602). The attribute map 910 provides or shows the attributes of each point in the patch 902 in the bounding box 900. The attributes in the attribute map 910 can be, for example, the color components of the points. The color components can be based on the RCG color model, the YUV color model, or another known color model.

[0114] Currently, patch projection ( Figure 5 The 3D image 504 is projected onto Figure 6 The process of packaging the patches as a set of patches 602 on the bounding box 600 of the image processing unit (i.e., the process of packaging the patches as a set of patches 602 on the bounding box 600 of the image processing unit) is performed without changing the orientation of the patches. That is, the patches are not rotated or altered relative to their original orientation. However, this is a suboptimal solution for efficient packaging. To overcome this problem, the present disclosure provides techniques that allow flexible patch orientation (e.g., rotation, mirroring, or axis swapping of patches). Using this flexible patch orientation scheme, patches can be more efficiently packaged within the bounding box. Due to this more efficient packaging, the area occupied by the patches within the bounding box can be reduced compared to techniques where patch rotation is not available or not allowed, which results in higher decoding efficiency (e.g., fewer bits required to decode the codestream).

[0115] Fig.10is an embodiment of a representation of a patch orientation index 1000. The patch orientation index 1000 includes various predefined patch orientations 1008 of the patch 1002. For identification purposes, the different patch orientations 1008 in the patch orientation index 1000 are provided with numbers from #0 to #7. The initial patch orientation 1008 designated as #0 may be referred to as an anchor patch orientation. As shown, the patch orientation 1008 designated as #0 rotated 90 degrees clockwise will result in a patch orientation 1008 with #1. Similarly, the patch orientation 1008 designated as #1 rotated 90 degrees clockwise will result in a patch orientation 1008 with #2, and the patch orientation 1008 designated as #2 rotated 90 degrees clockwise will result in a patch orientation 1008 with #3.

[0116] Still refer to Fig.10 , the patch orientation 1008 designated as #0 is mirrored (e.g., flipped along a vertical axis) to obtain a patch orientation 1008 having #4. The patch orientation 1008 designated as #4 is rotated 90 degrees clockwise to obtain a patch orientation 1008 having #5, the patch orientation 1008 designated as #5 is rotated 90 degrees clockwise to obtain a patch orientation 1008 having #6, and the patch orientation 1008 designated as #6 is rotated 90 degrees clockwise to obtain a patch orientation 1008 having #7.

[0117] In an embodiment, pairs of patch orientations 1008 are formed to produce a simple orientation. For example, a patch orientation 1008 designated as #0 and a patch orientation 1008 designated as #7 form a simple orientation. Similarly, a patch orientation 1008 designated as #1 and a patch orientation 1008 designated as #6 form a simple orientation, a patch orientation 1008 designated as #2 and a patch orientation 1008 designated as #5 form a simple orientation, and a patch orientation 1008 designated as #3 and a patch orientation 1008 designated as #4 form a simple orientation. Thus, the patch orientation index 1000 provides four simple orientations.

[0118] The exhaustive orientation (also called the exhaustive orientation graph) includes all patch orientations 1008 in the patch orientation index 1000. That is, Fig.10 In the illustrated embodiment, the exhaustive orientation includes all eight patch orientations (labeled #0 to #7). Using the exhaustive orientation can provide better compression than the simple orientation, but at the expense of additional computation required by the encoder.

[0119] In an embodiment, patch 1002 may have a default patch orientation (e.g., one of patch orientations 1008). For example, patch 1002 may have a default rotation. Any of the patch orientations 1008 (e.g., from #0 to #7) may be the default rotation. The default patch orientation may be signaled in a sequence parameter set (SPS), a picture parameter set (PPS), or in a frame group of a codestream (e.g., at a patch level).

[0120] In an embodiment, patch 1002 may have a preferred patch orientation (e.g., one of patch orientations 1008). For example, patch 1002 may have a preferred rotation. Any of the patch orientations 1008 (e.g., from #0 to #7) may be a preferred rotation. The preferred patch orientation may be signaled in the SPS, PPS, or in a frame group of a codestream (e.g., at a patch level).

[0121] In an embodiment, a flag is used to indicate that a patch (e.g., patch 1002) can be rotated. In an embodiment, the flag is designated as a patch rotation enable flag. In an embodiment, the patch rotation enable flag is a one-bit flag. When the patch rotation enable flag has a first value (e.g., zero), the patch cannot be rotated. When the patch rotation enable flag has a second value (e.g., one), the patch can be rotated.

[0122] In an embodiment, a flag is used to indicate that a patch (e.g., patch 1002) has a simple orientation (e.g., one of two patch rotations 1008). In an embodiment, the flag is designated as a limited rotation enabled flag. In an embodiment, the limited rotation enabled flag is a one-bit flag. When the limited rotation enabled flag has a first value (e.g., zero), a detailed orientation is available for the patch. When the limited rotation enabled flag has a second value (e.g., one), a simple orientation is available for the patch.

[0123] In an embodiment, a flag is used to indicate that a patch (e.g., patch 1002) has been rotated. In an embodiment, the flag is designated as a patch rotation presence flag. In an embodiment, the patch rotation presence flag is a one-bit flag. When the patch rotation presence flag has a first value (e.g., zero), a default patch orientation is used. When the patch rotation presence flag has a second value, one of eight patch orientations 1008 in a preferred patch rotation or an exhaustive orientation is used, as described more fully below, depending on the value of a limited rotation enable flag. In an embodiment, the limited rotation enable flag is a three-bit flag.

[0124] Fig.11 11 is an embodiment of a patch directional decoding process 1100. The patch directional decoding process 1100 can be used to decode a coded bitstream to reconstruct a volumetric image. In block 1102, a default patch rotation and a preferred patch rotation of a patch (e.g., a 2D patch) are obtained from the coded bitstream. The default patch rotation and the preferred patch rotation can be represented by one of index numbers #0 to #7.

[0125] At block 1104, the patch is processed. In an embodiment, the processing of the patch includes obtaining atlas information, extracting the 2D position of the patch, the 3D position of the patch, and the above-mentioned flag of the patch. The atlas information is information that allows the patch to be decoded and mapped from a 2D representation to a 3D representation for reconstruction. In an embodiment, the atlas information includes a patch list.

[0126] At block 1106, a value of a patch rotation enable flag is determined. When the patch rotation enable flag has a first value (e.g., zero), the patch cannot be rotated. Therefore, at block 1108, the patch is not rotated. Then, at block 1110, the patch is processed based on atlas information (also referred to as auxiliary information) to reconstruct a volumetric image.

[0127] Returning to block 1106, when the patch rotation enabled flag has a second value (e.g., one), the patch may be rotated. At block 1112, the value of the limited rotation enabled flag is determined. When the limited rotation enabled flag has a first value (e.g., zero), the patch is rotated using one of eight available patch rotations according to the exhaustive orientation mode.

[0128] At block 1114, a value of a patch rotation presence flag is determined. When the patch rotation presence flag has a first value (e.g., zero), a default patch orientation is used. Therefore, at block 1116, a default rotation is used for the patch. In an embodiment, because all patches in a set of patches use the same default patch orientation, no additional signaling is required. Then, at block 1110, the patch is processed based on the atlas information to reconstruct a volumetric image.

[0129] Returning to block 1114, when the patch rotation present flag has a second value (e.g., one), one of the available orientations (indicated by one of the index #0 values ​​#7) is used. In an embodiment, a three-bit flag is used to signal the patch orientation to be used. Then, at block 1110, the patch is processed based on the atlas information to reconstruct a volumetric image.

[0130] Returning to box 1112, when the limited rotation enabled flag has a second value (e.g., one), the patch is rotated using one of the two patch rotations according to the simple orientation mode. In box 1120, the value of the patch rotation present flag is determined. When the patch rotation present flag has a first value (e.g., zero), the default patch orientation is used. Therefore, in box 1116, the default rotation is used for the patch. In an embodiment, because all patches in a set of patches use the same default patch orientation, no additional signaling is required. Then, in box 1110, the patch is processed based on the atlas information to reconstruct the volumetric image.

[0131] When the patch rotation present flag has a second value (e.g., one), the preferred patch rotation is used. Therefore, at block 1122, the preferred rotation is used for the patch. In an embodiment, no additional signaling is required because all patches in a set of patches use the same preferred patch orientation. Then, at block 1110, the patch is processed based on the atlas information to reconstruct a volumetric image.

[0132] Fig.12 The method 1200 is an embodiment of a point cloud decoding (PCC) method 1200 implemented by a decoder (e.g., entropy decoding unit 70). The method 1200 may be performed to decode an encoded bitstream to reconstruct a volumetric image. At block 1202, a bitstream containing a patch rotation enable flag and atlas information for a two-dimensional (2D) patch is received. The atlas information (also known as auxiliary information) contains sufficient information for reconstructing a point cloud by mapping the decoded components together, wherein the components include point geometry, point attributes, occupancy maps, patches, etc.

[0133] At block 1204, it is determined whether the 2D patch can be rotated based on a patch rotation enabled flag. In an embodiment, when the patch rotation enabled flag is set to a value of one (1), the patch can be rotated.

[0134] At block 1206, the 2D patch is rotated. The 2D patch may be rotated to a default patch rotation, a preferred patch rotation, or one of a plurality of available patch rotations as described herein. That is, in an embodiment, the 2D patch may be rotated according to a simple orientation mode (where the default patch rotation or the preferred patch rotation is used), or may be rotated according to an exhaustive orientation mode (where any of the eight available patch rotations is used).

[0135] At block 1208, a three-dimensional (3D) image is reconstructed using the atlas information and the rotated 2D patch. Once reconstructed, the 3D image may be displayed to a user on a display of an electronic device (eg, a smartphone, a tablet, a laptop, etc.).

[0136] Fig.13 The method 1300 may be performed to encode a volumetric image into a bitstream for transmission to a decoder. At block 1302, a three-dimensional (3D) image (e.g., a volumetric image) is obtained. At block 1304, a plurality of two-dimensional (2D) projections are determined for the 3D image using a plurality of available patch rotations. In an embodiment, the available patch rotations are Fig.10 The patch rotation shown in 1008. However, other patch rotations may be used.

[0137] At block 1306, one of the plurality of 2D projections is selected. A 2D projection in the plurality of 2D projections may be selected when the 2D projection produces the most efficient bounding box packing relative to the other 2D projections. The most efficient bounding box packing may use the least area, may minimize the computational intensity of the encoder, and the like.

[0138] At block 1308, a plurality of flags are set according to the selected one of the plurality of 2D projections. In an embodiment, each of the following flags is set: a patch rotation flag, a limited rotation enabled flag, a patch rotation present flag, and a flag identifying which patch rotation among a plurality of available patch rotations.

[0139] In block 1310, a code stream is generated, the code stream including the above-mentioned multiple flags and atlas information for reconstructing a 3D image. In block 1312, the code stream is stored for transmission to a decoder. In an embodiment, the code stream is sent to a decoder.

[0140] Fig.14 1 is a schematic diagram of a decoding device 1400 (e.g., encoder 22, decoder 28, etc.) according to an embodiment of the present disclosure. The decoding device 1400 is suitable for implementing the methods and processes disclosed herein. The decoding device 1400 includes an inlet port 1410 and a receiver unit (Rx) 1420 for receiving data; a processor, a logic unit, or a central processing unit (CPU) 1430 for processing data; a transmitter unit (Tx) 1440 and an outlet port 1450 for sending data; and a memory 1460 for storing data. The decoding device 1400 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the inlet port 1410, the receiver unit 1420, the transmitter unit 1440, and the outlet port 1450, which are used for input or output of optical or electrical signals.

[0141] The processor 1430 is implemented by hardware and software. The processor 1430 can be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 1430 communicates with the inlet port 1410, the receiver unit 1420, the transmitter unit 1440, the outlet port 1450, and the memory 1460. The processor 1430 includes a decoding module 1470. The decoding module 1470 implements the above-disclosed embodiments. In an embodiment, the decoding module 1470 is a reconstruction module for projecting reconstructed volumetric images. Therefore, the inclusion of the decoding module 1470 provides substantial improvements to the functionality of the decoding device 1400 and implements the conversion of the decoding device 1400 to different states. Alternatively, the decoding module 1470 is implemented as instructions stored in the memory 1460 and executed by the processor 1430.

[0142] The decoding device 1400 may also include an input and / or output (I / O) device 1480 for communicating data with a user. The I / O device 1480 may include an output device, such as a display for displaying video data, a speaker for outputting audio data, etc. The I / O device 1480 may also include an input device, such as a keyboard, a mouse, a trackball, etc., and / or include a corresponding interface for interacting with such an output device.

[0143] The memory 1460 includes one or more disks, tape drives, and solid-state drives, and can be used as an overflow data storage device to store programs when a program is selected to be executed, and to store instructions and data read during program execution. The memory 1460 can be volatile and non-volatile, and can be a read-only memory (ROM), a random-access memory (RAM), a ternary content-addressable memory (TCAM), and a static random-access memory (SRAM).

[0144] Fig.15 1 is a schematic diagram of an embodiment of a decoding apparatus 1500. In an embodiment, the decoding apparatus 1500 is implemented in a decoding device 1502. The decoding device 1502 comprises a receiving device 1501. The receiving device 1501 is used to receive an image for encoding or receive a code stream for decoding. The decoding device comprises a sending device 1507 coupled to the receiving device 1501. The sending device 1507 is used to send a code stream to a decoder or send a decoded image to a display device (e.g., one of the I / O devices 1408).

[0145] The decoding device 1502 includes a storage device 1503. The storage device 1503 is coupled to at least one of the sending device 1501 or the receiving device 1507. The storage device 1503 is used to store instructions. The decoding device 1502 also includes a processing device 1505. The processing device 1505 is coupled to the storage device 1503. The processing device 1503 is used to execute the instructions stored in the storage device 1503 to perform the method disclosed herein.

[0146] In an embodiment, syntax suitable for implementing the concepts disclosed herein is provided.

[0147] The following describes possible syntax definitions. Auxiliary information data unit modification

[0148] From the above, it should be understood that for the default projection process, the orientation of the patch can be different. A 1-bit flag can be used to signal the orientation of the patch in a simplified manner. A mechanism for switching between the default orientation and the preferred orientation is introduced.

[0149] Although several embodiments have been provided in the present disclosure, it will be appreciated that the disclosed systems and methods may be implemented in many other specific forms without departing from the spirit or scope of the present disclosure. The examples shown are to be considered illustrative rather than restrictive, and the invention is not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0150] In addition, without departing from the scope of the present disclosure, the techniques, systems, subsystems, and methods described and illustrated in a discrete or separate manner in various embodiments may be combined or integrated with other systems, components, techniques, or methods. Other examples of changes, substitutions, and alterations may be determined and made by those skilled in the art without departing from the spirit and scope disclosed herein.

Claims

1. An image processing method, characterized in that: The method comprises: receiving a bitstream, the bitstream comprising at least one flag associated with a two-dimensional (2D) patch; determining a rotation mode of the 2D patch based on the at least one marker; Rotating the 2D patch based on the rotation mode; and Reconstructing a three-dimensional (3D) image using the rotation result of the 2D patch; Wherein, the at least one flag includes a first flag, and when the first flag has a first value, the first flag indicates that the 2D patch is rotated based on a rotation mode selected from a first mode set, or when the first flag has a second value, the first flag indicates that the 2D patch is rotated based on a rotation mode selected from a second mode set.

2. The method according to claim 1, characterized in that The at least one flag further includes indication information, where the indication information is used to indicate a default rotation mode, a preferred rotation mode, or a rotation mode in the first mode set or the second mode set.

3. The method according to claim 1, characterized in that The first mode set includes two rotation modes, and the at least one flag further includes a 1-bit second flag, where the second flag is used to indicate which of the two rotation modes is used to rotate the 2DPatch.

4. The method according to claim 1, characterized in that: The second mode set includes more than two rotation modes, and the at least one flag further includes a 1-bit third flag. When the third flag has a first value, the third flag indicates a default rotation mode selected from the more than two rotation modes. When the third flag has a second value, the fourth flag is used to indicate a specific rotation mode.

5. The method according to claim 4, characterized in that The second mode set includes 8 rotation modes.

6. The method according to claim 5, characterized in that The fourth flag has 3 bits.

7. The method according to claim 1, characterized in that The number of rotation modes in the first mode set is different from the number of rotation modes in the second mode set.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: The 3D image is displayed on a display device.

9. An image processing method, characterized in that: The method comprises: Obtaining three-dimensional (3D) images; determining a plurality of two-dimensional (2D) projections of the 3D image using a plurality of available patch rotations; selecting one of the plurality of 2D projections; Setting at least one flag for the 2D patch according to the selected 2D projection; and generating a bitstream, wherein the bitstream includes the at least one flag and atlas information, and the at least one flag and the atlas information are used to reconstruct the 3D image; Wherein, the at least one flag includes a first flag, and when the first flag has a first value, the first flag indicates that the 2D patch is rotated based on a rotation mode selected from a first mode set, or when the first flag has a second value, the first flag indicates that the 2D patch is rotated based on a rotation mode selected from a second mode set.

10. The method according to claim 9, characterized in that The at least one flag further includes indication information, where the indication information is used to indicate a default rotation mode, a preferred rotation mode, or a rotation mode in the first mode set or the second mode set.

11. The method according to claim 9, characterized in that The first mode set includes two rotation modes, and the at least one flag further includes a 1-bit second flag, where the second flag is used to indicate which of the two rotation modes is used to rotate the 2DPatch.

12. The method according to claim 9, characterized in that The second mode set includes more than two rotation modes, and the at least one flag further includes a 1-bit third flag. When the third flag has a first value, the third flag indicates a default rotation mode selected from the more than two rotation modes. When the third flag has a second value, the fourth flag is used to indicate a specific rotation mode.

13. The method according to claim 12, characterized in that The second mode set includes 8 rotation modes.

14. The method according to claim 13, characterized in that The fourth flag has 3 bits.

15. The method according to claim 9, characterized in that The number of rotation modes in the first mode set is different from the number of rotation modes in the second mode set.

16. The method according to any one of claims 9 to 15, characterized in that: The method further comprises: Send the code stream, and / or store the code stream.

17. An image processing device, characterized in that: include: A receiver, configured to receive a code stream, wherein the code stream includes at least one flag associated with a two-dimensional (2D) patch; a memory coupled to the receiver, the memory storing instructions; as well as A processor, coupled to the memory, configured to execute the instructions to cause the decoding device to: determining a rotation mode of the 2D patch based on the at least one marker; Rotating the 2D patch based on the rotation mode; and Reconstructing a three-dimensional (3D) image using the rotation result of the 2D patch; Wherein, the at least one flag includes a first flag, and when the first flag has a first value, the first flag indicates that the 2D patch is rotated based on a mode selected from a first mode set, or when the first flag has a second value, the first flag indicates that the 2D patch is rotated based on a mode selected from a second mode set.

18. The decoding device according to claim 15, characterized in that The image processing device further comprises a display for displaying the 3D image.

19. An image processing device, characterized in that: include: a receiver for receiving a three-dimensional (3D) image; a memory coupled to the receiver, the memory storing instructions; A processor, coupled to the memory, configured to execute the instructions to enable the image processing device to: determining a plurality of two-dimensional (2D) projections of the 3D image using a plurality of available patch rotations; selecting one of the plurality of 2D projections; Setting at least one flag for the 2D patch according to the selected 2D projection; as well as generating a bitstream, wherein the bitstream includes the at least one flag and atlas information, and the at least one flag and the atlas information are used to reconstruct the 3D image; Wherein, the at least one flag includes a first flag, and when the first flag has a first value, the 2D patch is rotated based on a rotation mode selected from a first mode set, or when the second flag has a second value, the 2D patch is rotated based on a rotation mode selected from a second mode set.

20. The image processing according to claim 19, characterized in that The image processing device further includes a transmitter coupled to the processor, the transmitter is used to send the code stream to the decoder, and / or the processor executes the instructions to enable the image processing device to store the code stream in a storage device.

21. A system, characterized in that: The invention comprises the image processing device according to claim 17 or 18 and the image processing device according to claim 19 or 20.

22. A code stream, characterized in that: The code stream includes at least one flag and atlas information, the at least one flag is used to determine the rotation mode of a two-dimensional (2D) patch, and the at least one flag and the atlas information are used to reconstruct a three-dimensional (3D) image; wherein the at least one flag includes a first flag, when the first flag has a first value, the first flag indicates that the 2D patch is rotated based on a rotation mode selected from a first mode set, or when the first flag has a second value, the first flag indicates that the 2D patch is rotated based on a rotation mode selected from a second mode set.

23. A storage device, characterized in that: The storage device stores the code stream as claimed in claim 22.

Images