Method, device, computer equipment and storage medium for encoding video code stream using video point cloud coding

Abstract

The present application provides a method and apparatus for encoding a video bitstream using video point cloud coding. The method includes: obtaining a source point cloud; generating a geometric map and a texture map based on the source point cloud; generating a reconstructed geometric map and a reconstructed texture map using lossy coding without recoloring, wherein the resolution of the reconstructed geometric map is the same as the resolution of the reconstructed texture map; obtaining a point in the source point cloud corresponding to a geometric pixel of the geometric map and a texture pixel of the texture map; selecting the color of the texture pixel based on the color of the obtained point; and generating an encoded video bitstream based on the selected color, geometric map, and texture map.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 197,196 filed on June 4, 2021 and U.S. Application No. 17 / 495,301 filed on October 6, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to a set of advanced video coding and decoding technologies, and specifically to a method, apparatus, computer equipment and storage medium for encoding a video bitstream using video point cloud coding. Background Art

[0004] Advanced three-dimensional (3D) representations of the world are enabling more immersive interactions and communications. They also enable machines to understand, interpret, and navigate our world. Point clouds have been widely used as 3D representations of the world. For example, they can be used in autonomous vehicles for object detection and localization, in geographic information systems (GIS) for mapping, and in cultural heritage for visualizing and archiving cultural heritage objects and collections. Several use cases associated with point cloud data have been identified, and some corresponding requirements for point cloud representation and compression have been developed.

[0005] Point clouds contain a set of high-dimensional, e.g., three-dimensional (3D), points, each of which includes 3D position information and additional attributes such as color, reflectivity, etc. They can be captured using multiple cameras and depth sensors, or lidar in various scenarios, and can consist of thousands to billions of points to realistically represent the original scene.

[0006] Compression techniques are needed to reduce the amount of data required to represent point clouds, either for faster transmission or reduced storage. ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) has created an ad hoc group (MPEG-PCC) to standardize compression techniques for static or dynamic clouds. Summary of the invention

[0007] In each embodiment, a method for encoding a video bitstream using video point cloud coding is performed by at least one processor and includes: obtaining a source point cloud; generating a geometric map and a texture map based on the source point cloud; using lossy coding without recoloring to generate a reconstructed geometric map and a reconstructed texture map, wherein the resolution of the reconstructed geometric map is the same as the resolution of the reconstructed texture map; obtaining a point in the source point cloud corresponding to a geometric pixel of the geometric map and a texture pixel of the texture map; selecting a color of the texture pixel based on the color of the obtained point; and generating an encoded video bitstream based on the selected color, the geometric map, and the texture map.

[0008] In each embodiment, a device for encoding a video bitstream using video point cloud coding includes: at least one memory for storing program code; and at least one processor for reading the program code and operating according to the instructions of the program code, the program code including: a first acquisition code for enabling the at least one processor to obtain a source point cloud; a first generation code for enabling the at least one processor to generate a geometric map and a texture map based on the source point cloud; a second generation code for enabling the at least one processor to generate a reconstructed geometric map and a reconstructed texture map using lossy coding without recoloring, wherein the resolution of the reconstructed geometric map is the same as the resolution of the reconstructed texture map; a second acquisition code for enabling the at least one processor to obtain a point in the source point cloud corresponding to a geometric pixel of the geometric map and a texture pixel of the texture map; a selection code for enabling the at least one processor to select the color of the texture pixel based on the color of the obtained point; and a third generation code for enabling the at least one processor to generate an encoded video bitstream based on the selected color, the geometric map, and the texture map.

[0009] In each embodiment, a non-volatile computer-readable medium stores computer instructions for encoding a video bitstream using video point cloud coding, which, when executed by at least one processor, causes the at least one processor to: obtain a source point cloud; generate a geometric map and a texture map based on the source point cloud; use lossy coding without recoloring to generate a reconstructed geometric map and a reconstructed texture map, wherein the resolution of the reconstructed geometric map is the same as the resolution of the reconstructed texture map; obtain a point in the source point cloud corresponding to a geometric pixel of the geometric map and a texture pixel of the texture map; select a color of the texture pixel based on the color of the obtained point; and generate an encoded video bitstream based on the selected color, the geometric map, and the texture map. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The following detailed description and accompanying drawings will make further features, properties and various advantages of the subject matter of the present application more apparent, in which:

[0011] Figure 1 is a schematic diagram of a simplified block diagram of a communication system of an embodiment;

[0012] Figure 2 is a schematic diagram of a simplified block diagram of a streaming system of an embodiment;

[0013] Figure 3 is a schematic diagram of a simplified block diagram of a video encoder of an embodiment;

[0014] Figure 4 is a schematic diagram of a simplified block diagram of a video decoder of an embodiment;

[0015] Figure 5 is an example of a forward search between a source point cloud and a reconstructed point cloud of an embodiment;

[0016] Figure 6 is an example of a backward search between a source point cloud and a reconstructed point cloud of an embodiment;

[0017] Fig. 7A is an example of an encoder pipeline with recoloring of one embodiment;

[0018] Figure 7B is an example of an encoder pipeline without recoloring of one embodiment;

[0019] Figure 8 is an example of a partially occupied block of one embodiment;

[0020] FIG. 9A to FIG. 9B is a flow chart of a method performed by an embodiment;

[0021] Fig.10 is a schematic diagram of a device of an embodiment;

[0022] Fig.11 is a schematic diagram of a computer system suitable for implementing various embodiments. DETAILED DESCRIPTION

[0023] The idea behind video-based point cloud compression (V-PCC) is to use existing video codecs to compress the geometry, occupancy, and texture of a dynamic point cloud into three separate video sequences. The additional metadata required to interpret the three video sequences can be compressed separately. A small portion of the entire bitstream is metadata, which can be efficiently encoded / decoded using software implementations. The main part of this information can be handled by the video codec.

[0024] The present invention relates to an annealing iterative geometry smoothing scheme to avoid over-smoothing in an iterative smoothing framework. The present invention relates to using a combination of mean statistics and median statistics to derive a reference point, aiming to reduce the computational complexity of using a pure median.

[0025] refer to Figures 1 to 4 , which describes an embodiment of the present application for implementing the encoding structure and decoding structure of the present application. The encoding structure and decoding structure of the present application can implement various aspects of the V-PCC described above.

[0026] Figure 1 A simplified block diagram of a communication system 100 of an embodiment of the present application is shown. The system 100 may include at least two terminals 110, 120 interconnected by a network 150. For unidirectional transmission of data, the first terminal 110 may encode video data at a local location for transmission to another terminal 120 via the network 150. The second terminal 120 may receive the encoded video data of another terminal from the network 150, decode the encoded data and display the recovered video data. Unidirectional data transmission may be common in media service applications and the like.

[0027] Figure 1 A second pair of terminals 130, 140 is shown, which are provided to support bidirectional transmission of encoded video, such as may occur during a video conference. For bidirectional transmission of data, each terminal 130, 140 can encode video data captured at a local location for transmission to the other terminal via a network 150. Each terminal 130, 140 can also receive encoded video data transmitted by the other terminal, can decode the encoded data, and can display the recovered video data at a local display device.

[0028] exist Figure 1 In the present application, the terminals 110-140 can be, for example, servers, personal computers, smart phones, and / or any other type of terminals. For example, the terminals (110-140) can be laptops, tablet computers, media players, and / or dedicated video conferencing equipment. Network 150 represents any number of networks that transmit encoded video data between the terminals 110-140, including, for example, wired and / or wireless communication networks. The communication network 150 can exchange data in circuit switching channels and / or packet switching channels. Typical networks include telecommunication networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network 150 may be unimportant to the operation of the present application unless explained below.

[0029] As an example of application of the subject matter of this application, Figure 2 The deployment of video encoders and decoders in a streaming environment is shown. The subject matter of the present application can be used with other video-enabled applications including, for example, video conferencing, digital television, applications storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0030] like Figure 2 As shown, the streaming system 200 may include a capture subsystem 213, which includes a video source 201 and an encoder 203. The streaming system 200 may further include at least one streaming server 205 and / or at least one streaming client 206.

[0031] A video source 201 may create, for example, a stream 202 including a 3D point cloud corresponding to a 3D video. The video source 201 may include, for example, a 3D sensor (e.g., a depth sensor) or a 3D imaging technology (e.g., one or more digital cameras), and a computing device for generating a 3D point cloud using data received from the 3D sensor or the 3D imaging technology. The sample stream 202, which may have a high amount of data compared to an encoded video stream, may be processed by an encoder 203 coupled to the video source 201. The encoder 203 may include hardware, software, or a combination thereof to enable or implement various aspects of the subject matter of the present application as described in detail below. The encoder 203 may also generate an encoded video stream 204. The encoded video stream 204, which may have a lower amount of data compared to the uncompressed stream 202, may be stored on a streaming server 205 for future use. One or more streaming clients 206 may access the streaming server 205 to extract a video stream 209, which may be a copy of the encoded video stream 204.

[0032] The streaming client 206 may include a video decoder 210 and a display 212. The video decoder 210 may, for example, decode a video stream 209 that is an incoming copy of an encoded video stream 204 and create an outgoing video sample stream 211 that may be presented on a display 212 or another presentation device (not shown). In some streaming systems, the video streams 204, 209 may be encoded according to certain video encoding / compression standards. Examples of such standards include, but are not limited to, ITU-T H.265, Versatile Video Coding (VVC), and MPEG / V-PCC.

[0033] refer to Figure 3 to Figure 4 , some aspects of V-PCC that can be performed by embodiments of the present application are described below.

[0034] Figure 3An exemplary functional block diagram of the video encoder 203 according to an embodiment of the present application is shown.

[0035] like Figure 3 As shown, the video encoder 203 may receive one or more point cloud frames 350, and generate a geometry image 352, a texture image 356, and an occupancy map 334 based on the point cloud frame 350. The video encoder 203 may compress the geometry image 352 into a compressed geometry image 362, compress the texture image 356 into a compressed texture image 364, and compress the occupancy map 334 into a compressed occupancy map 372. The multiplexer 328 of the video encoder 203 may form a compressed code stream 374, which includes the compressed geometry image 362, the compressed texture image 364, and the compressed occupancy map 372.

[0036] More specifically, in various embodiments, the video encoder 203 may include a block generation module 302 that divides the point cloud frame 350 into a plurality of patches. A block is a useful entity of V-PCC. The block generation process includes decomposing the point cloud frame 350 into a minimum number of blocks with smooth boundaries while also minimizing reconstruction errors. The encoder of the present application may perform various methods to generate such a decomposition operation.

[0037] The video encoder 203 may include a block packing module 304 that performs a packing process. The packing process includes mapping the extracted multiple blocks onto a 2D grid while minimizing unused space and ensuring that each M×M (e.g., 16×16) block of the grid is associated with a different block. An efficient block packing operation directly affects compression efficiency by minimizing unused space or by ensuring temporal consistency. The block packing module 304 may generate an occupancy map 334.

[0038] The video encoder 203 may include a geometry image generation module 306 and a texture image generation module 308. To better handle the situation where multiple points are projected to the same sample, each block may be projected onto two images referred to as layers. For example, the geometry image generation module 306 and the texture image generation module 308 may utilize the 3D to 2D mapping computed during the packing process of the block packing module 304 to store the geometry and texture of the point cloud as images (aka "layers"). The generated images / layers may be stored as one or more video frames and compressed by a video codec (e.g., an HM video codec) using configuration information provided as parameters.

[0039] In various embodiments, based on the input point cloud frame 350 and the occupancy map 334, the geometry image generation module 306 generates a geometry image 352, and the texture image generation module 308 generates a texture image 356. In one embodiment, the geometry image 352 can be represented by a monochrome frame of size WxH in YUV420-8 bit format. In one embodiment, the occupancy map 334 image consists of a binary map that indicates whether each cell of the grid belongs to empty space or to a point cloud. In order to generate the texture image 356, the texture image generation module 308 can use the reconstructed / smoothed geometry 358 to calculate a plurality of colors to be associated with a plurality of resampled points.

[0040] The video encoder 203 may also include an image filling module 314 and an image filling module 316, which are used to fill the geometric image 352 and the texture image 356, respectively, to form a filled geometric image 354 and a filled texture image 360. Image filling (also known as "background filling") simply fills the unused space of the image with redundant information. A good background filling technique is a filling technique that minimally increases the bit rate without introducing significant coding distortion around block boundaries. The image filling module 314 and the image filling module 316 can use the occupancy map 334 to form the filled geometric image 354 and the filled texture image 360, respectively. In various embodiments, the video encoder 203 may include a group expansion module 320 to form a filled texture image 360.

[0041] The video encoder 203 may include a video compression module 322 and a video compression module 324 for compressing the filled geometry image 354 and the filled texture image 360 ​​into a compressed geometry image 362 and a compressed texture image 364, respectively.

[0042] The video encoder 203 may include an entropy compression module 318 for losslessly encoding 366 the occupancy map 334 and a video compression module 326 for losslessly encoding 368 the occupancy map 334 .

[0043] In various embodiments, the video encoder 203 may include a smoothing module 310 for generating a smoothed geometric shape 358 by using the block information 332 and the reconstructed geometric image 365 provided by the video compression module 322. The smoothing process of the smoothing module 310 may be intended to mitigate potential discontinuities that may occur at block boundaries due to compression artifacts. The smoothed geometric shape 358 may be used by the texture image generation module 308 to generate a texture image 356.

[0044] The video encoder 203 may further include an auxiliary block information compression module 312 for forming compressed auxiliary block information 370 , which is provided by the multiplexer 328 into a compressed code stream 374 .

[0045] Figure 4 An exemplary functional block diagram of the video decoder 210 according to an embodiment of the present application is shown.

[0046] like Figure 4 As shown, the video decoder 210 may receive the encoded code stream 374 from the video encoder 203 to obtain the compressed texture image 362, the compressed geometric image 364, the compressed occupancy map 372, and the compressed auxiliary block information 370. The video decoder 210 may decode the compressed texture image 362, the compressed geometric image 364, the compressed occupancy map 372, and the compressed auxiliary block information 370 to respectively obtain the decompressed texture image 460, the decompressed geometric image 462, the decompressed occupancy map 464, and the decompressed auxiliary block information 466. Then, the video decoder 210 may generate a reconstructed point cloud 474 based on the decompressed texture image 460, the decompressed geometric image 462, the decompressed occupancy map 464, and the decompressed auxiliary block information 466.

[0047] In various embodiments, the video decoder 210 may include a demultiplexer 402 for separating the compressed texture image 362 , the compressed geometry image 364 , the compressed occupancy map 372 , and the compressed auxiliary block information 370 in the received compressed code stream 374 .

[0048] The video decoder 210 may include a video decompression module 404, a video decompression module 406, an occupancy map decompression module 408, and an auxiliary block information decompression module 410, which respectively decode the compressed texture image 362, the compressed geometry image 364, the compressed occupancy map 372, and the compressed auxiliary block information 370.

[0049] The video decoder 210 may include a geometry reconstruction module 412 that obtains a reconstructed (three-dimensional) geometry 468 based on the decompressed geometry image 462 , the decompressed occupancy map 464 and the decompressed auxiliary block information 466 .

[0050] The video decoder 210 may include a smoothing module 414 that smoothes the reconstructed geometry 468 to obtain a smoothed geometry 470. The smoothing process may be intended to mitigate potential discontinuities that may occur at block boundaries due to compression artifacts.

[0051] The video decoder 210 may include a texture reconstruction module 416 that is configured to obtain a reconstructed texture 472 based on the decompressed texture image 460 and the smoothed geometry 470.

[0052] The video decoder 210 may include a color smoothing module 418 that smooths the color of the reconstructed texture 472 to obtain a reconstructed point cloud 474. Non-adjacent blocks in 3D space are typically packed into a 2D video in a contiguous order. This means that pixel values from non-adjacent blocks may be mixed together by a block-based video codec. The color smoothing of the color smoothing module 418 may be aimed at reducing visible artifacts that appear at block boundaries.

[0053] Recolorization in Video-based Point Cloud Compression (V-PCC)

[0054] In the MPEG PCC Test Model Category 2 (TMC2) model (which may correspond to V-PCC), when the geometry is lossily encoded and repeated positions are merged, a recoloring process may be performed on the encoder side, which may be referred to as color transfer. An example of the recoloring algorithm is given below with reference to Figure 5 and Figure 6 which gives an example of the recoloring algorithm.

[0055] Given a source point cloud 502, attributes, and a target point p included in the reconstructed point cloud 504 r , the color transfer procedure (i.e., the recoloring process) may determine the attribute value of the reconstructed point cloud that minimizes the distortion of the attributes. The method may be implemented as follows.

[0056] For each target point p r : 1 - Find N1 (1 < N1) nearest neighbor points of p r in the source point cloud 502 and create a set of points represented by Ψ1. This step may correspond to the Figure 5 forward search shown in.

[0057] 2 - Find a set of points in the source point cloud 502 such that p r belongs to a set of N2 nearest neighbor points of this set of points in the reconstructed point cloud 504. Denote this set of points by Ψ2. This step may correspond to the Figure 6 forward search shown in.

[0058] 3 - Calculate the distance weighted average of the points in Ψ1 and Ψ2 by the following formula:

[0059]

[0060] where Δ(a, b) represents the Euclidean distance between point a and point b, and c(q) represents the color of point q.

[0061] 4- Calculation and The average (or weighted average, where the number of points in each group serves as the weight) of the points in the cluster is taken and used as the color of the centroid.

[0062] 5- In some embodiments, if the absolute difference between the color of some points and the centroid is greater than a threshold th c , you can perform a backward search starting from the centroid to exclude these points.

[0063] 6- In some embodiments, and The average value (or weighted average value, with the number of points in each group as the weight) can be updated and migrated to p r .

[0064] In the current design of TMC2, the recoloring process can be quite complex because a K-dimensional (KD) tree data structure is used in the nearest neighbor search and the recoloring operation is applied to each point in the reconstructed point cloud.

[0065] In the current design of TMC2, the recoloring process can be quite complex because a K-dimensional (KD) tree data structure is used in the nearest neighbor search and the recoloring operation is applied to each point in the reconstructed point cloud. In various embodiments, the entire recoloring process can be bypassed by generating a texture map directly from the original point cloud. In various embodiments, the use of the recoloring process can be conditional. For example, the recoloring process can be applied to areas with large geometric distortion due to lossy geometric compression.

[0066] The embodiments disclosed herein may be used alone or in any order. Further, each embodiment (e.g., encoder and decoder) may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-volatile computer-readable medium.

[0067] Bypassing Recoloring

[0068] like Fig. 7A As shown, in V-PCC, recoloring can be applied when the geometry is lossy coded, and the colors in the reconstructed point cloud are recalculated based on the reconstructed geometry, or for example interpolated. Then, a texture map is generated based on the recolored point cloud.

[0069] For example, Fig. 7AAs shown, in the method 700A using recoloring, an occupancy map 711 and a geometry map 712 may be generated using an input point cloud 710. Then, a reconstructed occupancy map 713 and a reconstructed geometry map 714 may be generated using lossy coding, and a geometrically reconstructed point cloud 715 may be generated based on the reconstructed occupancy map 713 and the reconstructed geometry map 714. The geometrically reconstructed point cloud 715 may be recolored using the input point cloud 710 to generate a recolored point cloud 716, and the recolored point cloud 716 may be used together with the occupancy map 711 to generate a texture map 717. A reconstructed texture map 718 may be generated using lossy coding, and a reconstructed point cloud 719 may be generated using the reconstructed texture map 718 and the geometrically reconstructed point cloud 715.

[0070] In various embodiments, when generating a geometric map, a texture map can be directly generated using the input point cloud, and then the generated occupancy map, geometric map, and texture map can be encoded and reconstructed. An example of this process is shown in Figure 7B , where the recoloring process can be completely removed and the entire workflow is greatly simplified.

[0071] For example, Figure 7B As shown, in the method 700B without recoloring, the input point cloud 710 can be directly used to generate an occupancy map 721, a geometry map 722, and a texture map 723, and lossy coding can be used to generate a reconstructed occupancy map 721, a reconstructed geometry map 722, and a reconstructed texture map 723. Then, the reconstructed point cloud 727 can be generated using the reconstructed occupancy map 721, the reconstructed geometry map 722, and the reconstructed texture map 723.

[0072] In various embodiments, a texture map may be generated using a geometric map. The texture map may have the same resolution as the geometric map, and they share the same atlas information. Assume that T x,y is the occupied pixel in the texture map and G x,y are corresponding pixels in the geometric figure, and they correspond to the same point P in 3D space. The original color value of P can then be assigned as T x,y The value of .

[0073] If the occupancy map is lossy encoded, the texture map generated by bypassing reshading can be further processed based on the reconstructed occupancy map. Accordingly, "half-occupied" pixels can be filled with some values, which are pixels that are occupied in the reconstructed full-size occupancy map but not occupied in the original full-size occupancy map.

[0074] The distortion of the occupancy map can be caused by two main factors, namely, downsampling of the occupancy map and lossy compression of the occupancy map. Assume that the occupancy map is losslessly encoded with the precision of B×B blocks, where B>1 is an integer. In this case, when a block of the occupancy map is occupied by at least one point, all points of the block in the reconstructed occupancy map will be occupied.

[0075] Consider the example of B=4. Figure 8 As shown, a 4×4 block is partially occupied, where the pixels marked with "X" are occupied and their corresponding pixels in the texture map are valid. The remaining pixels are defined as "half-occupied" because they are not occupied in the original occupancy map, but are occupied in the reconstructed occupancy map. Therefore, in the reconstructed block, the "half-occupied" pixels will be considered occupied. In various embodiments, the "half-occupied" pixels can be filled with appropriate values ​​to improve the reconstruction quality. The filling value can be determined using the occupied pixels in the image.

[0076] In various embodiments, the filling process of "half-occupied" pixels is performed on the basis of B×B non-overlapping blocks. For each B×B block, the "half-occupied" pixels are filled with the average value of the occupied pixels in the same block.

[0077] In various embodiments, the filling process of "half-occupied" pixels is performed on the basis of B×B non-overlapping blocks. For each B×B block, the "half-occupied" pixels are filled using the average value of multiple nearest neighboring occupied pixels in the same block. The averaging operation can be weighted using the distance to the filled pixel.

[0078] In various embodiments, a filling process of "half-occupied" pixels is performed on the entire image. For each "half-occupied" pixel, the average value of multiple nearest neighboring occupied pixels in the image can be used to fill it. The averaging operation can be weighted by the distance to the filled pixel.

[0079] Conditional recoloring

[0080] Conditional recoloring may involve a compromise between a normal recoloring scheme and a bypass recoloring scheme. In various embodiments, a bypass recoloring scheme may be used to directly derive some pixels in the texture map from the original point cloud, while the remaining pixels may be derived from the reconstructed point cloud using a normal recoloring algorithm. The method applied to a pixel may be determined by the geometric distortion of the corresponding point in 3D. For example, if the geometric distortion is large, a normal recoloring scheme is applied, otherwise a bypass recoloring scheme is applied. Note that the geometric distortion consists of the distortion in the geometry map and the distortion in the occupancy map.

[0081] In various embodiments, if the geometry (eg, geometry map and occupancy map) is losslessly encoded, a recoloring bypass scheme may be applied to all pixels of the texture map, and a normal recoloring scheme may be omitted.

[0082] In various embodiments, "half-occupied" pixels may be processed by a normal recoloring scheme.

[0083] In various embodiments, all pixels of a BxB block including at least one "half-occupied" pixel may be processed by a normal recoloring scheme.

[0084] In various embodiments, the original geometry may be compared to the corresponding compressed reconstructed geometry. For a given 2D position, if the compression process causes the corresponding pixel value in the geometry to differ greatly, the corresponding pixel in the texture map may be processed using a normal recoloring scheme; otherwise, it may be processed using a scheme that bypasses recoloring. A threshold may be defined to determine whether the change in geometry is large. A fixed threshold may be used for all sequences, or the threshold may be changed for each sequence, or the threshold may be adaptively updated for each sequence. The threshold may be sent in the bitstream as high-level syntax information or metadata.

[0085] Note that any combination of the conditions in the embodiments disclosed herein may be applied.

[0086] Accordingly, in each embodiment, a texture map can be directly generated using the input point cloud without recoloring using the reconstructed point cloud. This scheme can be referred to as a scheme that bypasses recoloring. If the occupancy map is lossy encoded, the texture map generated by bypassing recoloring can be further processed based on the reconstructed occupancy map. The goal can be to fill "half-occupied" pixels with some values, which are occupied in the reconstructed full-size occupancy map but not occupied in the original full-size occupancy map. The filled value can be determined using the occupied pixels in the image. The filling process of the "half-occupied" pixels can be performed on a B×B non-overlapping block basis. The filling process of the "half-occupied" pixels can be performed on the entire image.

[0087] In various embodiments, some pixels in the texture map may be derived directly from the original point cloud by a bypass recoloring scheme, and the remaining pixels may be derived from the reconstructed point cloud by a normal recoloring algorithm (which may be referred to as a conditional recoloring scheme). The method used for a pixel may be determined by the geometric distortion of the corresponding point in 3D. If the geometric distortion is large, normal recoloring may be applied, otherwise bypass recoloring may be applied. Note that the geometric distortion may include distortion in the geometry map and distortion in the occupancy map.

[0088] Fig.9Ais a flow chart of a method 900A for encoding a video bitstream using video point cloud coding. In some implementations, Fig.9A One or more process blocks of may be performed by encoder 203. In some implementations, Fig.9A One or more process blocks of may be performed by another device or a group of devices (eg, decoder 210 ) separate from or including encoder 203 .

[0089] like Fig.9A As shown, in operation 911 , method 900A may include obtaining a source point cloud.

[0090] At operation 912 , method 900A may include generating a geometry map and a texture map based on the source point cloud.

[0091] In operation 913, method 900A may include generating a reconstructed geometry map and a reconstructed texture map using lossy coding, wherein the resolution of the reconstructed geometry map is the same as the resolution of the reconstructed texture map. In various embodiments, the reconstructed geometry map and the reconstructed texture map may be generated without using reshading.

[0092] In operation 914 , method 900A may include obtaining a point of the source point cloud corresponding to a geometry pixel of the geometry map and a texture pixel of the texture map.

[0093] In operation 915 , method 900A may include selecting a color for the texel based on the obtained color of the point.

[0094] In operation 916 , method 900A may include generating an encoded video stream based on the selected color, the geometry map, and the texture map.

[0095] Fig. 9B is a flow chart of a method 900B for encoding a video bitstream using video point cloud coding. In some implementations, Fig. 9B One or more process blocks of may be performed by encoder 203. In some implementations, Fig. 9B One or more process blocks of may be performed by another device or a group of devices, such as the decoder 210, separate from the encoder 203 or including the encoder 203.

[0096] like Fig. 9B As shown, in operation 921 , method 900B may include generating an occupancy map based on a source point cloud.

[0097] In operation 922, method 900B may include generating a reconstructed occupancy map using lossy encoding, wherein the resolution of the reconstructed occupancy map is the same as the resolution of the reconstructed geometry map and the resolution of the reconstructed texture map. In various embodiments, the reconstructed occupancy map may be generated without using recoloring.

[0098] In various embodiments, the color of the texels may be selected based on the occupancy map.

[0099] In various embodiments, the texture pixel may correspond to a half-occupied pixel, which is unoccupied in the occupancy map and occupied in the reconstructed occupancy map, and the color of the texture pixel may be selected based on an average value of multiple occupied pixels in the same block.

[0100] In various embodiments, the texture pixel may correspond to a half-occupied pixel that is unoccupied in the occupancy map and occupied in the reconstructed occupancy map, and the color of the texture pixel may be selected based on a distance-weighted average of a plurality of nearest neighboring occupied pixels.

[0101] In various embodiments, the texture pixel may correspond to an occupied pixel that is occupied in the occupancy map and occupied in the reconstructed occupancy map, and a color value of a half-occupied pixel may be determined using recoloring, the half-occupied pixel being unoccupied in the occupancy map and occupied in the reconstructed occupancy map.

[0102] In various embodiments, for a block including half-occupied pixels, recoloring may be used to determine multiple color values ​​for multiple pixels in the block.

[0103] In various embodiments, the above points may be obtained based on a comparison between the geometric image and the reconstructed geometric image.

[0104] Although FIG. 9A to FIG. 9B Example block diagrams of methods 900A and 900B are shown, but in some implementations, FIG. 9A to FIG. 9B Methods 900A and 900B may include additional blocks, fewer blocks, different blocks, or blocks arranged differently than those shown in the method 900A. Additionally or alternatively, two or more blocks of method 900A may be performed in parallel. In various embodiments, as needed, FIG. 9A to FIG. 9B Any number of boxes can be in any order with FIG. 9A to FIG. 9B any other boxes.

[0105] Fig.10 1 is a diagram of an apparatus 1000 for encoding a video bitstream using video point cloud coding in various embodiments. Fig.10As shown in , the apparatus 800 includes a first acquisition code 1010 , a first generation code 1020 , a second generation code 1030 , a second acquisition code 1040 , a selection code 1050 and a third generation code 1060 .

[0106] The first acquisition code 1010 may be used to enable at least one processor to obtain a source point cloud.

[0107] The first generating code 1020 may be configured to cause at least one processor to generate a geometry map and a texture map based on a source point cloud.

[0108] The second generation code 1030 may be used to enable at least one processor to generate a reconstructed geometric map and a reconstructed texture map using lossy coding, wherein a resolution of the reconstructed geometric map is the same as a resolution of the reconstructed texture map.

[0109] The second acquisition code 1040 may be used to enable at least one processor to obtain a point in the source point cloud corresponding to a geometric pixel of the geometric image and a texture pixel of the texture image.

[0110] The selection code 1050 may be used to cause at least one processor to select a color for the texture pixel based on the obtained color of the point.

[0111] The third generation code 1060 may be used to enable at least one processor to generate a coded video stream based on the selected color, geometry map, and texture map.

[0112] The above techniques may be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. Fig.11 A computer system 1100 suitable for implementing certain embodiments of the present application is shown.

[0113] Computer software may be encoded using any suitable machine code or computer language, and may be generated using assembly, compilation, linking or similar mechanisms. The code includes instructions that may be directly executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc., or executed through operations such as code interpretation and microcode execution.

[0114] These instructions may be executed in a variety of types of computers or components, including, for example, personal computers, tablets, servers, smart phones, gaming devices, IoT devices, etc.

[0115] Fig.11The components shown for computer system 1100 are examples and are not intended to impose any limitations on the scope of use or functionality of computer software implementing embodiments of the present application. The configuration of the components should not be interpreted as having any dependency or requirement on any one or combination of components in the non-limiting embodiments of computer system 1100.

[0116] Computer system 1100 may include certain human-machine interface input devices. Such human-machine interface input devices may respond to input from one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, tapping), visual input (e.g., gestures), olfactory input (not shown). Human-machine interface devices may also be used to capture certain media that are not necessarily directly related to human conscious input, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and videos (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0117] The human-machine interface input device may include one or more of the following (only one of each is shown): keyboard 1101 , mouse 1102 , touchpad 1103 , touch screen 1110 , data gloves, joystick 1105 , microphone 1106 , scanner 1107 , camera 1108 .

[0118] The computer system 1100 may also include certain human-computer interface output devices. Such human-computer interface output devices may stimulate the senses of one or more human users through, for example, tactile output, sound, light, and smell / taste. Such human-computer interface output devices may include tactile output devices (e.g., tactile feedback of a touch screen 1110, a data glove, or a joystick 1105, but there may also be tactile feedback devices that are not used as input devices). For example, such devices may be audio output devices (e.g., speakers 1109, headphones (not shown)), visual output devices, and printers (not shown), wherein visual output devices such as screens 1110, virtual reality glasses (not shown), holographic displays, and smoke cans (not shown), screens 1110 include cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, organic light emitting diode (OLED) screens, each with or without touch screen input capabilities, each with or without tactile feedback capabilities, and some of these screens are capable of outputting two-dimensional visual output information or more than three-dimensional output information by, for example, stereoscopic image output.

[0119] The computer system 1100 may also include human-accessible storage devices and their associated media, such as optical media (including CD / DVD ROM / RW 1120 with CD / DVD) or similar media 1121, thumb drives 1122, removable hard drives or solid-state drives 1123, traditional magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown), and the like.

[0120] Those skilled in the art should also understand that the term "computer-readable medium" used in conjunction with the subject matter of the present application does not include transmission media, carrier waves, or other transient signals.

[0121] The computer system 1100 may also include an interface for connecting one or more communication networks. The network may be, for example, a wireless network, a wired network, an optical network. The network may also be a local network, a wide area network, a metropolitan area network, an Internet of Vehicles and an industrial network, a real-time network, a delay-tolerant network, etc. Examples of networks include local area networks (e.g., Ethernet, wireless LAN), cellular networks (including global mobile communication systems (GSM), third-generation mobile communication systems (3G), fourth-generation mobile communication systems (4G), fifth-generation mobile communication systems (5G), long-term evolution (LTE), etc.), television wired or wireless wide-area digital networks (including cable television, satellite television, and terrestrial broadcast television), Internet of Vehicles and industrial networks (including CANBus), etc. Some networks typically require an external network interface adapter that is connected to some universal data port or peripheral bus 1149 (e.g., a universal serial bus (USB) port of the computer system 1100); other networks are typically connected to the system bus in the following manner so as to be integrated into the kernel of the computer system 1100 (e.g., an Ethernet interface integrated into a personal computer system or a cellular network interface integrated into a smartphone computer system). By using any of these networks, the computer system 1100 can communicate with other entities. Such communications can be one-way, receive-only (e.g., broadcast TV), one-way send-only (e.g., CANbus to certain CANbus devices), or two-way communications, such as with other computer systems using a local area network or a wide area digital network. Such communications can include communications with a cloud computing environment 1155. Certain protocols and protocol stacks can be used on each of those networks and network interfaces as described above.

[0122] The above-mentioned human-machine interface devices, human-accessible storage devices, and network interfaces may be connected to the kernel 1140 of the computer system 1100 .

[0123] The core 1140 may include one or more central processing units (CPUs) 1141, graphics processing units (GPUs) 1142, dedicated programmable processing units in the form of field programmable gate arrays (FPGAs) 1143, hardware accelerators 1144 for specific tasks, etc. These devices, as well as read-only memory (ROM) 1145, random access memory 1146, internal mass storage 1147 (e.g., internal non-user accessible hard disk drives, SSDs), etc., may be interconnected via a system bus 1148. In some computer systems, the system bus 1148 may be accessed in the form of one or more physical plugs, thereby being expanded by additional CPUs, GPUs, etc. Peripheral devices may be connected to the core's system bus 1148 directly or via a peripheral bus 1149. The architecture of the peripheral bus includes PCI, USB, etc. The core 1140 may include a graphics adapter 1150.

[0124] The CPU 1141, GPU 1142, FPGA 1143, and accelerator 1144 may execute certain instructions, the combination of which may constitute the above-mentioned computer code. The computer code may be stored in ROM 1145 or RAM 1146. Intermediate data may also be stored in RAM 1146, while permanent data may be stored in, for example, internal mass storage 1147. Fast storage and reading of any memory device may be enabled by using a cache memory, which may be closely associated with one or more CPUs 1141, GPUs 1142, mass storage devices 1147, ROM 1145, RAM 1146, and the like.

[0125] The computer readable medium may have computer code thereon, and various computer-implemented operations may be performed on the computer code. The medium and computer code may be specially designed and constructed for the purpose of this application, or may be medium and code well known and available to those skilled in the art of computer software.

[0126] As an example and not limitation, a computer system having architecture 1100, and in particular, kernel 1140, can provide functionality implemented by one or more processors (including CPU, GPU, FPGA, accelerator, etc.) executing software in one or more tangible computer-readable media. Such computer-readable media can be media associated with a user-accessible mass storage as described above, as well as some storage of the non-volatile kernel 1140, such as a kernel internal mass storage device 1147 or ROM 1145. The software implementing various embodiments of the present application can be stored in such devices and executed by the kernel 1140. Depending on specific needs, the computer-readable medium may include one or more memory devices or chips. The software can enable the kernel 1140, in particular the processors therein (including CPU, GPU, FPGA, etc.), to perform specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM 1146, and modifying these data structures according to software-defined processes. In addition or as an alternative, the computer system can provide the same functions as logical hard wiring or other components in a circuit (e.g., accelerator 1144), which can replace software or operate with software to perform specific processes or specific parts of specific processes described herein. References to software may include logic, and vice versa, where appropriate. References to computer-readable media may include circuits (e.g., integrated circuits (ICs)) storing executing software, circuits including executing logic, or both, where appropriate. The present application includes any suitable combination of hardware and software.

[0127] Although the present application has described a number of non-limiting embodiments, various changes, permutations, and substitutions of the embodiments are within the scope of the present application. It should therefore be understood that those skilled in the art will be able to design many systems and methods that, although not explicitly shown or described herein, embody the principles of the present application and are therefore within the spirit and scope of the present application.

Claims

1. A method for encoding a video stream using video point cloud coding, characterized in that: include: Get source point cloud; generating a geometric map and a texture map based on the source point cloud; Generating a reconstructed geometry map and a reconstructed texture map using lossy coding without reshading, wherein the resolution of the reconstructed geometry map is the same as the resolution of the reconstructed texture map; Obtaining a point in the source point cloud corresponding to a geometric pixel of the geometric map and a texture pixel of the texture map, wherein the texture pixel corresponds to a half-occupied pixel, the half-occupied pixel is unoccupied in the occupancy map and is occupied in the reconstructed occupancy map, and a color of the texture pixel is selected based on an average value of a plurality of occupied pixels in the same block; Selecting a color of the texture pixel based on the obtained color of the point; and An encoded video stream is generated based on the selected colors, the geometry map, and the texture map.

2. The method according to claim 1, characterized in that Further including: generating an occupancy map based on the source point cloud; A reconstructed occupancy map is generated using lossy encoding without recoloring, wherein a resolution of the reconstructed occupancy map is the same as a resolution of the reconstructed geometry map and a resolution of the reconstructed texture map.

3. The method according to claim 1, characterized in that The color of the texel is selected based on a distance-weighted average of a plurality of nearest neighboring occupied pixels.

4. The method according to claim 1, characterized in that: in, Color values ​​of half-occupied pixels that are unoccupied in the occupancy map and occupied in the reconstructed occupancy map are determined using the recoloring.

5. The method according to claim 1, characterized in that For a block including the half-occupied pixel, a plurality of color values ​​for a plurality of pixels in the block are determined using the recoloring.

6. The method according to claim 1, characterized in that The points are obtained based on a comparison between the geometric map and the reconstructed geometric map.

7. A device for encoding a video code stream using video point cloud coding, characterized in that: include: A first acquisition module is used to obtain a source point cloud; A first generating module, configured to generate a geometric map and a texture map based on the source point cloud; A second generation module, configured to generate a reconstructed geometric map and a reconstructed texture map using lossy coding without recoloring, wherein a resolution of the reconstructed geometric map is the same as a resolution of the reconstructed texture map; a second acquisition module, configured to obtain a point in the source point cloud corresponding to a geometric pixel of the geometric map and a texture pixel of the texture map, wherein the texture pixel corresponds to a semi-occupied pixel, the semi-occupied pixel is unoccupied in the occupancy map and is occupied in the reconstructed occupancy map, and a color of the texture pixel is selected based on an average value of a plurality of occupied pixels in the same block; A selection module, configured to select the color of the texture pixel based on the obtained color of the point; and The third generating module is used to generate a coded video stream based on the selected color, the geometric map and the texture map.

8. The device according to claim 7, characterized in that Further including: a fourth generating module, configured to generate an occupancy map based on the source point cloud; A fifth generation module is configured to generate a reconstructed occupancy map using lossy coding without recoloring, wherein a resolution of the reconstructed occupancy map is the same as a resolution of the reconstructed geometry map and a resolution of the reconstructed texture map.

9. The device according to claim 7, characterized in that The color of the texel is selected based on a distance-weighted average of a plurality of nearest neighboring occupied pixels.

10. The device according to claim 7, characterized in that Color values ​​of half-occupied pixels that are unoccupied in the occupancy map and occupied in the reconstructed occupancy map are determined using the recoloring.

11. The device according to claim 7, characterized in that For a block including the half-occupied pixel, a plurality of color values ​​for a plurality of pixels in the block are determined using the recoloring.

12. The device according to claim 7, characterized in that The points are obtained based on a comparison between the geometric map and the reconstructed geometric map.

13. A computer device comprising a processor and a memory, characterized in that: Instructions are stored in the memory, and the instructions can be executed by the processor to implement the method according to any one of claims 1-6.

14. A non-volatile computer-readable medium storing computer instructions for encoding a video stream using video point cloud coding, characterized in that: When the computer instructions are executed by at least one processor, the at least one processor implements the method according to any one of claims 1 to 6.

Images