Method and device for video processing and medium

Abstract

Embodiments of the present disclosure provide a solution for video processing. A method for video processing is presented. The method comprises: for conversion between a target frame of a point cloud sequence and a bit stream of the point cloud sequence, applying downsampling to a point according to the importance of the point in the target frame, the importance of the point being associated with the visual sensitivity of a human eye to the point; obtaining a final sampling point cloud by combining the plurality of down-sampled points, where a first set of down-sampled points are down-sampled based on importance and a second set of down-sampled points are down-sampled based on structure retention information; and performing conversion based on the final sampling point cloud.

Description

Technical Field

[0001] Embodiments of the present disclosure relate generally to video processing techniques, and more particularly, to point cloud geometry compression based on visual perception. Background Art

[0002] A point cloud is a collection of individual data points in a three-dimensional (3D) plane, where each point has defined coordinates on the X, Y, and Z axes. Therefore, point clouds can be used to represent the physical contents of a three-dimensional space. Point clouds have proven to be a promising way to represent 3D visual data for a variety of immersive applications, from augmented reality to self-driving cars.

[0003] Point cloud codec standards have evolved primarily through the development of the well-known MPEG organization. MPEG stands for Moving Picture Experts Group, one of the main standardization organizations dealing with multimedia. In 2017, the MPEG 3D Graphics Codec Group (3DG) released a call for proposals (CFP) to begin developing a point cloud codec standard. The final standard will include two types of solutions. Video-based point cloud compression (V-PCC or VPCC) is suitable for point sets with relatively uniform point distribution. Geometry-based point cloud compression (G-PCC or GPCC) is suitable for more sparse distributions. However, the codec efficiency of conventional point cloud codec techniques is generally expected to be further improved. Summary of the Invention

[0004] Embodiments of the present disclosure provide a solution for video processing.

[0005] In a first aspect, a method for video processing is provided. The method includes: for conversion between a target frame of a point cloud sequence and a bitstream of the point cloud sequence, downsampling points in the target frame based on their importance, wherein the importance of a point is associated with the visual sensitivity of the human eye to the point; obtaining a final sampled point cloud by combining multiple downsampled points, wherein a first group of downsampled points is downsampled based on importance and a second group of downsampled points is downsampled based on structure-preserving information; and performing conversion based on the final sampled point cloud. In this way, compression quality can be improved.

[0006] In a second aspect, another method for video processing is proposed. The method includes: for conversion between a target frame of a point cloud sequence and a bitstream of the point cloud sequence, reprocessing a set of features associated with the target frame; obtaining a reconstructed point cloud by upsampling the target frame and applying feature enhancement to the upsampled target frame; updating the reconstructed point cloud by adding the residual between the real point cloud and the reconstructed point cloud; and performing conversion based on the updated reconstructed point cloud and the reprocessed set of features. In this way, compression quality can be improved.

[0007] In a third aspect, a device for video processing is provided, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to the first aspect of the present disclosure.

[0008] In a fourth aspect, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores instructions for causing a processor to execute the method according to the first aspect of the present disclosure.

[0009] In a fifth aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by an apparatus for video processing. The method includes: applying downsampling to points in a target frame of a point cloud sequence based on their importance, wherein the importance of a point is associated with the visual sensitivity of the human eye to the point; obtaining a final sampled point cloud by combining multiple downsampled points, wherein a first group of downsampled points is downsampled based on importance and a second group of downsampled points is downsampled based on structure-preserving information; and generating a bitstream based on the final sampled point cloud.

[0010] In a sixth aspect, a method for storing a bitstream of a video is provided. The method includes: applying downsampling to points in a target frame of a point cloud sequence based on their importance, wherein the importance of a point is associated with the visual sensitivity of a human eye to the point; obtaining a final sampled point cloud by combining a plurality of downsampled points, wherein a first group of downsampled points is downsampled based on importance and a second group of downsampled points is downsampled based on structure-preserving information; generating a bitstream based on the final sampled point cloud; and storing the bitstream in a non-transitory computer-readable recording medium.

[0011] In a seventh aspect, another non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a video processing apparatus. The method includes: reprocessing a set of features associated with a target frame of a point cloud sequence; obtaining a reconstructed point cloud by upsampling the target frame and applying feature enhancement to the upsampled target frame; updating the reconstructed point cloud by adding a residual between a true point cloud and the reconstructed point cloud; and generating a bitstream based on the updated reconstructed point cloud and the reprocessed set of features.

[0012] In an eighth aspect, a method for storing a bitstream of a video is provided. The method includes: reprocessing a set of features associated with a target frame of a point cloud sequence; obtaining a reconstructed point cloud by upsampling the target frame and applying feature enhancement to the upsampled target frame; updating the reconstructed point cloud by adding a residual between a true point cloud and the reconstructed point cloud; generating a bitstream based on the updated reconstructed point cloud and the reprocessed set of features; and storing the bitstream in a non-transitory computer-readable recording medium.

[0013] This summary is intended to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become more apparent through the following detailed description with reference to the accompanying drawings.In the exemplary embodiments of the present disclosure, the same reference numerals generally refer to the same components.

[0015] Figure 1 A block diagram illustrating an example video encoding and decoding system according to some embodiments of the present disclosure is shown;

[0016] Figure 2 shows a block diagram of a first example video encoder according to some embodiments of the present disclosure;

[0017] Figure 3 A block diagram illustrating an example video decoder according to some embodiments of the present disclosure is shown;

[0018] Figure 4 The original U-Net network framework is shown;

[0019] Figure 5 The process of the geometric compression method based on visual perception is shown;

[0020] Figure 6 The flow of the importance sampling method while preserving structure is shown;

[0021] Figure 7 A flowchart of a method for video processing according to an embodiment of the present disclosure is shown;

[0022] Figure 8 A flowchart showing a method for video processing according to an embodiment of the present disclosure is shown; and

[0023] Figure 9 A block diagram is shown of a computing device in which various embodiments of the present disclosure may be implemented.

[0024] Throughout the drawings, same or similar reference numbers generally refer to same or similar elements. DETAILED DESCRIPTION

[0025] The principles of the present disclosure will now be described with reference to some embodiments. It should be understood that these embodiments are described only for the purpose of illustrating and helping those skilled in the art to understand and implement the present disclosure, and do not imply any limitation on the scope of the present disclosure. In addition to the methods described below, the disclosure described herein can also be implemented in various ways.

[0026] In the following description and claims, unless defined otherwise, all scientific and technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs.

[0027] References in this disclosure to "one embodiment," "an embodiment," "an example embodiment," and the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment is required to include the particular feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an example embodiment, whether or not explicitly described, it is considered within the knowledge of those skilled in the art to affect such feature, structure, or characteristic in relation to other embodiments.

[0028] It should be understood that although the terms "first" and "second" and the like can be used to describe various elements, these elements should not be limited to these terms. These terms are only used to distinguish one element from another. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element without departing from the scope of the example embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the listed terms.

[0029] The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the example embodiments. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the terms "comprise," "including," "having," "including," and / or "comprising" when used herein indicate the presence of the features, elements, and / or components, etc., but do not exclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. Sample Environment

[0030] Figure 1 is a block diagram illustrating an example point cloud encoding and decoding system 100 that may utilize the techniques of the present disclosure. As shown, the point cloud encoding and decoding system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a point cloud encoding device, and the destination device 120 may also be referred to as a point cloud decoding device. In operation, the source device 110 may be configured to generate encoded point cloud data, and the destination device 120 may be configured to decode the encoded point cloud data generated by the source device 110. The techniques of the present disclosure are generally directed to encoding and decoding (encoding and / or decoding) point cloud data, i.e., supporting point cloud compression. The encoding and decoding may efficiently compress and / or decompress point cloud data.

[0031] Source device 100 and destination device 120 may include any of a wide variety of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets (such as smartphones and mobile phones), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, vehicles (e.g., land or sea vehicles, spacecraft, aircraft, etc.), robots, LIDAR devices, satellites, extended reality devices, etc. In some cases, source device 100 and destination device 120 may be equipped for wireless communication.

[0032] The source device 100 may include a data source 112, a memory 114, a GPCC encoder 116, and an input / output (I / O) interface 118. The destination device 120 may include an input / output (I / O) interface 128, a GPCC decoder 126, a memory 124, and a data consumer 122. According to the present disclosure, the GPCC encoder 116 of the source device 100 and the GPCC decoder 126 of the destination device 120 may be configured to apply the techniques of the present disclosure related to point cloud encoding and decoding. Therefore, the source device 100 represents an example of an encoding device, while the destination device 120 represents an example of a decoding device. In other examples, the source device 100 and the destination device 120 may include other components or arrangements. For example, the source device 100 may receive data (e.g., point cloud data) from an internal source or an external source. Similarly, the destination device 120 may be connected to an external data consumer interface instead of including the data consumer in the same device.

[0033] In general, data source 112 represents a source of point cloud data (i.e., raw, unencoded point cloud data) and can provide a sequential series of "frames" of point cloud data to GPCC encoder 116, which encodes the frames of point cloud data. In some examples, data source 112 generates point cloud data. Data source 112 of source device 100 can include a point cloud capture device, such as any of a variety of cameras or sensors, for example, one or more cameras, an archive containing previously captured point cloud data, a 3D scanner, or a light detection and ranging (LIDAR) device, and / or a data feed interface that receives point cloud data from a data content provider. Thus, in some examples, data source 112 can generate point cloud data based on signals from a LIDAR device. Alternatively or additionally, point cloud data can be computer-generated from a scanner, camera, sensor, or other data. For example, data source 112 can generate point cloud data, or produce a combination of real-time point cloud data, archived point cloud data, and computer-generated point cloud data. In each case, the GPCC encoder 116 encodes the captured, pre-captured, or computer-generated point cloud data. The GPCC encoder 116 can rearrange the frames of the point cloud data from the order in which they were received (sometimes referred to as "display order") to the codec order for encoding and decoding. The GPCC encoder 116 can generate one or more bitstreams comprising the encoded point cloud data. The source device 100 can then output the encoded point cloud data via the I / O interface 118 for receipt and / or retrieval by, for example, the I / O interface 128 of the destination device 120. The encoded point cloud data can be transmitted directly to the destination device 120 via the network 130A via the I / O interface 118. The encoded point cloud data can also be stored on the storage medium / server 130B for access by the destination device 120.

[0034] The memory 114 of the source device 100 and the memory 124 of the destination device 120 can represent general-purpose memory. In some examples, the memory 114 and the memory 124 can store raw point cloud data, such as the raw point cloud data from the data source 112 and the raw, decoded point cloud data from the GPCC decoder 126. Additionally or alternatively, the memory 114 and the memory 124 can store software instructions, such as those executable by the GPCC encoder 116 and the GPCC decoder 126, respectively. Although the memory 114 and the memory 124 are shown separately from the GPCC encoder 116 and the GPCC decoder 126 in this example, it should be understood that the GPCC encoder 116 and the GPCC decoder 126 can also include internal memory for functionally similar or equivalent purposes. Furthermore, the memory 114 and the memory 124 can store encoded point cloud data, such as the encoded point cloud data output from the GPCC encoder 116 and input to the GPCC decoder 126. In some examples, portions of memory 114 and memory 124 may be allocated as one or more buffers, eg, to store raw, decoded, and / or encoded point cloud data. For example, memory 114 and memory 124 may store point cloud data.

[0035] I / O interface 118 and I / O interface 128 may represent a wireless transmitter / receiver, a modem, a wired network component (e.g., an Ethernet card), a wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where I / O interface 118 and I / O interface 128 include wireless components, I / O interface 118 and I / O interface 128 may be configured to transmit data (such as encoded point cloud data) according to a cellular communication standard (such as 4G, 4G-LTE (Long Term Evolution), LTE Advanced, 5G, etc.). In some examples where I / O interface 118 includes a wireless transmitter, I / O interface 118 and I / O interface 128 may be configured to transmit data (such as encoded point cloud data) according to other wireless standards (such as the IEEE 802.11 specification). In some examples, source device 100 and / or destination device 120 may include corresponding system-on-chip (SoC) devices. For example, source device 100 may include a SoC device to perform the functions attributed to GPCC encoder 116 and / or I / O interface 118 , and destination device 120 may include a SoC device to perform the functions attributed to GPCC decoder 126 and / or I / O interface 128 .

[0036] The techniques of the present disclosure can be applied to encoding and decoding to support any of a variety of applications, such as communication between autonomous vehicles, communication between scanners, cameras, sensors and processing devices (such as local or remote servers), geographic mapping, or other applications.

[0037] The I / O interface 128 of the destination device 120 receives the encoded bitstream from the source device 110. The encoded bitstream may include signaling information defined by the GPCC encoder 116 and used by the GPCC decoder 126, such as syntax elements with values ​​representing a point cloud. The decoded data is used by the data consumer 122. For example, the data consumer 122 may use the decoded point cloud data to determine the position of a physical object. In some examples, the data consumer 122 may include a display to present an image based on the point cloud data.

[0038] The GPCC encoder 116 and the GPCC decoder 126 can be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technology is partially implemented in software, the device can store the instructions of the software in a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the technology of the present disclosure. Each of the GPCC encoder 116 and the GPCC decoder 126 can be included in one or more encoders or decoders, either of which can be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device. The device including the GPCC encoder 116 and / or the GPCC decoder 126 may include one or more integrated circuits, microprocessors, and / or other types of devices.

[0039] The GPCC encoder 116 and the GPCC decoder 126 may operate in accordance with a codec standard, such as the Video Point Cloud Compression (VPCC) standard or the Geometric Point Cloud Compression (GPCC) standard. The present disclosure may generally refer to the encoding and decoding (e.g., encoding and decoding) of a frame to include the process of encoding data or decoding data. The encoded bitstream typically includes a series of values ​​for syntax elements representing codec decisions (e.g., codec mode).

[0040] A point cloud can contain a collection of points in 3D space and can have attributes associated with the points. Attributes can be color information such as R, G, B or Y, Cb, Cr, or reflectance information or other properties. Point clouds can be captured by various cameras or sensors (such as LIDAR sensors and 3D scanners) and can also be computer-generated. Point cloud data is used in a variety of applications, including but not limited to architecture (modeling), graphics (3D models for visualization and animation), and the automotive industry (LIDAR sensors to aid navigation).

[0041] Figure 2 is a block diagram illustrating an example of a GPCC encoder 200 according to some embodiments of the present disclosure, which may be Figure 1 An example of a GPCC encoder 116 in the system 100 is shown. Figure 3 is a block diagram illustrating an example of a GPCC decoder 300 according to some embodiments of the present disclosure. The GPCC decoder 300 may be Figure 1 An example of the GPCC decoder 126 in the system 100 is shown.

[0042] In both the GPCC encoder 200 and the GPCC decoder 300, the point cloud position is first encoded and decoded. The attribute encoding and decoding depends on the decoded geometry. Figure 2 and Figure 3 , the region adaptive hierarchical transform (RAHT) unit 218, the surface approximation analysis unit 212, the RAHT unit 314, and the surface approximation synthesis unit 310 are options commonly used for category 1 data. The level of detail (LOD) generation unit 220, the lifting unit 222, the LOD generation unit 316, and the inverse lifting unit 318 are options commonly used for category 3 data. All other units are common between category 1 and category 3.

[0043] For category 3 data, the compressed geometry is typically represented as an octree from the root down to the leaf level for individual voxels. For category 1 data, the compressed geometry is typically represented by a pruned octree (i.e., an octree from the root down to the leaf level for blocks larger than a voxel) plus a model that approximates the surface within each leaf of the pruned octree. In this way, category 1 and category 3 data share the octree codec mechanism, while category 1 data can additionally utilize a surface model to approximate the voxels within each leaf. The surface model used is a triangulation that consists of 1-10 triangles per block, resulting in a triangle soup. Therefore, category 1 geometry codecs are called trisoup geometry codecs, while category 3 geometry codecs are called octree geometry codecs.

[0044] exist Figure 2In the example, the GPCC encoder 200 may include a coordinate transformation unit 202, a color transformation unit 204, a voxelization unit 206, an attribute transfer unit 208, an octree analysis unit 210, a surface approximation analysis unit 212, an arithmetic coding unit 214, a geometric reconstruction unit 216, a RAHT unit 218, an LOD generation unit 220, a lifting unit 222, a coefficient quantization unit 224 and an arithmetic coding unit 226.

[0045] like Figure 2 As shown in the example of , the GPCC encoder 200 can receive a set of locations and a set of attributes. The locations can include the coordinates of a point in the point cloud. The attributes can include information about the point in the point cloud, such as a color associated with the point in the point cloud.

[0046] The coordinate transformation unit 202 can apply a transformation to the coordinates of the point to transform the coordinates from the original domain to the transformed domain. This disclosure may refer to the transformed coordinates as transformed coordinates. The color transformation unit 204 can apply a transformation to convert the color information of the attribute to a different domain. For example, the color transformation unit 204 can convert the color information from the RGB color space to the YCbCr color space.

[0047] In addition, Figure 2 In the example of , the voxelization unit 206 can voxelize the transformed coordinates. Voxelization of the transformed coordinates can include quantizing and removing some points of the point cloud. In other words, multiple points of the point cloud can be classified into a single "voxel", which can then be treated as a point in some aspects. In addition, the octree analysis unit 210 can generate an octree based on the voxelized transformed coordinates. In addition, in Figure 2 In the example of FIG, the surface approximation analysis unit 212 can analyze the points to potentially determine a surface representation of the set of points. The arithmetic coding unit 214 can perform arithmetic coding on syntax elements representing information about the octree and / or information about the surface determined by the surface approximation analysis unit 212. The GPCC encoder 200 can output these syntax elements in a geometry bitstream.

[0048] The geometric reconstruction unit 216 can reconstruct the transformed coordinates of points in the point cloud based on the octree, data indicating the surface determined by the surface approximation analysis unit 212, and / or other information. Due to voxelization and surface approximation, the number of transformed coordinates reconstructed by the geometric reconstruction unit 216 may differ from the number of original points in the point cloud. This disclosure may refer to the generated points as reconstructed points. The attribute transfer unit 208 can transfer attributes of the original points of the point cloud to the reconstructed points of the point cloud data.

[0049] Furthermore, the RAHT unit 218 may apply RAHT coding to the attributes of the reconstruction points. Alternatively or additionally, the LOD generation unit 220 and the lifting unit 222 may apply LOD processing and lifting, respectively, to the attributes of the reconstruction points. The RAHT unit 218 and the lifting unit 222 may generate coefficients based on the attributes. The coefficient quantization unit 224 may quantize the coefficients generated by the RAHT unit 218 or the lifting unit 222. The arithmetic coding unit 226 may apply arithmetic coding to syntax elements representing the quantized coefficients. The GPCC encoder 200 may output these syntax elements in the attribute bitstream.

[0050] exist Figure 3 In the example, the GPCC decoder 300 may include a geometric arithmetic decoding unit 302, an attribute arithmetic decoding unit 304, an octree synthesis unit 306, an inverse quantization unit 308, a surface approximation synthesis unit 310, a geometric reconstruction unit 312, a RAHT unit 314, an LOD generation unit 316, an inverse lifting unit 318, a coordinate inverse transformation unit 320 and a color inverse transformation unit 322.

[0051] The GPCC decoder 300 may obtain a geometry bitstream and an attribute bitstream. The geometry arithmetic decoding unit 302 of the decoder 300 may apply arithmetic decoding (e.g., CABAC or other types of arithmetic decoding) to syntax elements in the geometry bitstream. Similarly, the attribute arithmetic decoding unit 304 may apply arithmetic decoding to syntax elements in the attribute bitstream.

[0052] The octree synthesis unit 306 may synthesize the octree based on syntax elements parsed from the geometry bitstream. In the case where surface approximation is used in the geometry bitstream, the surface approximation synthesis unit 310 may determine the surface model based on the syntax elements parsed from the geometry bitstream and based on the octree.

[0053] Furthermore, the geometric reconstruction unit 312 may perform reconstruction to determine the coordinates of the points in the point cloud. The coordinate inverse transformation unit 320 may apply an inverse transformation to the reconstructed coordinates to convert the reconstructed coordinates (positions) of the points in the point cloud from the transformed domain back to the original domain.

[0054] In addition, Figure 3 In the example of , the inverse quantization unit 308 may inverse quantize the property value. The property value may be based on syntax elements obtained from the property bitstream (eg, including syntax elements decoded by the property arithmetic decoding unit 304).

[0055] Depending on how the attribute values ​​are encoded, the RAHT unit 314 may perform RAHT decoding to determine color values ​​for the points of the point cloud based on the inverse quantized attribute values. Alternatively, the LOD generation unit 316 and the inverse lifting unit 318 may use a level of detail based technique to determine color values ​​for the points of the point cloud.

[0056] In addition, Figure 3 In the example of , the color inverse transform unit 322 can apply an inverse color transform to the color values. The inverse color transform can be the inverse of the color transform applied by the color transform unit 204 of the encoder 200. For example, the color transform unit 204 can transform the color information from the RGB color space to the YCbCr color space. Correspondingly, the color inverse transform unit 322 can transform the color information from the YCbCr color space to the RGB color space.

[0057] Figure 2 and Figure 3 The various units are shown to help understand the operations performed by the encoder 200 and the decoder 300. These units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. A fixed-function circuit refers to a circuit that provides a specific function and is preset with respect to the operations that can be performed. A programmable circuit refers to a circuit that can be programmed to perform various tasks and provides flexible functionality in the operations that can be performed. For example, a programmable circuit can execute software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. A fixed-function circuit can execute software instructions (e.g., to receive parameters or output parameters), but the type of operation performed by the fixed-function circuit is generally immutable. In some examples, one or more of these units can be different circuit blocks (fixed-function or programmable), and in some examples, one or more of these units can be integrated circuits.

[0058] The following describes some exemplary embodiments of the present disclosure in detail. It should be understood that the section headings used in this document are for ease of understanding and do not limit the embodiments disclosed in a section to that section. In addition, although some embodiments are described with reference to GPCC or other specific point cloud codecs, the disclosed technology is also applicable to other point cloud codec technologies. In addition, although some embodiments describe the point cloud encoding and decoding steps in detail, it should be understood that the corresponding decoding steps for de-encoding will be implemented by the decoder. 1. Brief Overview The present disclosure relates to point cloud encoding and decoding technology. Specifically, it relates to learning-based point cloud geometry compression. This concept can be combined with point cloud encoding and decoding standards, such as the geometry-based point cloud compression (G-PCC) currently under development. 2. Abbreviation G-PCC geometry-based point cloud compression MPEG Moving Picture Experts Group 3. Introduction In point cloud compression, traditional octree, grid, map, and attribute codecs provide the fundamental concepts and frameworks for compression. Following their encoding principles and modular structures, various signal processing methods have been used to design new modules or optimize and enhance existing ones. This is also true for learning-based point cloud compression, which can replace traditional modules with neural network models and optimize model parameters through data-driven optimization. In deep learning of point clouds (Guo, Yulan et al., "Deep learning for 3d point clouds: A survey", IEEE Transactions on Analysis and Machine Intelligence 43.12 (2020): 4338-4364), for CV tasks such as point cloud classification and segmentation, there are many encoder network structures that can effectively learn compact features that contain rich details of the original point cloud. For processing tasks such as point cloud generation, complementation, enhancement, and denoising, there are many decoder network structures that can output high-quality point clouds from less information. There are also point cloud autoencoder networks that can learn symmetric encoding and decoding transformations. Through the transformation, the original data can be transformed into a latent space that is more conducive to effective compression. Quach et al. (Quach, Maurice, Giuseppe Valenzise, ​​and Frederic Dufaux, “Learning convolutional transforms for lossy point cloud geometry compression,” in 2019 IEEE International Conference on Image Processing (ICIP), IEEE, 2019), Wang et al. (Wang, Jianqiang et al., “Lossy point cloud geometry compression via end-to-end learning,” in IEEE Transactions on Circuits and Systems for Video Technology 31.12(2021): 4909-4923), and Guarda et al. (Guarda, André FR, Nuno MM Rodrigues, and Fernando Pereira, “Point cloud coding: Adopting a deep A deep learning-based approach (Point Cloud Encoding and Decoding: A Deep Learning-Based Approach), presented at the 2019 Picture Coding and Decoding Workshop (PCS), IEEE, 2019, proposed a deep learning-based point cloud compression framework. This approach transforms the point cloud geometry into an occupancy grid in the form of a volumetric model and designs a 3D CNN-based autoencoder network to learn the encoding and decoding transformation through end-to-end training. A binary loss function is used to optimize geometric distortion, and intermediate features are estimated probabilistically using a factorization-based entropy model. 3.1 U-Net Model U-Net (Ronneberger, Olaf, Philipp Fischer and Thomas Brox, "U-net: Convolutional networks for biomedical image segmentation", International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, Cham, 2015) was originally proposed to solve the problem of medical image segmentation. Its network structure is as follows Figure 4 As shown in Figure 2. Overall, this is also an encoder-decoder structure, where the encoder part can be seen as feature extraction and the decoder part is upsampling. Since the overall structure of the network is a large letter U, it is called U-Net. This structure first convolves and pools the image, then upsamples or deconvolves the obtained feature map to obtain a larger feature map, splices it with the previous feature map of the same dimension, and then convolves and upsamples the spliced ​​feature map to obtain the corresponding feature map, which is then spliced, convolved, and upsampled with the previous feature. After four upsamplings, a prediction result with the same size as the input image is obtained. Since this U-shaped network structure can effectively obtain context and location information, it has been widely used in various tasks. 3.2 Sparse Convolution To exploit the sparsity of point clouds, scholars have conducted many explorations, such as octree-based CNN (Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong, “O-CNN: Octree-based convolutional neural networks for 3D shape analysis,” ACM Transactions on Graphics (TOG), vol. 36, no. 4, pp. 1-11, 2017) and sparse CNN (Christopher Choy, JunYoung Gwak, and Silvio Savarese, “4D spatio-temporal convnets: Minkowski convolutional neural networks,” in Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3075-3084; Benjamin Graham, Martin Engelcke, and Laurens van der Maaten, “3D semantic segmentation 3D semantic segmentation with submanifold sparse convolutional networks,” in Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, pp. 9224–9232; and Benjamin Graham, “Unsupervised learning with sparse space-and-time autoencoders,” ArXiv, vol. abs / 1811.10355, 2018. For sparse CNNs, the data tensor is represented by a set of coordinates C and associated features F. The convolution aggregates features only for positively occupied coordinates. It is defined in Christopher Choy, JunYoung Gwak, and Silvio Savarese, “4D spatio-temporal convnets: Minkowski convolutional neural networks,” in Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3075–3084: Among them C in and C out are the input coordinates and the output coordinates. and are the input eigenvector and output eigenvector at coordinate t. N 3 (t,C in )={i|t+i∈C in ,i∈N 3} Define a 3D convolution kernel covering the set of locations centered at t, where i is in C in There is an offset in W i Denotes the kernel value at offset i. This sparse convolution exploits the sparsity of point clouds to reduce complexity and is only computed on occupied voxels. 4. Question Existing learning-based point cloud geometry compression methods have the following problems: 1. Most current learning-based point cloud geometry compression methods achieve compression by transmitting low-resolution point clouds and corresponding features. However, these low-resolution point clouds often suffer from redundancy and illogical structure. In theory, regions with simple structures require fewer points to decode, while regions with complex structures require more points to fit the region's structure. However, the point clouds obtained by most methods are uniform. 2. Most learning-based point cloud compression methods do not perform additional processing on the features obtained from entropy decoding, but directly use the features to reconstruct the point cloud geometry. This will weaken the feature representation ability of the original features. 3. Currently, progressive upsampling is a common reconstruction method, but the equal importance of each point in each stage of the reconstruction constraint may lead to poor visual quality of reconstruction. 4. Upsampling is a dense reconstruction of point cloud coordinates, but this reconstruction capability is relatively limited. Combining other processing methods with point cloud upsampling methods can improve the final reconstruction quality of the point cloud. However, most current methods no longer process the point cloud obtained by upsampling to further improve point cloud accuracy. 5. Detailed solution In order to solve the above problems and some other problems not mentioned, the following methods are disclosed. The solutions should be considered as examples to explain the general concept and should not be interpreted in a narrow sense. In addition, these solutions can be applied alone or combined in any way. 1) In the following discussion, the term "encoder" refers to a model that encodes and decodes information to be transmitted through a signal. The term "decoder" refers to a model that decodes compressed bits to obtain the information transmitted through the signal. 2) It is proposed to downsample the point cloud based on the importance of each point to obtain salient points. a. In one example, the importance of points can reflect the visual sensitivity of human eyes to these points. i. In one example, the importance of a point with higher visual sensitivity may be higher than the importance of a point with lower visual sensitivity. ii. In one example, the importance of high-frequency points may be higher than that of low-frequency points. 1. In one example, high-frequency points may have higher visual sensitivity than low-frequency points. b. In one example, importance may be evaluated based on geometric properties of the point cloud, such as geometric structure, local information of the geometry, or global information of the geometry. i. In addition, the geometric structure can be used to evaluate the importance of each point in the point cloud. 1. In one example, a combination of local and global geometric information may be used to characterize the geometric structure. a. In one example, local geometric information can be represented by a fast point feature histogram of a point cloud. b. In one example, global geometric information can be obtained by evaluating clusters among all clusters obtained from the point cloud clustering process. c. In one example, importance can be evaluated based on attribute characteristics of the point cloud (such as color information). d. In one example, importance can be assessed based on both geometric and attribute properties of the point cloud. e. In one example, a learning-based method can be used to obtain the importance of each point, such as the importance based on local geometric information and global geometric information. i. In one example, a neural network-based learning method can be used to obtain the importance of each point. 1. In one example, a neural network similar to a U-Net structure network can be used. a. In one example, sparse convolution can be used as a basic operation in a convolutional network. f. In one example, the importance of each point in the point cloud learned by the network can be ranked. g. In one example, points with higher importance can be sampled by sorting them in descending order. h. In one example, high frequency points can be sampled by sorting them in descending order of importance. 3) It is proposed to use structure-preserving information to obtain the skeleton structure of point clouds. a. In one example, the structure-preserving information can be represented by a density representation, a point set representation, etc. b. In one example, a point set representation (such as a uniform point set, a farthest sampling point set, etc.) can be used to obtain the backbone structure of the point cloud. i. In one example, the farthest sampling method can be used to obtain the farthest sampling point set. 4) A combination of different downsampling methods is proposed to obtain the final sampled point cloud. a. In one example, the final sampled point cloud can be obtained by combining local structure importance, global structure importance, and structure preservation information. i. In one example, local importance sampling and global importance sampling can be used to sample 10% of the points. ii. In one example, structure-preserving sampling may be used to sample 10% of the points. iii. In one example, the points sampled by local-global importance sampling and the points sampled by structure-preserving sampling may be merged to obtain a final sampled point cloud. 5) It is proposed to encode and decode the final sampling point cloud and transmit it to the decoder through a signal. a. In one example, the final sampled point cloud can be encoded and decoded by a point cloud codec. i. In one example, the point cloud codec can be G-PCC, V-PCC, Draco, etc. 6) It is proposed to encode and decode the features and transmit them to the decoder through signals. a. In one example, the feature can be encoded and decoded using fixed-length codec, unary codec, truncated unary codec, etc. b. In one example, features can be encoded and decoded in a predictive manner. 7) It is proposed to reprocess the initial features obtained by the decoder. a. In one example, feature reprocessing can simply use convolution to expand the feature dimension. b. In one example, feature reprocessing can be a complex change point expansion operation. c. In one example, feature reprocessing can be performed using a change-point feature expansion operation with a downsampling and upsampling symmetric structure. i. In one example, sparse convolution can be used to implement downsampling operations for feature reprocessing. 1. In one example, multiple consecutive downsampling operations may be used. For example, three consecutive downsampling operations may be used. ii. In one example, lossless sparse deconvolution can be used to implement the upsampling operation. 1. In one example, multiple consecutive upsampling operations may be used. For example, three consecutive upsampling operations may be used. 8) It is proposed to obtain the reconstructed point cloud based on the upsampling result of the decoded point cloud. a. In one example, the reconstructed point cloud can be directly obtained through a single upsampling. b. In one example, the reconstructed point cloud can be directly obtained through multiple progressive upsamplings. i. In one example, N upsampling operations can be used to reconstruct the point cloud, for example, N = 3. 1. In one example, N can be predefined. 2. In one example, N can be transmitted to the decoder through a signal. a. In one example, N can be encoded and decoded using fixed-length encoding and decoding, unary encoding and decoding, truncated unary encoding and decoding, etc. b. In one example, N can be encoded and decoded in a predictive manner. c. In one example, a generative convolution based on sparse convolution can be used to implement point cloud upsampling. d. In one example, a feature enhancement module can be used to enhance the features obtained after each upsampling. i. In one example, an Inception-Residual block can be used to enhance the features. e. In one example, a multi-stage unbalanced loss function can be used to constrain the neural network during the training process of upsampling. i. In one example, the binary cross-entropy value can be used as the loss function in the first stage. ii. In one example, the number of points used in the loss function can be different in different stages. 1. In one example, the top M% of the points with the importance of the real point cloud can be used to constrain the reconstructed point cloud in the first stage. 2. In one example, the top N% of the points with the importance of the real point cloud can be used to constrain the reconstructed point cloud in the first stage. 3. In one example, the top K% of the points with the importance of the real point cloud can be used to constrain the reconstructed point cloud in the last stage. 4. In one example, M < N < K, for example, M = 40, N = 70, K = 100. 9) It is proposed to learn the residual between the points obtained through upsampling and the real point cloud through a learning-based method. a. In one example, a neural network method can be used to learn the residual between the upsampled points and the real points. i. In one example, a neural network similar to the U-Net structure network can be used to learn the residual between the reconstructed points and the real point cloud. 1. In one example, a U-Net network can be used to learn the residual based on sparse convolution operations. b. In one example, supervised learning can be used to learn the residual between the reconstructed point cloud and the true point cloud. i. In one example, the error between the learned residuals of the reconstructed points and the true points can be used to update the neural network. ii. In one example, chamfer distance can be used as a loss function for residual learning. 1. In one example, the chamfer distance may be calculated as follows: Where S1 and S2 are sets of two point clouds. x and y are the coordinates of the points in S1 and S2 respectively. 10) Whether and / or how the above disclosed methods are applied may be signaled in the bitstream / frame / slice / slice / octree / etc. from the encoder to the decoder. 11) Whether and / or how to apply the above disclosed methods may depend on the coded information, such as dimension, color format, color component, slice / picture type. 6. Examples An example of the process of the geometric compression method based on visual perception is as follows. First, the point cloud is downsampled according to the importance of the points, so that points with high importance are sampled more and points with low importance are sampled less, thereby achieving adaptive downsampling. Second, the initial features of the decoder are reprocessed to improve the representation ability of the features and construct feature context for point cloud reconstruction. Third, progressive upsampling is used to reconstruct the point cloud to reduce the reconstruction difficulty, and an unbalanced loss is used at each stage to constrain the reconstruction result, where the loss is more biased towards areas with more complex structures. Therefore, the unbalanced loss constraint can make the reconstruction of more complex areas better. Finally, the geometric coordinate residual between the point cloud obtained by progressive upsampling and the real point cloud is learned, so that the point cloud coordinates obtained by upsampling plus the geometric coordinate residual are more similar to the real point cloud coordinates. Figure 5 An example of the encoding and decoding process of the visual perception-based geometric compression method is depicted in . Figure 6 Depicts an example of the pipeline of a geometry-preserving importance sampling method.

[0059] The following describes further details of embodiments of the present disclosure, which relate to point cloud geometry compression based on visual perception. The embodiments of the present disclosure should be considered as examples to explain general concepts and should not be interpreted in a narrow sense. In addition, these embodiments can be applied alone or combined in any way.

[0060] As used herein, the term "point cloud sequence" may refer to a sequence of one or more point clouds. The term "point cloud frame" or "frame" may refer to a point cloud in a point cloud sequence.

[0061] Figure 7 FIG. 7 is a flow chart of a method 700 for video processing according to an embodiment of the present disclosure. The method 700 may be implemented during conversion between a current frame of a point cloud sequence and a bitstream of the point cloud sequence.

[0062] At block 710 , for conversion between a target frame of a point cloud sequence and a bitstream of the point cloud sequence, downsampling is applied to the points in the target frame according to their importance, wherein the importance of a point is associated with the visual sensitivity of the human eye to the point.

[0063] At block 720, a final sampled point cloud is obtained by combining the plurality of downsampled points.A first set of downsampled points is downsampled based on importance, and a second set of downsampled points is downsampled based on structure preserving information.

[0064] At block 730, conversion is performed based on the final sampled point cloud. In some embodiments, the conversion may include encoding the target frame into a bitstream. Alternatively, the conversion may include decoding the target frame from the bitstream. This can improve the quality of compression compared to traditional solutions, thereby increasing the efficiency of point cloud encoding and decoding.

[0065] In some embodiments, points with higher visual sensitivity are more important than points with lower visual sensitivity. In some other embodiments, high-frequency points are more important than low-frequency points. For example, high-frequency points have higher visual sensitivity than low-frequency points. In some further embodiments, high-frequency points are sampled by arranging them in descending order of importance.

[0066] In some embodiments, whether and / or how to obtain a final sampled point cloud by combining multiple downsampled points is indicated from the encoder to the decoder in one of the following: a bitstream, a frame, a slice, a slice, or an octree. In some embodiments, whether and / or how to obtain a final sampled point cloud by combining multiple downsampled points depends on codec information, where the codec information includes at least one of the following: dimension, color format, color component, slice type, or picture type.

[0067] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by a video processing device. The method includes: applying downsampling to points in a target frame of a point cloud sequence based on their importance, wherein the importance of a point is associated with the visual sensitivity of the human eye to the point; obtaining a final sampled point cloud by combining multiple downsampled points, wherein a first group of downsampled points is downsampled based on importance and a second group of downsampled points is downsampled based on structure-preserving information; and generating a bitstream based on the final sampled point cloud.

[0068] According to some further embodiments of the present disclosure, a method for storing a bitstream of a video is provided. The method includes: applying downsampling to points in a target frame of a point cloud sequence based on their importance, wherein the importance of a point is associated with the visual sensitivity of a human eye to the point; obtaining a final sampled point cloud by combining a plurality of downsampled points, wherein a first group of downsampled points is downsampled based on importance and a second group of downsampled points is downsampled based on structure-preserving information; generating a bitstream based on the final sampled point cloud; and storing the bitstream in a non-transitory computer-readable recording medium.

[0069] Figure 8 FIG. 8 is a flow chart of a method 800 for video processing according to an embodiment of the present disclosure. The method 800 may be implemented during conversion between a current frame of a point cloud sequence and a bitstream of the point cloud sequence.

[0070] At block 810 , reprocessing is performed on a set of features associated with a target frame for conversion between a target frame of a point cloud sequence and a bitstream of the point cloud sequence.

[0071] At block 820, a reconstructed point cloud is obtained by applying upsampling to the target frame and applying feature enhancement to the upsampled target frame. In some embodiments, an initial residual block is used to enhance a set of features.

[0072] At block 830 , the reconstructed point cloud is updated by adding the residual between the true point cloud and the reconstructed point cloud.

[0073] At block 840, conversion is performed based on the updated reconstructed point cloud and the reprocessed set of features. In some embodiments, the conversion may include encoding the target frame into a bitstream. Alternatively, the conversion may include decoding the target frame from the bitstream. This can improve point cloud accuracy compared to traditional solutions, thereby increasing the efficiency of point cloud encoding and decoding.

[0074] In some embodiments, whether and / or how to obtain the final sampled point cloud by combining multiple downsampled points is indicated from the encoder to the decoder in one of the following: a bitstream, a frame, a slice, a slice, or an octree.

[0075] In some embodiments, whether and / or how to obtain a final sampled point cloud by combining multiple downsampled points depends on codec information, where the codec information includes at least one of the following: dimension, color format, color component, slice type, or picture type.

[0076] According to another embodiment of the present disclosure, a non-transitory computer-readable recording medium is provided. The non-transitory computer-readable recording medium stores a bitstream of a video generated by a method performed by an apparatus for video processing. The method includes: reprocessing a set of features associated with a target frame of a point cloud sequence; obtaining a reconstructed point cloud by upsampling the target frame and applying feature enhancement to the upsampled target frame; updating the reconstructed point cloud by adding a residual between a real point cloud and the reconstructed point cloud; and generating a bitstream based on the updated reconstructed point cloud and the reprocessed set of features.

[0077] According to further embodiments of the present disclosure, a method for storing a bitstream of a video is provided. The method includes: reprocessing a set of features associated with a target frame of a point cloud sequence; obtaining a reconstructed point cloud by upsampling the target frame and applying feature enhancement to the upsampled target frame; updating the reconstructed point cloud by adding a residual between the real point cloud and the reconstructed point cloud; generating a bitstream based on the updated reconstructed point cloud and the reprocessed set of features; and storing the bitstream in a non-transitory computer-readable recording medium.

[0078] The embodiments of the present disclosure may be described according to the following items, features of which may be combined in any reasonable way.

[0079] Item 1. A method for video processing, comprising: for conversion between a target frame of a point cloud sequence and a bitstream of the point cloud sequence, applying downsampling to the points in the target frame according to the importance of the points, wherein the importance of the points is associated with the visual sensitivity of the human eye to the points; obtaining a final sampled point cloud by combining a plurality of downsampled points, wherein a first group of downsampled points are downsampled based on importance and a second group of downsampled points are downsampled based on structure-preserving information; and performing the conversion based on the final sampled point cloud.

[0080] Item 2. The method according to Item 1, wherein the importance of a point with higher visual sensitivity is higher than the importance of a point with lower visual sensitivity.

[0081] Item 3. The method according to Item 1, wherein the importance of high-frequency points is higher than the importance of low-frequency points.

[0082] Item 4. The method of Item 3, wherein the high-frequency point has a higher visual sensitivity than the low-frequency point.

[0083] Item 5. The method according to Item 1, wherein high frequency points are sampled by arranging the importance in descending order.

[0084] Item 6. A method according to any one of items 1-5, wherein whether and / or how the final sampled point cloud is obtained by combining the multiple downsampled points is indicated from the encoder to the decoder in one of the following: a bitstream, a frame, a slice, a slice or an octree.

[0085] Item 7. A method according to any one of Items 1-5, wherein whether and / or how the final sampled point cloud is obtained by combining the multiple downsampled points depends on codec information, wherein the codec information includes at least one of the following: dimension, color format, color component, slice type or picture type.

[0086] Item 8. A method for video processing, comprising: for conversion between a target frame of a point cloud sequence and a bitstream of the point cloud sequence, reprocessing a set of features associated with the target frame; obtaining a reconstructed point cloud by applying upsampling to the target frame and applying feature enhancement to the upsampled target frame; updating the reconstructed point cloud by adding a residual between a real point cloud and the reconstructed point cloud; and performing the conversion based on the updated reconstructed point cloud and the reprocessed set of features.

[0087] Item 9. The method of Item 8, wherein an initial residual block is used to enhance the set of features.

[0088] Item 10. A method according to any of Items 8-9, wherein whether and / or how the final sampled point cloud is obtained by combining the multiple downsampled points is indicated from the encoder to the decoder in one of the following: a bitstream, a frame, a slice, a slice or an octree.

[0089] Item 11. A method according to any one of Items 8-9, wherein whether and / or how the final sampled point cloud is obtained by combining the multiple downsampled points depends on codec information, wherein the codec information includes at least one of the following: dimension, color format, color component, slice type or picture type.

[0090] Item 12. A method according to any of Items 1-11, wherein the converting comprises encoding the target frame into the bitstream.

[0091] Item 13. A method according to any of Items 1-11, wherein the converting comprises decoding the target frame from the bitstream.

[0092] Item 14. An apparatus for video processing, comprising a processor and non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of items 1-7.

[0093] Item 15. A non-transitory computer-readable storage medium storing instructions for causing a processor to perform the method according to any one of Items 1-7.

[0094] Item 16. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by an apparatus for video processing, wherein the method comprises: applying downsampling to points in a target frame of a point cloud sequence according to their importance, wherein the importance of the points is associated with the visual sensitivity of a human eye to the points; obtaining a final sampled point cloud by combining a plurality of downsampled points, wherein a first group of downsampled points are downsampled based on importance and a second group of downsampled points are downsampled based on structure-preserving information; and generating the bitstream based on the final sampled point cloud.

[0095] Item 17. A method for storing a bitstream of a video, comprising: applying downsampling to points in a target frame of a point cloud sequence based on their importance, wherein the importance of the points is associated with the visual sensitivity of a human eye to the points; obtaining a final sampled point cloud by combining a plurality of downsampled points, wherein a first group of downsampled points are downsampled based on importance and a second group of downsampled points are downsampled based on structure-preserving information; generating the bitstream based on the final sampled point cloud; and storing the bitstream in a non-transitory computer-readable recording medium.

[0096] Item 18. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by an apparatus for video processing, wherein the method comprises: performing reprocessing on a set of features associated with a target frame of a point cloud sequence; obtaining a reconstructed point cloud by applying upsampling to the target frame and applying feature enhancement to the upsampled target frame; updating the reconstructed point cloud by adding a residual between a real point cloud and the reconstructed point cloud; and generating the bitstream based on the updated reconstructed point cloud and the reprocessed set of features.

[0097] Item 19. A method for storing a bitstream of a video, comprising: performing reprocessing on a set of features associated with a target frame of a point cloud sequence; obtaining a reconstructed point cloud by applying upsampling to the target frame and applying feature enhancement to the upsampled target frame; updating the reconstructed point cloud by adding a residual between a real point cloud and the reconstructed point cloud; generating the bitstream based on the updated reconstructed point cloud and the reprocessed set of features; and storing the bitstream in a non-transitory computer-readable recording medium. Example device

[0098] Figure 9A block diagram of a computing device 900 in which various embodiments of the present disclosure may be implemented is shown. The computing device 900 may be implemented as a source device 110 (or a GPCC encoder 116 or 200) or a destination device 120 (or a GPCC decoder 126 or 300), or may be included in a source device 110 (or a GPCC encoder 116 or 200) or a destination device 120 (or a GPCC decoder 126 or 300).

[0099] It should be understood that Figure 9 The computing device 900 shown in FIG. 9 is for illustrative purposes only and is not intended to in any way imply any limitation on the functionality and scope of the disclosed embodiments.

[0100] like Figure 9 As shown, computing device 900 comprises a general computing device 900. Computing device 900 may include at least one or more processors or processing units 910, memory 920, storage unit 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960.

[0101] In some embodiments, the computing device 900 can be implemented as any user terminal or server terminal with computing capabilities. The server terminal can be a server, a large computing device, etc. provided by a service provider. The user terminal can be, for example, any type of mobile terminal, fixed terminal, or portable terminal, including a mobile phone, a station, a unit, a device, a multimedia computer, a multimedia tablet computer, an Internet node, a communicator, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device, or any combination thereof, and includes accessories and peripherals of these devices, or any combination thereof. It is conceivable that the computing device 900 can support any type of interface to the user (such as a "wearable" circuit device, etc.).

[0102] The processing unit 910 may be a physical processor or a virtual processor and may implement various processes based on a program stored in the memory 920. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capabilities of the computing device 900. The processing unit 910 may also be referred to as a central processing unit (CPU), a microprocessor, a controller, or a microcontroller.

[0103] The computing device 900 typically includes various computer storage media. Such media can be any media accessible by the computing device 900, including but not limited to volatile media and non-volatile media, or removable media and non-removable media. The memory 920 can be a volatile memory (e.g., registers, cache, random access memory (RAM)), a non-volatile memory (such as read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM) or flash memory) or any combination thereof. The storage unit 930 can be any removable or non-removable medium and can include machine-readable media, such as memory, a flash drive, a disk or other media that can be used to store information and / or data and can be accessed in the computing device 900.

[0104] The computing device 900 may also include additional removable / non-removable storage media, volatile / non-volatile storage media. Figure 9 Although not shown, a magnetic disk drive for reading from and / or writing to a removable nonvolatile magnetic disk, and an optical disk drive for reading from and / or writing to a removable nonvolatile optical disk may be provided. In this case, each drive may be connected to a bus (not shown) via one or more data medium interfaces.

[0105] The communication unit 940 communicates with another computing device via a communication medium. In addition, the functions of the components in the computing device 900 can be implemented by a single computing cluster or multiple computing machines communicating via a communication connection. Thus, the computing device 900 can operate in a networked environment using logical connections to one or more other servers, networked personal computers (PCs), or other general-purpose network nodes.

[0106] Input device 950 may be one or more of various input devices, such as a mouse, keyboard, trackball, voice input device, etc. Output device 960 may be one or more of various output devices, such as a display, speaker, printer, etc. With the aid of communication unit 940, computing device 900 may also communicate with one or more external devices (not shown), such as storage devices and display devices. Computing device 900 may also communicate with one or more devices that enable a user to interact with computing device 900, or any device that enables computing device 900 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.), if necessary. Such communication may be performed via an input / output (I / O) interface (not shown).

[0107] In some embodiments, some or all components of the computing device 900 may not be integrated into a single device, but may also be arranged in a cloud computing architecture. In a cloud computing architecture, components may be provided remotely and may work together to implement the functionality described in this disclosure. In some embodiments, cloud computing provides computing, software, data access, and storage services without requiring the end user to know the physical location or configuration of the systems or hardware providing these services. In various embodiments, cloud computing provides services via a wide area network (such as the Internet) using appropriate protocols. For example, a cloud computing provider provides an application over a wide area network that can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data may be stored on servers at a remote location. Computing resources in a cloud computing environment may be consolidated or distributed across locations in remote data centers. Cloud computing infrastructure can provide services through shared data centers, although they appear to users as a single access point. Therefore, cloud computing architecture can be used to provide the components and functionality described herein from a service provider at a remote location. Alternatively, they can be provided from a conventional server or installed directly or otherwise on a client device.

[0108] The computing device 900 may be used to implement video encoding / decoding in the embodiments of the present disclosure. The memory 920 may include one or more video encoding / decoding modules 925 having one or more program instructions. These modules can be accessed and executed by the processing unit 910 to perform the functions of the various embodiments described herein.

[0109] In an example embodiment performing video encoding, an input device 950 may receive video data as input to be encoded 970. The video data may be processed by, for example, a video codec module 925 to generate an encoded bitstream. The encoded bitstream may be provided as output 980 via an output device 960.

[0110] In an example embodiment performing video decoding, an input device 950 may receive an encoded bitstream as input 970. The encoded bitstream may be processed by, for example, a video codec module 925 to generate decoded video data. The decoded video data may be provided as output 980 via an output device 960.

[0111] Although the present disclosure has been specifically shown and described with reference to the preferred embodiments of the present disclosure, it will be understood by those skilled in the art that various changes in form and details may be made without departing from the spirit and scope of the present application as defined by the appended claims. Such changes are intended to be encompassed by the scope of the present application. Therefore, the foregoing description of the embodiments of the present application is not intended to be limiting.

Claims

1. A video processing method, comprising: For conversion between a target frame of a point cloud sequence and a bitstream of the point cloud sequence, downsampling the points in the target frame according to their importance, wherein the importance of the points is associated with the visual sensitivity of the human eye to the points; obtaining a final sampled point cloud by combining a plurality of downsampled points, wherein a first set of downsampled points are downsampled based on importance and a second set of downsampled points are downsampled based on structure preserving information; as well as The conversion is performed based on the final sampled point cloud. 2 . The method according to claim 1 , wherein the importance of a point having higher visual sensitivity is higher than the importance of a point having lower visual sensitivity. The method according to claim 1 , wherein the importance of high frequency points is higher than that of low frequency points. The method according to claim 3 , wherein the high-frequency point has a higher visual sensitivity than the low-frequency point. The method according to claim 1 , wherein high frequency points are sampled by arranging the importance in descending order.

6. The method according to any one of claims 1 to 5, wherein whether and / or how the final sampled point cloud is obtained by combining the plurality of downsampled points is indicated from the encoder to the decoder in one of the following: BitTorrent, frame, piece, slices, or Octree.

7. The method according to any one of claims 1 to 5, wherein whether and / or how to obtain the final sampled point cloud by combining the multiple downsampled points depends on codec information, wherein the codec information includes at least one of the following: Dimension, Color format, Color component, Slice type, or Image type.

8. A video processing method, comprising: For conversion between a target frame of a point cloud sequence and a bitstream of the point cloud sequence, reprocessing a set of features associated with the target frame; Obtaining a reconstructed point cloud by applying upsampling to the target frame and applying feature enhancement to the upsampled target frame; updating the reconstructed point cloud by adding a residual between the true point cloud and the reconstructed point cloud; as well as The converting is performed based on the updated reconstructed point cloud and the reprocessed set of features. The method of claim 8 , wherein an initial residual block is used to enhance the set of features.

10. The method according to any one of claims 8 to 9, wherein whether and / or how the final sampled point cloud is obtained by combining the plurality of downsampled points is indicated from the encoder to the decoder in one of the following: BitTorrent, frame, piece, slices, or Octree.

11. The method according to any one of claims 8 to 9, wherein whether and / or how to obtain the final sampled point cloud by combining the plurality of downsampled points depends on codec information, wherein the codec information comprises at least one of the following: Dimension, Color format, Color component, Slice type, or Image type.

12. The method according to any one of claims 1 to 11, wherein the converting comprises encoding the target frame into the bitstream.

13. The method according to any one of claims 1 to 11, wherein the converting comprises decoding the target frame from the bitstream.

14. An apparatus for video processing, comprising a processor and a non-transitory memory having instructions thereon, wherein the instructions, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 13. 15 . A non-transitory computer-readable storage medium storing instructions for causing a processor to execute the method according to claim 1 .

16. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by an apparatus for video processing, wherein the method comprises: applying downsampling to the points in a target frame of the point cloud sequence according to their importance, wherein the importance of the points is associated with the visual sensitivity of the human eye to the points; obtaining a final sampled point cloud by combining a plurality of downsampled points, wherein a first set of downsampled points are downsampled based on importance and a second set of downsampled points are downsampled based on structure preserving information; as well as The bitstream is generated based on the final sampling point cloud.

17. A method for storing a bitstream of a video, comprising: applying downsampling to the points in a target frame of the point cloud sequence according to their importance, wherein the importance of the points is associated with the visual sensitivity of the human eye to the points; obtaining a final sampled point cloud by combining a plurality of downsampled points, wherein a first set of downsampled points are downsampled based on importance and a second set of downsampled points are downsampled based on structure preserving information; generating the bitstream based on the final sampling point cloud; as well as The bitstream is stored in a non-transitory computer-readable recording medium.

18. A non-transitory computer-readable recording medium storing a bitstream of a video generated by a method performed by an apparatus for video processing, wherein the method comprises: performing reprocessing on a set of features associated with a target frame of the point cloud sequence; Obtaining a reconstructed point cloud by applying upsampling to the target frame and applying feature enhancement to the upsampled target frame; updating the reconstructed point cloud by adding a residual between the true point cloud and the reconstructed point cloud; as well as The bitstream is generated based on the updated reconstructed point cloud and the reprocessed set of features.

19. A method for storing a bitstream of a video, comprising: performing reprocessing on a set of features associated with a target frame of the point cloud sequence; Obtaining a reconstructed point cloud by applying upsampling to the target frame and applying feature enhancement to the upsampled target frame; updating the reconstructed point cloud by adding a residual between the true point cloud and the reconstructed point cloud; generating the bitstream based on the updated reconstructed point cloud and the reprocessed set of features; as well as The bitstream is stored in a non-transitory computer-readable recording medium.

Images