Task-aware point cloud downsampling

JP2023549320A5Active Publication Date: 2024-11-12INTERDIGITAL PATENT HOLDINGS INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023525557
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-11-12
Filing Date
2021-11-12
Publication Date
2024-11-12
Estimated Expiration
2041-11-12

AI Technical Summary

Technical Problem

Processing large point clouds is computationally expensive for consumer devices due to their high data volume, which is a challenge for applications like autonomous driving, VR/AR, and topography, as they require significant computational resources.

Method used

A method for generating point-level and set-level feature vectors using neural networks to downsample point clouds, integrating the downsampling process with subsequent tasks through end-to-end training, ensuring the downsampled point clouds are task-aware and adaptive to specific machine tasks.

Benefits of technology

The method reduces computational burden by generating representative points and features, allowing efficient processing of point clouds for various tasks such as scene flow estimation and compression, while maintaining accuracy and adaptability to different applications.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The method includes using a neural network to generate a point-level feature vector for each point of the point cloud and a set-level feature vector for the point cloud. A representative location is generated based on the point-level feature vectors and the set-level feature vector. The representative location and the set-level feature vector are output as a set descriptor.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical Field]

[0001] The present principles generally relate to the field of point cloud processing, and may also be understood in the context of analysis, interpolation, representation and understanding of point cloud signals. [Background technology]

[0002] This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present principles, which are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present principles. Accordingly, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0003] Point clouds are a data format used across several business fields, including autonomous driving, robotics, AR / VR, civil engineering, computer graphics, and the animation / film industry. 3D LIDAR sensors are being deployed in self-driving cars, and affordable LIDAR sensors are included, for example, in the Apple iPad Pro 2020 and the Intel RealSense LIDAR Camera L515. With advances in sensing technology, three-dimensional (3D) point cloud data is becoming more practical and is expected to be a valuable enabler in the mentioned applications.

[0004] At the same time, point cloud data may consume a large portion of network traffic, for example between connected cars on 5G networks, and in immersive communications (virtual reality / augmented reality (VR / AR)). Understanding and communicating point clouds inherently leads to efficient representation formats. In particular, raw point cloud data needs to be properly organized and processed for world modeling and sensing purposes.

[0005] Furthermore, point clouds may represent a continuous representation of the same scene containing multiple moving objects, which are called dynamic point clouds, as compared to static point clouds captured from a static scene or static objects. Dynamic point clouds may be organized into frames where different frames are captured at different times.

[0006] 3D point cloud data is essentially a collection of discrete samples on the surface of an object or scene. In practice, a large number of points are required to fully represent the real world with point samples. For example, a typical VR immersive scene contains millions of points, while a point cloud map typically contains hundreds of millions of points. Therefore, processing such large-scale point clouds is computationally expensive, especially for consumer devices with limited computing power, such as smartphones, tablets, and automobile navigation systems.

[0007] Point cloud data is important for various applications such as autonomous driving, VR / AR, topography and cartography, but consuming large point clouds directly incurs significant computational costs. Therefore, it is important to adaptively downsample the input point cloud to facilitate subsequent tasks. Such a downsampling process is useful for scene flow estimation, point cloud compression, and other general computer vision tasks. Summary of the Invention

[0008] The following presents a simplified summary of the present principles to provide a basic understanding of some aspects of the present principles. This summary is not an extensive overview of the present principles. It is not intended to identify key or critical elements of the present principles. The following summary merely presents some aspects of the present principles in a simplified form as a prelude to the more detailed description provided below.

[0009] The present principles relate to a method for generating a point-level feature vector for each point of a point cloud and a set-level feature vector for the point cloud using a neural network. A representative position is generated based on the point-level feature vectors and the set-level feature vectors. The representative position and the set-level feature vector are output as a set descriptor.

[0010] In another embodiment, a method for deriving a point cloud from a data stream includes obtaining a downsampled point cloud and a residual point cloud from the data stream. The downsampled point cloud is provided to a predictor construction module to obtain a predicted point cloud. The point cloud is derived by adding the predicted point cloud to the residual point cloud.

[0011] The present principles also relate to a device comprising at least one processor associated with at least one memory configured to implement an embodiment corresponding to the above method. [Brief explanation of the drawings]

[0012] The present disclosure will be better understood, and other particular features and advantages will become apparent, on reading the following description, which makes reference to the accompanying drawings, in which: [Figure 1] 1 illustrates a method 10 for downsampling an input point cloud X by n points for a subsequent machine task, in accordance with a non-limiting embodiment of the present principles. [Figure 2] 1 illustrates a schematic diagram of an SD function, in accordance with a non-limiting embodiment of the present principles; [Figure 3] An example is shown where point A is selected as the representative point because it has the largest weight. [Figure 4] 10 illustrates a fifth embodiment for downsampling an input point cloud according to present principles. [Figure 5] 1 shows a schematic diagram of how the task-aware point cloud downsampling method of the present principles is integrated with a subsequent machine task. [Figure 6] 10 illustrates a seventh embodiment of a unified task-aware point cloud downsampling method. [Figure 7] An embodiment of the task-aware point cloud downsampling method according to the present principles is used to demonstrate a method for point cloud compression. [Figure 8] 1 shows an embodiment of a decoder according to the present principles; [Figure 9] 2 illustrates an example architecture of a device 30 that may be configured to implement the method described in connection with FIG. DETAILED DESCRIPTION OF THE INVENTION

[0013] The present principles are described more fully below with reference to the accompanying drawings, in which examples of the present principles are shown. However, the present principles may be embodied in many alternative forms and should not be construed as limited to the embodiments set forth herein. Accordingly, while the present principles are susceptible to various modifications and alternative forms, specific examples thereof are shown by way of example in the drawings and are described in detail herein. It is to be understood, however, that there is no intention to limit the present principles to the particular forms disclosed, but on the contrary, the present disclosure is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present principles as defined by the appended claims.

[0014] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the present principles. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It will be further understood that as used herein, the terms "comprises," "comprising," "includes," and / or "including" specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. Furthermore, when an element is referred to as "responsive to" or "connected to" another element, it may be directly responsive to or connected to the other element, or intervening elements may be present. In contrast, when an element is referred to as "directly responsive to" or "directly connected to" another element, there are no intervening elements present. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0015] In this specification, terms such as "first," "second," etc. may be used to describe various elements, but it should be understood that these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element without departing from the teachings of the present principles.

[0016] Some of the figures include arrows on communication paths to indicate the primary direction of communication, however, it should be understood that communication may occur in the opposite direction to the depicted arrow.

[0017] Some examples are described with reference to block diagrams and operational flowcharts, in which each block represents circuit elements, modules, or portions of code, with each block including one or more executable instructions for implementing a specified logical function. It should also be noted that in other implementations, the functions noted in the blocks may occur out of the order noted. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may be executed in the reverse order, depending on the functionality involved.

[0018] As used herein, "by one example" or "in one example" means that a particular feature, structure, or characteristic described in connection with this embodiment may be included in at least one implementation of the present principles. The appearances of the phrase "by one example" or "in one example" in various places in this specification do not necessarily all refer to the same embodiment, and in separate or alternative embodiments are not necessarily mutually exclusive of other embodiments.

[0019] Reference numerals appearing in the claims are by way of example only and shall have no limiting effect on the scope of the claims. Although not expressly stated, the present embodiments and variations may be used in any combination or subcombination.

[0020] The automotive industry and autonomous vehicles are areas where point clouds can be used. Autonomous vehicles should be able to "explore" their environment and make good driving decisions based on the reality of their immediate surroundings. Typical sensors such as LIDAR produce (dynamic) point clouds that are used by decision engines. These point clouds are not intended to be seen by the human eye; they are usually sparse, not necessarily color-coded, and dynamic with a high capture frequency. These point clouds may have other attributes, such as reflectivity provided by LIDAR, because this attribute can indicate the material of the sensed object and help make that decision.

[0021] Virtual reality and immersive worlds have been widely discussed recently and are predicted by many to be the future of 2D flat video. The basic idea is to immerse the viewer in the environment around them, as opposed to standard TV, where the viewer simply sees the virtual world in front of them. There are several levels of immersion, depending on the viewer's degrees of freedom within the environment. Point clouds are a good format candidate for delivering virtual reality worlds. They can be static or dynamic and are typically no larger than an average size, e.g., a few million points at a time.

[0022] Point clouds can also be used for various purposes, such as cultural heritage / architecture, where objects such as statues or buildings are scanned in 3D to share the spatial configuration of the object without sending or physically visiting the object. This also provides a way to store information and data about the object, for example, in cases where a temple may be destroyed by an earthquake. Such point clouds are usually static, color-coded, and large in volume.

[0023] Another use case is in topography and cartography, where 3D representations and maps are not limited to flat surfaces but can include relief. Google Maps is an example of a 3D map that uses meshes instead of point clouds. Nevertheless, point clouds can be a preferred data format for 3D maps, and such point clouds are usually static, color-coded, and relatively large in volume.

[0024] World modeling and sensing via point clouds is a technique that enables machines to gain knowledge about the 3D world around them, and is useful for the applications discussed above.

[0025] 3D point cloud data is essentially discrete samples on the surface of an object or scene. In practice, a large number of points are required to fully represent the real world with point samples. Therefore, processing such large-scale point clouds is computationally expensive, especially for consumer devices with limited computing power, such as smartphones, tablets, and automobile navigation systems.

[0026] To process the input point cloud at a reasonable computational cost, one solution is to first downsample it, and the downsampled point cloud summarizes the geometry of the input point cloud while having significantly fewer points. The downsampled point cloud is then fed to subsequent machine tasks for further consumption. However, point cloud data can be utilized for various tasks such as scene flow estimation, classification, detection, segmentation, and compression. Different tasks focus on different aspects of the point cloud. For example, classification depends on geometric salient points, object segmentation needs to distinguish points on one object from others, and scene flow estimation counts the dynamics of the point cloud. Therefore, a task-aware adaptive point cloud downsampling algorithm is useful. Thus, when facing different tasks, the same point cloud can be downsampled differently to facilitate subsequent tasks.

[0027] FIG. 1 shows a method 10 for downsampling an input point cloud X at n points for subsequent instrument tasks according to the present principle. In step 11, a first downsampled point cloud having m points (m < n) is selected. Using any applicable method, a set of m points such as point 110 of the input point cloud is selected. In step 12, for points 110 (referred to herein as "anchor points") within the first downsampled point cloud, the neighboring points are aggregated from the point cloud X to obtain a local point set 120. In this way, each anchor point within the first downsampled point cloud is associated with a local point set from the point cloud X. In step 13, each point set is fed to a module called the Set Distillation (SD) function herein, resulting in a representative point 130 and its corresponding set-level feature.

[0028] According to the present principles, given a point set (and other auxiliary information, if available), the SD function first calculates a point-level feature vector for each point in the point set and a set-level feature vector that describes the entire point set. This step is accomplished, for example, using a structured neural network module (referred to herein as P-Net) according to the present principles. A representative position is calculated by taking the point-level feature vector and the set-level feature vector for each point as input. This step is accomplished either through a deterministic approach or through a separate neural network module. The SD function then outputs the representative position along with the set-level features to represent the geometry of the point set. By using the SD function, the resulting representative position is not limited to the points in the point set.

[0029] In step 14, the m keypoints are aggregated as an updated downsampled point cloud, which is fed to subsequent tasks for further processing. The m set-level features are also optionally output and fed to subsequent tasks.

[0030] Downsampling method 10 is integrated with subsequent tasks and trained in an end-to-end manner, allowing downsampling method 10 to be task-aware, i.e., adaptive to the machine task. Meanwhile, through end-to-end training, the downsampled point cloud obtained by method 10 is capable of capturing the underlying geometry for a specific machine task, regardless of how the original input point cloud was sampled from the scene. Specifically, given two different point clouds sampling the same surface, i.e., one point cloud is a resampled version of the other, for the same subsequent machine task, method 10 will yield two downsampled point clouds that are very similar to each other.

[0031] Figure 2 shows a schematic diagram of an example of an SD function. Given a point set 20, the SD function feeds the point set into the PointNet architecture, as described, for example, in C.R. Qi, H. Su, K. Mo, and L.J. Guibas, "PointNet: Deep learning on point sets for 3D classification and segmentation," in proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 652-660, 2017. Module 21 of PointNet computes a point-level feature vector 22 for each point using shared multi-layer perception (MLP). These point-level feature vectors 22 are then aggregated with a max-pooling operation 23 to yield a set-level feature vector 24 that describes the entire point set.

[0032] According to the present principles, a set of weights 26 is calculated for the points in the entire point set. To do so, an affinity value (e.g., weight estimate) between each point-level feature vector 22 and the set-level feature vector 24 is provided by module 25, which calculates the dot product between them. This affinity value describes the degree to which its associated point represents the entire point set. Module 27 performs a weighted average 28 of the points using the calculated weights 26 to generate a representative position for the point set. The affinity values ​​are converted into a set of weights using a Softmax(·) function, so that all weight values ​​are greater than 0 and sum to 1. A weighted average of the x-coordinates of all points in the point set is performed using the obtained weights to obtain the x-coordinate of the generated representative point. Similarly, the y- and z-coordinates of the representative point are calculated using the weights. The generated x-, y-, and z-coordinates form the location of the representative point. The SD function outputs the representative point and the set-level features generated by PointNet.

[0033] In a first embodiment of the present principles, downsampling of a given point cloud X containing n points is performed using the presented SD function. In the first step, an initial downsampled point cloud with m points is generated using the farthest point sampling (FPS) method, and the resulting points are called "anchor points." Farthest point sampling is a known point cloud downsampling technique, and is described, for example, in "The Farthest Point Strategy for Progressive Image Sampling," IEEE Trans. on Image Processing, vol. 6, no. 9, pp. 1306-1315, 1997. FPS is based on iteratively selecting the next sample point within a minimum search area. Given a point cloud X and its sampled subset, the FPS algorithm selects the farthest point to the subset from the remaining points in X using some distance measure. This farthest point is then added to the subset. Here, the subset is initialized by randomly picking a point from X. The FPS algorithm repeats this point selection process until a certain condition is met, e.g., the number of points in the subset reaches a predefined threshold. This classical sampling method is deterministic and does not consider downstream tasks.

[0034] In the second step of the first embodiment, for each anchor point, its neighbors are collected through a ball query procedure. That is, all points in X that are within a predefined distance r to the anchor point are identified and collected to form a local point set for that anchor point. In the third step, all local point sets (m in total) are individually fed into the SD function to obtain an updated downsampled point cloud (having m points) along with m set-level features. In the fourth step, the m downsampled points (and optionally the set-level features) are fed into a subsequent task. This downsampling method is trained end-to-end with the subsequent machine task to make the neural network layer in the SD function task-aware, i.e., adapted to the subsequent task.

[0035] In the second embodiment, the calculation of the point-by-point weights of the SD function is different. Specifically, in the SD function of this embodiment, a distance is calculated for each point in the point set, which is the Euclidean distance between its point-level feature vector and the set-level feature vector. In this specification, this distance value is defined as d for i. i Denoted by d i The values ​​are plugged into a Gaussian kernel to calculate the weights, i.e., with a constant σ

[0036]

number

[0037] In the third embodiment, each downsampled point is obtained by selecting a critical point within a local point set. The difference between this embodiment and the first embodiment is the SD function, where the SD function selects a representative point from the input point set. Similar to the first embodiment, given a point set, the SD function calculates a set of weights for each point in the entire set. The SD function then directly returns the point with the largest weight as the representative point, as well as the set-level features generated by PointNet. Figure 3 shows an example in which point A has the largest weight and is therefore selected as the representative point. In a variation of the third embodiment, the method in the second embodiment for calculating the weights of points within a point set may also be used, where the weights are obtained through a Gaussian kernel.

[0038] In a fourth embodiment, the SD function takes as input not only a local point set from point cloud X but also a one-hot vector indicating which points are anchor points for the point set. As a result, the SD function in this embodiment can utilize knowledge of the anchor point locations to generate representative points. In particular, the SD function augments the position vector of each point in the point set by appending the anchor point's position vector (and, if available, the anchor's feature vector, which is another input to the SD function) prior to PointNet computation. With the anchor location information appended, the augmented point set is processed by PointNet to obtain point-level and set-level feature vectors for each point.

[0039] FIG. 4 illustrates a fifth embodiment of downsampling an input point cloud according to the present principles. Instead of generating a representative point by weighted averaging, this embodiment directly modifies the position of the anchor point 41 and then returns the modified position 42 as the representative point. Specifically, similar to the fourth embodiment, the SD function in this embodiment also takes as input a local point set 20 and a one-hot vector indicating which point is the anchor point 41 of the point set. Once the point-level feature vector 22 and the set-level feature 24 are obtained, they are fed into another neural network 43, referred to herein as "M-Net." The M-Net specifically outputs a modification vector 44 for the anchor point position. This may be implemented using the PointNet architecture. The representative point position 42 is obtained by adding the modification vector 44 and the anchor position 41. Finally, the SD function still returns the representative point and the set-level feature vector. The fifth embodiment can be combined with the fourth embodiment, in which the points fed into the SD function are first augmented with information from the anchor positions.

[0040] FIG. 5 shows a schematic diagram of how the task-aware point cloud downsampling method of the present principles can be integrated with a subsequent machine task. As an example, to illustrate this sixth embodiment, without loss of generality, the task of scene flow estimation for a 3D point cloud is considered. This task takes as input two consecutive 3D point cloud frames in a point cloud sequence, e.g., a first point cloud frame 51 and a second point cloud frame 52, and aims to estimate the scene flow from the first point cloud frame to the second point cloud frame, i.e., the movement of each 3D point from the first point cloud frame to the second point cloud frame. The difficulty is that point indices are lost from one frame to the next. In this scenario, the output scene flow 53 includes a set of 3D vectors, each associated with a point in the first point cloud frame. The 3D vectors represent how points from the first point cloud frame physically move onto the surface of the second point cloud frame. In other words, the scene flow between two point cloud frames represents the dynamics of point clouds, which is essential for many practical applications, e.g., autonomous driving, AR / VR, and robotics.

[0041] In this sixth embodiment, the downsampling method presented in the previous embodiment is applied multiple times. The overall neural network architecture of this embodiment has an hourglass structure with skip connections. The method of this sixth embodiment includes a first stage of generating first and second downsampled point clouds from first and second point cloud frames, respectively. This is achieved using two task-aware downsampling modules 54a and 54b (based on any one of the previous embodiments) for both inputs. Two consecutive point clouds in a point cloud sequence may be considered as one point cloud, carrying temporal information indicating whether a point belongs to the first point cloud or the second point cloud. In fact, the two point clouds in the point cloud sequence share the same reference frame, and their points can be merged into one point cloud. In a second stage, the point set of each point in the first downsampled point cloud is aggregated by searching for its nearest neighbors from the second downsampled point cloud. For each point in the first downsampled point cloud, the method uses the point's information (its location and point-level features) and its associated set of nearest neighbors to compute a first inter-frame feature that fuses information from both point cloud frames. This second stage is achieved using a neural network module 55, referred to herein as "F1-Net." In a third stage, the first downsampled point cloud is further downsampled using a task-aware downsampling module 54c according to the present principles, taking the point and associated inter-frame features as input. In a fourth stage, a second inter-frame feature is computed for each point in the first point cloud frame using an upsampled neural network module 56, referred to herein as "F2-Net." This F2-Net module corresponds to a stack of setup transformation layers, for example, as presented in "FlowNet3D: Learning scene flow in 3D point clouds," in proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 529-537, 2020.Such neural network modules hierarchically interpolate point-wise features. In the fifth stage, a scene flow vector is calculated for each point in the first point cloud frame using a feature-to-flow transform neural network module 57, referred to herein as "F3-Net." The F3-Net is implemented using point-wise MLP layers. According to the present principles, skip connections between the task-aware downsampling module and the F2-Net are added to merge information from earlier layers.

[0042] The entire neural network architecture of Figure 5, including the task-aware downsampling modules 54a-c and other neural network modules, is trained end-to-end using an end-point-error (EPE) loss function as described in "FlowNet3D." After the training process, the proposed task-aware downsampling module becomes well integrated with the other neural network modules, which facilitates the estimation of accurate scene flow vectors.

[0043] The integration of task-aware point cloud downsampling methods is illustrated in the context of a scene flow estimation task in Figure 5. The same principles may be applied to any other task dealing with point clouds such as those mentioned above without loss of generality.

[0044] FIG. 6 illustrates a seventh embodiment of the integrated task-aware point cloud downsampling method. In this seventh embodiment, a scene flow 53 is estimated based on two input point cloud frames 51 and 52. The method iteratively updates the estimated scene flow to fine-tune its accuracy through a flow interpolation module 61. An initial point-by-point scene flow for the first point cloud frame 51 is estimated using the method described above with reference to FIG. 5. Based on this initial scene flow, a point-by-point scene flow is generated for the first downsampled point cloud. This is achieved based on a scene flow interpolation neural network module 61, referred to herein as "I-Net." I-Net is implemented in the same manner as the setup transformation layer, as described in "FlowNet3D." The shifted downsampled point cloud is generated by shifting each point in the first downsampled point cloud by its associated scene flow vector. The point set is then aggregated by searching for its nearest neighbor from the second downsampled point cloud for each point in the shifted downsampled point cloud. In this way, each point in the first downsampled point cloud is associated with its shifted version as well as an updated set of nearest neighbors that use the shifted version as query points, which are more accurate / useful for scene flow estimation.

[0045] Using the first downsampled point cloud and all updated nearest neighbor point sets, the F1-Net 55, the second task-aware downsampling module 54c, the F2-Net 56, and the F3-Net 57 are re-run to obtain a second pointwise scene flow for the first point cloud frame. An alternative to this step is to re-run the F1-Net, the second task-aware downsampling module, the F2-Net, and the F3-Net based on the shifted downsampled point cloud and the updated nearest neighbor point set to obtain a residual pointwise scene flow. By adding the residual pointwise scene flow to the initial pointwise scene flow, a second pointwise scene flow for the first point cloud frame can be obtained. Finally, the second pointwise scene flow is output as a result. This recursive scene flow estimation scheme can be iteratively performed for more than two iterations until a certain condition is met, for example, until the number of iterations reaches a predetermined threshold.

[0046] For this seventh embodiment, integrating the task-aware point cloud downsampling method is described in the context of scene flow estimation. The same iterative principles may be applied, without loss of generality, to any other task dealing with point clouds such as those mentioned above.

[0047] 7 illustrates a method for point cloud compression using an embodiment of a task-aware point cloud downsampling method according to the present principles. In this encoder embodiment, a downsampled point cloud is used to construct a predicted point cloud for a predictive coding task. Given an input point cloud to be encoded, X, a downsampled point cloud is generated using a task-aware point cloud downsampling method 71, as described in connection with one of the previous embodiments. The downsampled point cloud, and optionally the generated set-level feature vectors, are then encoded by a first entropy encoder 72 to obtain a first bitstream BS1. The downsampled point cloud, and optionally the generated set-level feature vectors, are then encoded by a first entropy encoder 72 to obtain a predicted point cloud X that is close to X. PFinally, a second entropy encoder 74 generates the residual point cloud X R =XX P to obtain a second bitstream BS2. The two bitstreams are transmitted together to the decoder. The entropy encoders 72 and 74 can be either lossless or lossy.

[0048] Figure 8 shows a decoder embodiment of the present principles. The downsampled point cloud (and feature vectors, if available) is decoded from the first bitstream BS1 by a decoder module 81 and fed to a predictor construction module 82 to generate a predicted point cloud

[0049]

number

[0050]

number

[0051]

number

[0052]

number

[0053] This decoder embodiment can be used for either inter-frame predictive coding or intra-frame predictive coding. It differs from conventional scalable coding in two aspects: on the one hand, the decoder does not restrict the downsampled point cloud to be a subset of the input point cloud. On the other hand, apart from the downsampled point cloud, the predicted point cloud can also be generated using feature vectors produced by the task-aware downsampling module of the encoder associated with Figure 7, which gives further flexibility to this predictive coding scheme.

[0054] Figure 9 shows an example architecture of a device 30 that may be configured to implement the methods described in connection with Figures 1, 5, 6, 7, and 8. Different embodiments of encoders and decoders according to the present principles may implement this architecture. Alternatively, the circuits of an encoder and / or decoder according to the present disclosure may be devices according to the architecture of Figure 9, coupled together, for example, via their bus 31 and / or via I / O interface 36.

[0055] The device 30 comprises the following elements coupled together by a data and address bus 31: A microprocessor 32 (or CPU), for example a DSP (Digital Signal Processor), ROM (Read Only Memory) 33; RAM (Random Access Memory) 34; a storage interface 35; an I / O interface 36 for receiving data to be transmitted from the application; A power source, such as a battery (not shown).

[0056] According to one example, the power source is external to the device. In each of the mentioned memories, the word "register" as used herein may correspond to a small area (a few bits) or a very large area (e.g., an entire program or a large amount of received or decoded data). The ROM 33 contains at least programs and parameters. The ROM 33 can store algorithms and instructions for performing techniques according to the present principles. When switched on, the CPU 32 uploads the program in the RAM and executes the corresponding instructions.

[0057] The RAM 34 contains in registers the programs executed by the CPU 32 and uploaded after switching on the device 30, input data in registers, intermediate data for different states of the methods in registers, and other variables used for the execution of the methods in registers.

[0058] The implementations described herein may be implemented in, for example, a method or process, an apparatus, a computer program product, a data stream, or a signal. Even when discussed only in the context of a single form of implementation (e.g., discussed only as a method or device), the implementation of the discussed features may also be implemented in other forms (e.g., a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. The method may be implemented in, for example, an apparatus such as a processor, which generally refers to a processing device, including a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0059] According to examples of the present disclosure, device 30 includes: Mobile devices and a communication device; Gaming devices, A tablet (or tablet computer) and ·Laptop and a still or video camera, for example equipped with a depth sensor; A still or video camera rig; an encoding chip; A server (for example, a broadcast server, a video-on-demand server, or a web server) and belongs to a set including:

[0060] Implementations of the various processes and features described herein may be embodied in a variety of different devices or applications, particularly devices or applications associated with, for example, data encoding, data decoding, view generation, texture processing, and other processing of images and associated texture and / or depth information. Examples of such devices include encoders, decoders, post-processors that process output from decoders, pre-processors that provide input to encoders, video coders, video decoders, video codecs, web servers, set-top boxes, laptops, personal computers, mobile phones, PDAs, and other communication devices. As should be clear, the devices may be mobile and installed in mobile vehicles.

[0061] Additionally, methods may be implemented by instructions executed by a processor, and such instructions (and / or data values ​​produced by an implementation) may be stored on a processor-readable medium, such as, for example, an integrated circuit, software carrier, or other storage device, e.g., a hard disk, a compact diskette ("CD"), an optical disk (e.g., a DVD, often referred to as a digital versatile disk or digital video disk), a random access memory ("RAM"), or a read-only memory ("ROM"). The instructions may form an application program tangibly embodied on the processor-readable medium. The instructions may be, for example, hardware, firmware, software, or a combination. The instructions may be found, for example, in an operating system, a separate application, or a combination of the two. A processor may thus be characterized, for example, as both a device configured to execute a process and a device that includes a processor-readable medium (e.g., a storage device) having instructions for executing a process. Furthermore, a processor-readable medium may store data values ​​produced by an implementation in addition to, or in place of, instructions.

[0062] As will be apparent to those skilled in the art, implementations may produce a variety of signals formatted to carry information that may be, for example, stored or transmitted. The information may include, for example, instructions for performing a method or data produced by one of the described implementations. For example, a signal may be formatted to carry, as data, rules for writing or reading syntax of a described embodiment, or to carry, as data, the actual syntax values ​​written by a described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The signal it carries may be, for example, analog or digital information. The signal may be transmitted over a variety of different wired or wireless links, as is known. The signal may be stored on a processor-readable medium.

[0063] Many implementations have been described. Nevertheless, it will be understood that various modifications may be made. For example, elements of different implementations may be combined, supplemented, modified, or deleted to produce other implementations. Additionally, those skilled in the art will understand that other structures and processes may be substituted for those disclosed, with the resulting implementation performing at least substantially the same function in at least substantially the same way to achieve at least substantially the same results as the disclosed implementations. Accordingly, these and other implementations are contemplated by this application.

Claims

1. 1. A method comprising the steps of: generating a point-level feature vector for each point of the point cloud by inputting a position vector of the point into a neural network, and generating a set-level feature vector of the point cloud by using the neural network; generating a representative position based on the point level feature vector and the set level feature vector; and outputting the representative positions and the set-level feature vector as a set descriptor.

2. generating the representative location, calculating a weighting factor for each point in the point cloud based on a similarity measure by calculating an inner product between the point-level feature vector and the set-level feature vector; and generating the representative location by a weighted average of all points using the weighting factors.

3. generating the point level feature vectors and the set level feature vectors, accessing an anchor point of the point cloud by performing farthest point sampling; generating an extended point for each point of the point cloud by adding a position vector of the anchor point to the position vector of the point; and using the expansion points as inputs to the neural network.

4. generating the representative location, accessing an anchor location of the point cloud by performing farthest point sampling; generating correction vectors for said anchor positions by an extended neural network; and generating the representative position by adding the correction vector to the anchor position.

5. The method of any one of claims 1 to 4, wherein the point cloud is first downsampled by a task-aware downsampling method.

6. The task recognition method is a predictive coding task, and the downsampled point cloud comprises: - coded by a first entropy coding method, a predictor construction module to obtain a predicted point cloud, said method comprising coding a residual point cloud, which is the difference between said point cloud and said predicted point cloud, by a second entropy coding method.

7. The method of claim 6 , wherein the set-level feature vector is encoded together with the downsampled point cloud by the first entropy encoding method.

8. 1. A method for extracting a point cloud from a data stream, comprising: - Obtaining a downsampled point cloud, a residual point cloud, and a set-level feature vector from the data stream; - feeding the downsampled point cloud and the set-level feature vector to a predictor construction module to obtain a predicted point cloud; - deriving said point cloud by adding said predicted point cloud to said residual point cloud.

9. 1. A device comprising a processor associated with a memory, the processor performing, for a point cloud: generating a point-level feature vector for each point of the point cloud by inputting a position vector of the point into a neural network, and generating a set-level feature vector of the point cloud by using the neural network; generating a representative position based on the point level feature vector and the set level feature vector; and outputting the representative positions and the set-level feature vector as a set descriptor.

10. The processor, calculating a weighting factor for each point in the point cloud based on a similarity measure by calculating an inner product between the point-level feature vector and the set-level feature vector; 10. The device of claim 9, configured to generate the representative location by: generating the representative location by a weighted average of all points using the weighting coefficients.

11. The processor, accessing an anchor point of the point cloud by performing farthest point sampling; generating an extended point for each point of the point cloud by adding a position vector of the anchor point to the position vector of the point; 10. The device of claim 9, configured to generate the point-level feature vectors and the set-level feature vectors by: using the expansion points as inputs of the neural network.

12. The processor, accessing an anchor location of the point cloud by performing farthest point sampling; generating correction vectors for said anchor positions by an extended neural network; 10. The device of claim 9, configured to generate the representative position by: generating the representative position by adding the correction vector to the anchor position.

13. The device of any one of claims 9 to 12, wherein the processor is configured to first downsample the point cloud by using a task-aware downsampling method.

14. The task recognition method is a predictive coding task, and the downsampled point cloud comprises: - coded by a first entropy coding method, The device of claim 13, further comprising: a predictor construction module for obtaining a predicted point cloud, the method comprising: coding, by a second entropy coding method, a residual point cloud, which is the difference between the point cloud and the predicted point cloud.

15. The device of claim 14 , wherein the set-level feature vector is encoded together with the downsampled point cloud by the first entropy encoding method.

16. 1. A device for extracting a point cloud from a data stream, the device comprising: a processor associated with a memory, the processor comprising: - Obtaining a downsampled point cloud, a residual point cloud, and a set-level feature vector from the data stream; - feeding the downsampled point cloud and the set-level feature vector to a predictor construction module to obtain a predicted point cloud; - deriving said point cloud by adding said predicted point cloud to said residual point cloud.