Temporal aggregation for online 3D object detection

Abstract

To provide a method for machine learning a function for outputting localized representations (for example, for object detection or scene segmentation) of objects within a 3D point cloud representing a real 3D scene.SOLUTION: A method includes a step of obtaining a dataset of sequences of 3D point cloud frames. Each frame is associated with a time in the sequence and includes localized representations corresponding to respective objects. The method further includes a step of training a function based on the obtained dataset. The training step includes, for each sequence of the dataset and each given frame of the sequence, training the function to output localized representations of objects in the given frame based on the given frame and on at least the frame at the previous time in the sequence.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present invention relates to the field of computer programs and systems, and more particularly to methods, systems and programs for machine learning a function configured to take as input 3D point cloud frames of a real scene and output a localized representation of each object in the real scene. [Background technology]

[0002] Current state-of-the-art techniques in indoor 3D scene understanding, such as FCAF3D (D. Rukhovich, A. Vorontsova, and A. Konushin, “FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection.” arXiv, Mar. 24, 2022. Accessed October 11, 2022 [online], available at http: / / arxiv.org / abs / 2112.00322) or TR3D (D. Rukhovich, A. Vorontsova, and A. Konushin, “TR3D: Towards Real-Time Indoor 3D Object Detection.” arXiv, Feb. 08, 2023. doi: 10.48550 / arXiv.2302.02858), rely on machine learning models trained in a supervised manner on annotated datasets. These models are trained on datasets containing indoor scenes, i.e., rooms with furniture. These scenes are represented according to the modality used to digitize them, typically 3D point clouds, camera feeds, or their 3D reconstructions.Public datasets currently in use include SUN RGB-D (S. Song, S. P. Lichtenberg, and J. Xiao, “SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite,” presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 567–576. Accessed November 16, 2022 [online], available at https: / / openaccess.thecvf.com / content_cvpr_2015 / html / Song_SUN_RGB-D_A_2015_CVPR_paper.html), ScanNet (A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Niesner, “ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes.” arXiv, Apr. 11, 2017. doi: 10.48550 / arXiv.1702.04405) and ARKitScenes (G. Baruch et al., “ARKitScenes: A diverse real-world dataset for 3D indoor scene understanding using mobile RGB-D data,” in Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 1), 2021. [Online], available at https: / / openreview.net / forum?id=tjZjv_qh_CE).

[0003] In particular, FCAF3D and TR3D belong to the convolutional neural network (CNN) class of 3D object detection methods, which has the following pipeline: (1) Take a 3D point cloud as input. (2) Convert the point cloud into a voxel grid, i.e., a 3D image representation. This is done by considering a grid discretization of space. If at least one point exists within a given volume element in 3D space, the corresponding 3D voxel is considered occupied. This can be seen as a generalization of pixel-like 2D image representations to the 3D domain. One of the key parameters of this voxelization is the grid size, which is typically 0.5–30 cm. The larger this grid size, the lower the resolution of the 3D representation and, therefore, the lower the potential quality of the model's subsequent predictions. However, increasing the grid size can reduce the computational and memory costs of the subsequent pipeline. (3) Apply a neural network consisting of multiple 3D convolutional layers. This can be done using the Minkowski Engine (C. Choy, J. Gwak, and S. Savarese, “4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks.” arXiv, Jun. 13, 2019. doi: 10.48550 / arXiv.1904.08755), which allows for efficient computation of 3D convolutions on a sparse (almost empty) voxel grid. (4) Object detection methods output multiple predictions, which may a) result in a large number (over 1000) of detected objects and b) may have significant overlap with each other. One intuition for this is that a detection model may correctly identify, for example, multiple parts of a given chair, as well as the entire chair itself, but output them as different detections. A common post-processing algorithm introduced in R-CNN (R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation.” arXiv, Oct. 22, 2014. doi: 10.48550 / arXiv.1311.2524.) is non-maximum suppression (NMS), which addresses these issues by removing smaller predictions of the same semantic category (e.g., “chair”) when their overlap with larger predictions reaches or exceeds a set threshold. This post-processing algorithm has no learned or otherwise trained parameters. When considering the predictions as bounding boxes, this overlap is measured as Intersection over Union (IoU), where IoU∈[0, 1]. For 3D bounding boxes, these union and intersection values ​​are calculated as volumes. For clarity, the 2D equivalents are visualized as areas in Figure 5. Example IoU thresholds for NMS are 0.25 and 0.5.

[0004] Taking advantage of the static nature of indoor scenes, state-of-the-art indoor 3D object detection methods preprocess data representing a given scene by removing time data associated with the acquisition method and simply grouping all inputs. As such, they are sometimes referred to as "offline" methods. Offline methods enforce invariance to the order in which data was acquired, forcing deep learning models to focus on spatial relationships and consider other objects in the room. However, this also enforces the bias that the input scene was scanned as a whole, forcing the detection model to rely on other objects in the room to guide predictions, limiting the quality of predictions in online detection scenarios. When considering the use of such models, one reasonable requirement is the ability to provide detection feedback to the user during acquisition, i.e., giving predictions about what the user has just scanned. This is called online detection. The purpose of this requirement is to suggest whether further scanning of a given part of the scene is necessary or whether the user can move on to other sections of the scene.

[0005] Furthermore, to provide user feedback during acquisition, such models must be evaluated as a whole on successive subsets of the entire scene, including points already acquired by the user. This suboptimal scheme requires redundant computations, thereby increasing the consumption of computer resources.

[0006] One example of such an offline method that attempts to solve an online problem is Apple's RoomPlan (“3D Parametric Room Representation with RoomPlan,” Apple Machine Learning Research. Accessed October 13, 2022 [online], available at https: / / machinelearning.apple.com / research / roomplan). To be able to detect recursive objects in real time, they use the following design choices: (1) They chose a 3D CNN architecture with a high voxel size of 15cm, making it very difficult to identify objects smaller than this size, such as books. As shown in the FCAF3D paper, reducing the resolution of state-of-the-art 3D CNNs from 1cm to 3cm generally results in a decrease in detection quality. This is likely why RoomPlan aims to detect only large objects (storage, sofas, tables, chairs, beds, refrigerators, ovens, stoves, dishwashers, washing machines or dryers, fireplaces, sinks, bathtubs, toilets, stairs, and TVs). (2) They use local detectors during online scanning to predict bounding boxes directly from disjoint subsets of the input point cloud, and then aggregate these bounding boxes using an undisclosed algorithm. It can be argued that online 3D object detection is already performed by outdoor object detection methods designed for robotics or autonomous driving. Indeed, to accurately detect potentially occluded objects in dynamic scenes, spatiotemporal relationships need to be taken into account.

[0007] However, outdoor models are tasked with detecting potential obstacles, i.e., large objects, whereas rooms (indoor scenes) feature both large furniture and small objects such as books. Furthermore, outdoor detection aims to detect objects such as pedestrians and cars, each of which occupies its own vertical space. Indeed, PointPillars (A.H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “PointPillars: Fast Encoders for Object Detection From Point Clouds,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 12697–12705. Accessed October 14, 2022 [online], available at https: / / openaccess.thecvf.com / content_CVPR_2019 / html / Lang_PointPillars_Fast_Encoders_for_Object_Detection_From_Point_Clouds_CVPR_2019_paper.html), BEVFusion (Z. Liu et al., “BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation,” arXiv, Jun. 16, 2022) State-of-the-art 3D outdoor detection models, such as EA-LSS (H. Hu et al., “EA-LSS: Edge-aware Lift-splat-shot Framework for 3D BEV Object Detection.” arXiv, Aug. 29, 2023. doi: 10.48550 / arXiv.2205.13542.) or EA-LSS (H. Hu et al., “EA-LSS: Edge-aware Lift-splat-shot Framework for 3D BEV Object Detection.” arXiv, Aug. 29, 2023. doi: 10.48550 / arXiv.2303.17895.), explicitly integrate this prior information by embedding multi-sensor inputs into a unified 2D bird's-eye view (i.e., top-down) view.This reduction of a 3D problem to a 2D problem is unlikely to work for indoor scenes containing truly 3D positioned objects, such as hanging plants or books on a table.

[0008] Therefore, there is a need for an improved solution for outputting localized representations of objects in a 3D point cloud representing a real 3D scene (e.g., for object detection or scene segmentation). Summary of the Invention

[0009] Accordingly, a computer-implemented method for machine learning a function is provided. The function is configured to take as input 3D point cloud frames of a real scene and output localized representations, each corresponding to a respective object in the real scene. The method includes obtaining a dataset of sequences of 3D point cloud frames, each frame associated with a time in the sequence. Each frame includes a localized representation corresponding to each object. The method further includes training the function based on the obtained dataset. The training includes, for each sequence of the dataset and each given frame of the sequence, training the function to output a localized representation of the object in the given frame based on the given frame and at least a frame at a previous time in the sequence.

[0010] The method may include one or more of the following. Functions include: a first neural network configured to take a frame as input and output at least one feature vector corresponding to a localized representation corresponding to each object in the frame; and a second neural network configured to aggregate the feature vectors output by the first neural network for the given frame and at least an earlier frame in time in the sequence; The first neural network is configured to take a frame as input and output two or more feature vectors, each corresponding to a different resolution level, and the second neural network is configured to perform two or more aggregations, each corresponding to the feature vectors at the same resolution level. The first neural network is a convolutional neural network (CNN) and / or the second neural network is a combination of a transformer neural network and a convolutional neural network. Each 3D point cloud frame of the real scene represents a partial view of the real scene. The training includes batch training. Each batch respects the temporal order of frames, and during batch training, the function does not output localized representations of the batch based on frames associated with future times. In the acquired dataset, for each sequence of point cloud frames (X1, X2, …, X T ), (where,

number

number

[0011] Functions obtainable according to the method are also provided.

[0012] A computer-implemented method of using the function is also provided, the method including providing a sequence of 3D point cloud frames of a real scene, and for each frame of the sequence, determining a localized representation corresponding to each object in the real scene in the frame by applying the function to the frame. The application of the function is based on feature vectors corresponding to localized representations corresponding to each object in at least the previous frame.

[0013] Further provided is a computer program comprising instructions for carrying out the method and / or method of use.

[0014] There is further provided a device comprising a data storage medium having computer programs and / or functions recorded thereon.

[0015] The device may form or provide a non-transitory computer-readable medium, such as in a Software as a Service (SaaS) or other server or cloud-based platform.

[0016] The device may alternatively comprise a processor coupled to the data storage medium. Thus, the device may form, in whole or in part, a computer system (e.g., the device is a subsystem of the overall system). The system may further comprise a graphical user interface coupled to the processor. [Brief explanation of the drawings]

[0017] [Figure 1] This method will now be described. [Figure 2] This method will now be described. [Figure 3] This method will now be described. [Figure 4] This method will now be described. [Figure 5] This method will now be described. [Figure 6]This method will now be described. [Figure 7] An example of this system is shown below. DETAILED DESCRIPTION OF THE INVENTION

[0018] With reference to the flowchart of FIG. 1 , a computer-implemented method for machine learning a function is proposed. The function is configured to take 3D point cloud frames of a real scene as input and output localized representations, each corresponding to a respective object in the real scene. The method includes obtaining a dataset of sequences of 3D point cloud frames, each frame associated with a time in the sequence. Each frame includes a localized representation corresponding to each object. The method further includes training the function based on the obtained dataset. The training includes, for each sequence of the dataset and each given frame of the sequence, training the function to output a localized representation of the object in the given frame based on the given frame and at least a frame at a previous time in the sequence.

[0019] The method constitutes an improved solution for outputting a localized representation of an object within a 3D point cloud representing a scene.

[0020] In practice, the method trains a function to perform this output based on a sequence of 3D point cloud frames, with each frame associated with a time in the sequence (i.e., the sequence is thus temporal, with each frame corresponding to a time in the sequence). Such a sequence may typically correspond to a real-time 3D scan of a real scene, where the scan progressively scans the scene, thereby acquiring 3D point cloud frames successively (i.e., at short, regular time intervals), with each frame corresponding to a spatial portion of the scene and acquired at a specific time in the scanning process. This may correspond, for example, to a user moving through a scene (e.g., an indoor room with furniture) and manipulating a scanning device to scan the scene. For a given input frame, the function is trained to output a localized representation that takes into account not only this frame but also at least the preceding frames in the sequence (i.e., at least the frames with a previous time in the sequence). Thus, the function learns to infer spatial relationships between different frames (i.e., spatial relationships between different regions of the scene) as well as temporal relationships between different frames in the sequence. This improves the accuracy of the output.

[0021] Furthermore, the function is trained with consideration given to conserving computational resources during its use (also known as the online / inference phase / stage). In practice, a real scene is captured by a sequence of 3D point cloud frames (e.g., because it is not possible to physically obtain the same measurements of the entire scene in a single point cloud / image taken from a single viewpoint, such as through 3D scanning, or such a sequence may correspond to a user operating a scanning device (e.g., on their phone) to sequentially capture the scene by walking through the scene and moving the device). Performing localized representation determination based on each individually acquired frame may result in insufficient accuracy because spatial and / or temporal relationships between frames are not taken into account, or at least not sufficiently taken into account. In contrast, performing localized representation determination using all frames together in the same calculation step increases the consumption of memory and computational resources of the computer system. The present method provides a balance between these two possibilities by determining, for each given frame in the sequence, a localized representation within the frame using computations (e.g., feature vectors, as described below) already performed for at least the previous frames in the sequence (e.g., only this frame, or the previous two frames, or the previous three frames). This allows spatial as well as temporal relationships between different frames in the sequence to be taken into account, as discussed above, thereby providing precision while allowing efficient reuse of computations already performed.In particular, during use, for a given frame processed by the function, only the calculations (e.g., feature vectors as described below, which, incidentally, are cheaper in terms of storage memory resources than the corresponding frame itself) made on at least the previous frame (e.g., only the previous frame, or only the previous two frames) need to be stored in the computer's RAM (Random Access Memory) or VRAM (Video RAM) or in the cache of the computer software application executing the method, accessible by the function, and used together with the given frame to output a localized representation within the given frame.

[0022] The method is for machine learning of a function, where the function is a neural network (also referred to as a "neural network function"), and the neural network may be a configuration of neural networks, optionally a configuration of one or more deterministic layers, with the configuration itself being considered a neural network. Thus, the method is a machine learning method, learning / training a function. As is known per se from the field of machine learning, processing an input by a neural network includes applying to the input an operation defined by data including weight values. Thus, training a neural network includes determining weight values ​​based on a dataset configured for such learning; such a dataset may be referred to as a learning dataset or training dataset. Thus, a dataset includes data pieces, each of which forms a respective training sample. The training samples represent the diversity of situations in which the neural network will be used after learning. Any training dataset herein may include more than 1,000, 10,000, 100,000, or 1,000,000 training samples. In the context of this disclosure, "training a neural network (or function) based on a dataset" means that the dataset is a learning / training dataset for a neural network based on which the values ​​of the weights (also called "parameters") are set. In this disclosure, a training dataset is an acquired dataset of a sequence of 3D point cloud frames from which a function is learned.

[0023] The function is configured (i.e., trained) to take 3D point cloud frames of a real scene as input and output a localized representation of each object in the real scene. The function may, for example, take a sequence of point cloud frames as input and calculate a localized representation for each frame in the sequence, each time using at least the previous frame (i.e., at least the frame having an earlier time in the sequence) or the calculations performed on said at least previous frame (e.g., its feature vector, as described above). Alternatively, the function may take each frame of the sequence as input individually and sequentially (one by one) according to the temporal order of the frames in the sequence and, for each frame, calculate a localized representation using at least the previous frame (i.e., at least the frame having an earlier time in the sequence) or the calculations performed on said at least previous frame (e.g., its feature vector, as described above) (e.g., by accessing the results of these calculations (e.g., feature vectors) from the computer's RAM or VRAM or from the cache of the computer software application executing the method, as described above). In either case, the function may output the localized representations computed for each frame, or, as described below, may apply a post-processing module to filter the predictions and keep the most relevant ones, which are then output.

[0024] A 3D point cloud frame is a 3D point cloud that corresponds to a partial view of a real scene, i.e., a 3D point cloud representation of this partial view. Any 3D point cloud or point cloud frame herein is a set of 3D points (i.e., each comprising a triplet of coordinates in 3D space), each representing a location within the scene (or, where appropriate, a partial view of a scene) represented by the point cloud or point cloud frame. Each point may, in examples, further comprise one or more additional coordinates (e.g., RGB coordinates) representing the color of the location. In these examples, the function thus takes into account the color within the scene. A real scene refers to a portion of the real world (e.g., a view of a real-world room such as a kitchen). Any scene herein may also be an indoor scene, such as a room with furniture. The function therefore has a structure that is configured, i.e., trained and adapted, to take a 3D point cloud frame as input. This does not exclude the function being used for multiple frames to, in use, output a localized representation of one of these frames taking into account the other frames, as described below.

[0025] The function outputs a localized representation of each object in the real scene (i.e., represented by the input point cloud frame). The localized representation of each object is data representing the object's geometric location within the scene and data representing the object's semantic class (e.g., a class is a type of object, such as a type of furniture, or a segment type for scene segmentation). The data representing the geometric location may be a bounding box (e.g., rectangular or circular) around or substantially around the object. The bounding box may be defined by a set of coordinates (x, y, z) representing a 3D location (e.g., its center), a size (w, l, h) (width, length, height), and an orientation θ. The data representing the semantic class may be any suitable type of data, such as a label or a string description describing the object within the box. The semantic class may be a semantic class for segmentation (e.g., a type of segment) or a type of object (e.g., a type of furniture or indoor object if the scene represents a room scene). All semantic classes herein may belong to a predetermined set of semantic classes (e.g., 10-40 semantic classes, e.g., 32 semantic classes), e.g., each corresponding to a type of object (e.g., oven, dishwasher, refrigerator, etc.). Any object herein may be a large object (e.g., a large piece of furniture such as a refrigerator) or a small object (e.g., a large indoor item such as a book). A small object herein is an object with a volume of 0.01 m 3 may be defined as an object whose maximum dimension is less than 0.3 m and / or whose maximum dimension is less than 0.3 m.

[0026] The method includes obtaining a data set of a sequence of 3D point cloud frames.

[0027] Thus, a dataset consists of sequences, each of which is a sequence of 3D point cloud frames. The point cloud frames in each sequence may all relate to the same scene. That is, each frame in the sequence represents a partial view of the same scene (and thus corresponds to a sequence). The sequences in a dataset may all or substantially all correspond to the same type of scene, e.g., all or substantially all are indoor scenes. For example, furnished rooms (e.g., all indoor scenes of a home), or rooms of the same type (e.g., all kitchen scenes, or other types of indoor rooms). The degree to which all or substantially all scenes represent the same type of room may vary and may depend on the intended use of the function. For example, if the function is intended to be specialized for rooms of the same type (e.g., kitchens), all or substantially all scenes considered in the dataset may be of this type (e.g., all kitchens). Alternatively, if the function is intended to be applied more generally to any indoor scene, e.g., any indoor room scene, the dataset may include sequences related to various types of indoor (e.g., room) scenes.

[0028] For each sequence, each frame is associated with a time in the sequence. This allows the frames to be ordered within the sequence according to time. For example, each sequence can be made up of (X1, X2, …, X T ), where:

number

[0029] Each frame includes a localized representation corresponding to each object. As explained above, each localized representation (corresponding to an object) includes data representing the geometric location of the object within the scene and data representing the object's semantic class (e.g., a class is a type of object, such as a type of furniture, or a segment type for scene segmentation). The data representing the geometric location may be a bounding box (e.g., rectangular or circular) around or substantially around the object. The bounding box may be defined by a set of coordinates (x, y, z) representing a 3D location (e.g., its center), a size (w, l, h) (width, length, height), and an orientation θ. The data representing the semantic class may be any suitable type of data, such as a label for the bounding box or a string description describing the object (or its class) within the box. The semantic class may be a semantic class for segmentation (e.g., a type of segment) or a type of object (e.g., a type of furniture or indoor object if the scene represents a room scene). All semantic classes herein may belong to a predetermined set of semantic classes, e.g., each corresponding to a type of object (e.g., oven, dishwasher, refrigerator, etc.). Thus, the function may be used for segmenting a real scene or detecting bounding boxes around objects in a scene and their associated semantic classes. The function may output localized representations of all or substantially all objects (e.g., objects associated with classes belonging to a predetermined set of classes, e.g., a predetermined set of furniture and / or indoor items). Thus, the training dataset may include frames capturing the appropriate variability and quantity of these objects, as is known in the field of machine learning.

[0030] Each sequence in the training dataset may be obtained from raw 3D data representing a scene corresponding to the sequence (e.g., data measured by a physical sensor (e.g., a scanning device (e.g., a 3D scan), etc.), or data corresponding to a virtual scan of the scene, etc.). An example process for obtaining a sequence based on such raw 3D data is described below. Obtaining the dataset may include running this process, or alternatively, may include obtaining (e.g., downloading) a dataset already obtained from this process from a (e.g., remote) memory or server or database or cloud (where the dataset is stored after acquisition).

[0031] The process begins with acquiring raw 3D data (i.e., for each sequence) as a video stream using a physical or virtual scanning device, where the value of each pixel corresponds to the distance between the object displayed in the image and the camera / scanning device (pixels may also include RGB data, as described above). Using known camera parameters (including its 3D position and orientation), each pixel is mapped to a point in 3D space according to a geometric backprojection operation T. Each of these images, called a depth image or depth map I, is backprojected into a separate 3D point cloud T(I).

[0032] The resolution of any depth image herein may range from a low resolution of 128x128 to 256x192 (e.g., for consumer mobile devices) to 1920x1440 or higher (e.g., for professional devices). Each pixel corresponds to a 3D point, and the scanning device may also calculate an additional confidence score ranging from 1 (best) to 3 (worst) that estimates the accuracy of each point's location, allowing it to reject potentially erroneously measured locations. Furthermore, the device may select when to capture an image based on heuristics that indicate sufficient difference between each depth map, such as the time of day, distance traveled, or camera rotation between each capture. In this context, the camera's position and orientation may be obtained, for example, by a combination of odometry from an on-device accelerometer and / or a registration algorithm.

[0033] As mentioned above, any point cloud (e.g., obtained from a depth map) described above may be acquired either from a real device in a real indoor scene or from a simulated camera moving in a virtual 3D scene. While real acquisition more closely matches the intended use case, simulated acquisition in a virtual environment with known geometry allows for the generation of robust synthetic data with rich semantic annotations without the need for human annotators. Point cloud depth maps may be acquired, inter alia, by applying the method for generating training datasets disclosed in European Patent Application EP 23305001.2, which is incorporated herein by reference.

[0034] In our implementation, in the acquired dataset, each sequence of point cloud frames is denoted by X1,X2,…,X T where,

number

number

[0035] In implementation, the values ​​of the above variables may include: The number of images (or views) V depends on the scene

number

number

number

[0036] Note that there are other ways to acquire point clouds, and these may be used in alternative implementations. For example, some professional devices do not output depth maps, but instead output a 3D point cloud directly based on a 360° scan. Note that such devices still need to deal with occlusion issues and require the user to move it to fully capture a complex scene. Therefore, regardless of how the point cloud is acquired, the dataset is calculated as frames (X1, X2, …, X T ) sequence.

[0037] In addition to obtaining the training dataset, the method includes training a function based on the obtained dataset (i.e., the obtained dataset is a training dataset for training the function). The training includes, for each sequence in the dataset and each given frame in the sequence, training the function to output a localized representation of the object in the given frame based on the given frame and at least the previous frame in the sequence (i.e., using only the previous frame in the sequence, or possibly one or more previous frames).

[0038] The function may include two neural networks (e.g., being or including a configuration). The two neural networks include a first neural network and a second neural network. The first neural network is configured to take a frame as input and output at least one feature vector corresponding to a localized representation for each object in a real scene. The feature vector, also known as an "embedding" in the field of machine learning, forms a compact representation that captures (e.g., all) of the localized representations in the frame (i.e., the 3D point cloud frame) taken as input by the first neural network. The second neural network is configured to aggregate the feature vectors output by the first neural network for a given frame and at least the previous frame in the sequence. In other words, for each sequence encountered, the first neural network takes all frames of the sequence as input (continuously or in a batch manner) and outputs a respective feature vector for each respective frame taken as input. For each feature vector output by the first neural network, the second neural network aggregates this feature vector with the feature vector of the frame associated with the previous time in the sequence, and possibly with one or more feature vectors corresponding to one or more frames associated with one or more times before the previous time (e.g., the previous time, or the two previous times, or the three previous times). The first neural network may be called the "local network" or "backbone network" and may be denoted by f. This neural network is evaluated exactly once for each point cloud frame in the input sequence separately. The second neural network may be called the "aggregation network" and may be denoted by g. This neural network receives the outputs of the local neural networks as a sequence and outputs either a prediction for the last frame in the sequence or a prediction for each frame in the sequence.The function may also include a detection head that is applied to the result of the aggregation, as described below.

[0039] The first neural network may be configured to take a frame as input and output two or more feature vectors, each corresponding to a different resolution level. In other words, the first neural network outputs two or more feature vectors for each input frame, each corresponding to a different resolution level of a localized representation of an object in the frame. These various resolution levels allow for different sizes of objects (i.e., objects of various sizes). In this case, the second neural network is configured to perform two or more aggregations, each corresponding to feature vectors at the same resolution level. In other words, for each frame (or feature vector) involved in the aggregation performed by the second neural network (i.e., a given input frame and one or more previous frames), two or more aggregations of these frame feature vectors are performed, one for each resolution level (i.e., one aggregation is performed for all frame feature vectors at the highest resolution level, one aggregation is performed for all frame feature vectors at the next higher resolution level, and so on down to the lowest resolution level). The first neural network may be a convolutional neural network (CNN). The second neural network may be a combination of a transformer neural network and convolution. The method may consider two, three, or four resolution levels. Each resolution level may be or correspond to a depth level. The resolution levels used for aggregation may be 8 cm, 16 cm, 32 cm, and 64 cm (levels 16 cm and 32 cm are particularly useful).

[0040] A local neural network (or backbone) f first computes the vectors of each point cloud frame X t are evaluated independently and the intermediate features (or embeddings) Z t =f(Xt ). In our implementation, these features are intermediate computations in a custom variant of TR3D (see D. Rukhovich, A. Vorontsova, and A. Konushin, “TR3D: Towards Real-Time Indoor 3D Object Detection.” arXiv, Feb. 08, 2023. doi: 10.48550 / arXiv.2302.02858, which is incorporated herein by reference), and apply a lightweight parametric function (called the “detection head” in the aforementioned reference) h to generate an initial frame-wise prediction.

number

number

number

[0041] The aggregate neural network g then computes the intermediate features Z output by the local neural networks. t Using the final prediction

number

number

number

[0042] In our implementation, the aggregation network g corresponds to a sequence of fusion aggregation modules (FAMs) to which detection heads h are attached. This is illustrated in Figure 2, which shows the architecture. FAMs are based on TransPillars (see Luo, G. Zhang, C. Zhou, T. Liu, S. Lu, and L. Pan, “TransPillars: Coarse-to-Fine Aggregation for Multi-Frame 3D Object Detection,” arXiv, Aug. 04, 2022. doi: 10.48550 / arXiv.2208.03141, which is incorporated herein by reference), and use a similar concept to 2D feature maps for online outdoor detection. Our implementation can adapt this concept to 3D features as follows: transform image tokens (patches), apply a transformer, and recombine the patches at the output. As mentioned above and known from the field of CNNs, features from different resolution levels (or depths) may be output to take into account various possible sizes of the object to be detected. For example, the aforementioned TR3D outputs two features per input point cloud, while FCAF3D (see D. Rukhovich, A. Vorontsova, and A. Konushin, “FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection,” arXiv, Mar. 24, 2022. Accessed October 11, 2022 [online], available at http: / / arxiv.org / abs / 2112.00322, which is incorporated herein by reference) outputs four features per input. Therefore, an implementation of an aggregation network consists of at least one FAM per resolution level. Figure 2 shows an aggregation network with two resolution levels and one FAM per resolution level, which corresponds to the implementation of the present method.

[0043] In our implementation, features from a high-resolution level are processed first and then passed to the next FAM, which fuses the features from that resolution level with the high-resolution level. The output of all FAMs is then fed to the detection head to output the final prediction. For clarity and readability, however, the connections between the FAMs and the detection head are not shown in Figure 2. In these implementations, each FAM consists of a combination of 3D convolutional and transformer layers (see A. Vaswani et al., “Attention Is All You Need,” arXiv, Dec. 05, 2017. doi: 10.48550 / arXiv.1706.03762, incorporated herein by reference) and deformable attention (see X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable DETR: Deformable Transformers for End-to-End Object Detection,” arXiv, Mar. 17, 2021. Accessed October 17, 2022 [online], available at http: / / arxiv.org / abs / 2010.04159, incorporated herein by reference). The FAM architecture is shown in Figure 3.

[0044] Note that the above descriptions of both the aggregation network g and the FAM are implementation details of neural networks that have been tested by the inventors and are believed to provide satisfactory results. However, alternative suitable architectures or modifications may be considered, such as using scaled dot-product attention (discussed in the aforementioned reference A. Vaswani et al., “Attention Is All You Need,” arXiv, Dec. 05, 2017. doi: 10.48550 / arXiv.1706.03762) instead of deformable attention, or using a coarse-to-fine fusion aggregation order instead of a fine-to-coarse order. Networks with either FCAF3D or TR3D variants of the local network f may also be considered. These alternatives have been tested and provide satisfactory results.

[0045] The function may optionally include a post-processing section that applies post-processing in the form of non-maximum suppression (NMS) to filter the predictions and retain the most relevant ones.

[0046] Training may involve minimizing a loss based on labeled training samples, as known in the art. As with most deep learning methods, the function may be trained using a stochastic gradient descent algorithm, such as the implementation AdamW (see I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization,” arXiv, Jan. 04, 2019. doi: 10.48550 / arXiv.1711.05101, which is incorporated herein by reference). In tests performed by the inventors, the unique training, validation, and test annotated datasets used for the function were synthetically generated using the HomeByMe virtual scene, as outlined above, using the method of the aforementioned European Patent Application EP23305001.2.

[0047] Training may involve batch training, where each batch respects the temporal order of frames, and during batch training, the function does not output localized representations of the batch based on frames associated with future times. In other words, for / during training, the aggregation network may operate in batch mode and output all predictions corresponding to input frames.

number

number

number

[0048] In the implementation, online neural networks

number

[0049] The implementation of the training pipeline is shown in Figure 4.

[0050] The results of tests performed by the inventors using the implementation discussed above are shown in Table 1 below. [Table 1]

[0051] Table 1 shows the online test results on the dataset obtained with the method of European patent application EP23305001.2.

[0052] As shown in the table above, quantitative results show that the implementation of the present method significantly outperforms the baseline approach. The baseline method may be described as: Online model

number

[0053] Common metrics used in 3D object detection are mean precision (mAP) and mean recall (mAR). These metrics are defined using an Intersection over Union (IoU) threshold to evaluate the model's predictions. For a given class c, a predicted bounding box is considered a correct prediction if it can be paired with a ground truth bounding box such that the IoU between the two bounding boxes is greater than a given threshold. IoU (Intersection over Union) (IoU ∈ [0, 1]) may be written as follows: For 3D bounding boxes, these union and intersection values ​​are calculated as volumes. For clarity, the 2D equivalent is visualized as area in Figure 5. Examples of IoU thresholds for non-maximum suppression (NMS) are 0.25 and 0.5. The pairings are unique, and a given ground truth box is assigned to at most one prediction, and vice versa. The pairings are performed in descending order of IoU until either all ground truth or predicted bounding boxes are exhausted.

[0054] Consider the following definitions: True Positive (TP): Correct model predictions, i.e., predictions that match a bounding box of the same class and have IoU > threshold. False Positive (FP): Incorrect model prediction, i.e., the model predicted a bounding box but the corresponding ground truth annotation did not match or IoU ≤ threshold. False Negatives (FN): Missing predictions. An annotation that did not match any prediction (i.e. the object was present but not detected by the model). Precision is the percentage of positive predictions that are actually correct.

number

number

[0055] A computer-implemented method of using the function obtainable (e.g., learned) by the method is also provided. The method includes providing a sequence of 3D point cloud frames of a real scene. Providing the sequence of 3D point cloud frames may involve a user or automatic operation of a 3D scanning device (e.g., integrated into the user's phone) and continuously or repeatedly scanning the scene with the device. The scanning may directly result in a 3D point cloud frame of the scene or may result in an image (e.g., a depth image) of the scene. In the latter alternative, providing the sequence may include converting each image into a 3D point cloud frame as known in the art, thereby obtaining the sequence. Each frame of the sequence is associated with a time, which may be, or be derived from, the acquisition time provided by the scanning device (e.g., up to rescaling, etc.). The method further includes, for each frame of the sequence, determining a localized representation corresponding to each object in the real scene in the frame by applying the function to the frame. Thus, application of the function is performed iteratively on each frame, one after the other. The application of the function is based on feature vectors corresponding to localized representations of each object in at least the previous frame. For example, the application of the function may include, for each frame, applying a first neural network to the frame, thereby extracting a feature vector for the frame. Next, the application of the function may include applying a second neural network to perform aggregation of the feature vector extracted for the frame with feature vectors extracted for at least a frame having a previous time (this means that in the aggregation, feature vectors of a frame having a previous time are used, and optionally, one or more feature vectors of frames having one or more previous times are also used). Final detection of localized representations may be performed by applying the aforementioned detection head to the result of the aggregation after aggregation.The training method and the use method may be integrated into the same computer-implemented method, which includes the steps of performing the training method and then performing the use method of the function learned by the training method.

[0056] The trained function thus provides computational efficiency and accuracy when operating in an online setting, i.e., when applying the method. Indeed, by design, the heavier local network (backbone) is evaluated only once for each point cloud frame, and the lighter aggregation network refines its output into a frame-wise prediction. Thus, when applying the aggregation network to perform feature vector aggregation, only the feature vectors of the current frame and the previous frame (and optionally, if appropriate, the feature vectors of one or more frames at one or more previous times) need to be stored in the RAM (Random Access Memory) or VRAM (Video RAM) of the underlying computer in the cache of the computer software application executing the method. In contrast, existing offline indoor 3D object detectors must be evaluated on the concatenation of all previous frames in the sequence to produce acceptable results. The accuracy comes from the fact that the aggregation network computes both spatial relationships, i.e., between different regions of a scene, and temporal relationships, i.e., between different frames of a sequence. This efficient use of RAM or cache, among other things, allows for real-time (e.g., virtually instantaneous) determination of localized representations within a scene while the scene is being captured, e.g., while being scanned by a user moving through the scene) determination, which also allows for real-time visual feedback to be provided to the user as to whether scene capture was performed properly.

[0057] Furthermore, during point cloud acquisition, a user may move the scanning device to capture different parts of a given indoor scene. Indeed, parts of the scene may be occluded by other objects or simply not present in the device's field of view. In this context, the present methods, functions, and usages allow for outputting 3D semantic information related to recently captured points based on data from previously acquired scenes.

[0058] The method is computer-implemented, meaning that the steps (or substantially all steps) of the method are performed by at least one computer, or any similar system. Thus, the method steps are performed fully automatically or semi-automatically by a computer. In an example, triggering of at least some steps of the method may be performed through user-computer interaction. The level of user-computer interaction required depends on the level of automation envisioned and may be balanced against the need to implement user preferences. In an example, this level may be user-defined and / or predefined.

[0059] A typical example of a computer implementation of the method is executing the method on a system adapted for this purpose. The system may include a processor coupled to a memory and a graphical user interface (GUI), where the memory has recorded thereon a computer program containing instructions for carrying out the method. The memory may also store a database. The memory is any hardware suitable for such storage, and may optionally include multiple physically separate parts (e.g., one for the program and one for the database).

[0060] FIG. 7 shows an example of a system, where the system is a client computer system, eg, a user's workstation.

[0061] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000 and a random access memory (RAM) 1070 also connected to the bus. The client computer further includes a graphical processing unit (GPU) 1110 associated with a video random access memory 1100 connected to the bus. The video RAM 1100 is also known in the art as a frame buffer. A mass storage controller 1020 manages access to a mass memory device such as a hard drive 1030. Mass memory devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, such as semiconductor memory devices such as EPROMs, EEPROMs, and flash memory devices, magnetic disks such as internal hard disks and removable disks, and magneto-optical disks. Any of the above may be supplemented by or incorporated into specially designed application-specific integrated circuits (ASICs). A network adapter 1050 manages access to a network 1060. The client computer may also include a cursor control device, a haptic device 1090 such as a keyboard, etc. The cursor control device is used in the client computer to allow a user to selectively position a cursor at any desired location on the display 1080. Furthermore, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes a number of signal generating devices for inputting control signals to the system. Typically, the cursor control device may be a mouse, and the buttons on the mouse are used to generate the signals. Alternatively or additionally, the client computer system may be equipped with a pressure-sensitive pad and / or a pressure-sensitive screen.

[0062] A computer program may include computer-executable instructions, including means for causing the system to perform the method. The program may be recordable on any data storage medium, including the system's memory. The program may be implemented, for example, in digital electronic circuitry, or computer hardware, firmware, software, or a combination thereof. The program may also be implemented as an apparatus, for example, as an article of manufacture tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps may be performed by a programmable processor executing a program of instructions to perform the functions of the method by manipulating input data and generating output. The processor is thus programmable and may be coupled to receive data and instructions from, and transmit data and instructions to, a data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language as appropriate. In either case, the language may be a compiled or interpreted language. The program may also be a complete installation program or update program. Application of the program to the system, in either case, results in instructions for performing the method. The computer program may alternatively be stored and executed on a server in a cloud computing environment, the server communicating with one or more clients over a network, in which case the processing unit executes the instructions contained in the program, thereby performing the method in the cloud computing environment.

Claims

1. 1. A computer-implemented method for machine learning a function configured to take as input 3D point cloud frames of a real scene and output a localized representation corresponding to each object in the real scene, the method comprising: acquiring a dataset of a sequence of 3D point cloud frames, each frame associated with a time in the sequence, each frame containing a localized representation corresponding to a respective object; training the function based on the acquired dataset, the training comprising, for each sequence in the dataset and each given frame of the sequence, training the function to output a localized representation of an object in the given frame based on the given frame and at least an earlier frame in the sequence; A method comprising:

2. The function is a first neural network configured to take a frame as input and output at least one feature vector corresponding to a localized representation of each object in the frame; a second neural network configured to aggregate feature vectors output by the first neural network for the given frame and at least an earlier frame in the sequence; The method of claim 1 , comprising:

3. the first neural network is configured to take a frame as input and output two or more feature vectors, each corresponding to a different resolution level; The second neural network is configured to perform two or more aggregations, each corresponding to a feature vector at the same resolution level. The method of claim 2.

4. the first neural network is a convolutional neural network (CNN); and / or The second neural network is a combination of a transformer neural network and a convolutional neural network. The method according to claim 2 or 3.

5. Each 3D point cloud frame of the real scene represents a partial view of the real scene.

5. The method according to any one of claims 1 to 4.

6. The training includes batch training.

6. The method according to any one of claims 1 to 5.

7. Each batch respects the temporal order of frames, and during the batch training, the function does not output a localized representation of the batch based on frames associated with future times. The method of claim 6.

8. In the acquired dataset, for each sequence of point cloud frames (X 1 , X 2 , …, X T ), (where [Equation 1] is the point cloud frame at time t, and n t points) is defined by the following rules: [Equation 2] where ceil is the ceiling function and I 1 , I 2 , …, I V is a sequence of depth images of the scene, and T is a backprojection operator that projects pixels of the depth images onto 3D points.

8. The method according to any one of claims 1 to 7.

9. The scene is an indoor scene 9. The method according to any one of claims 1 to 8.

10. The 3D point cloud frames of the acquired data set are derived from physical or virtual measurements.

10. The method according to any one of claims 1 to 9.

11. A function obtainable according to the method of any one of claims 1 to 10.

12. 12. A computer-implemented method of using a function according to claim 11, said method comprising: providing a sequence of 3D point cloud frames of a real scene; for each frame of the sequence, determining a localized representation corresponding to each object of the real scene in said frame by applying said function to said frame based on feature vectors corresponding to localized representations corresponding to each object in at least a previous frame; Instructions for use include:

13. A computer comprising instructions that, when executed by a computer, cause the computer to carry out the method of any one of claims 1 to 10 and / or the method of claim 12.

14. A computer-readable data storage medium having recorded thereon a computer program according to claim 13 and / or a function according to claim 11.

15. A computer system comprising a processor and a memory coupled to said processor, said memory storing a computer program according to claim 13 and / or a function according to claim 11.

Images