Temporal aggregation for online 3D target detection

Abstract

The present disclosure relates to temporal aggregation for online 3D target detection, in particular to a computer-implemented method for machine learning functions. The function is configured to take a 3D point cloud frame of a real scene as input and to output a localized representation. Each output representation is a respective object relative to a real scene. The method includes obtaining a data set of a sequence of 3D point cloud frames. Each frame is associated with a time in the sequence. Each frame includes a localized representation of each respective object. The method also includes training the function based on the obtained data set. The training includes, for each sequence of the data set and each given frame of the sequence, training a function to output a localized representation of the object in the given frame based on the given frame and at least a frame of the sequence at a previous time.

Description

Technical Field

[0001] This disclosure relates to the field of computer programs and systems, and more specifically, to methods, systems, and programs for machine learning functions configured to take 3D point cloud frames of a real scene as input and output a localized representation of each corresponding target in the real scene. Background Technology

[0002] The most advanced methods in the field of indoor 3D scene understanding include FCAF3D ("FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection" by D. Rukhovich, A. Vorontsova, and A. Konushin, arXiv, March 24, 2022, accessed October 11, 2022, [online], available at: http: / / arxiv.org / abs / 2112.00322) or TR3D ("TR3D: Towards Real-Time Indoor 3D Object Detection" by D. Rukhovich, A. Vorontsova, and A. Konushin). Both "Detection" (arXiv, February 8, 2023, doi: 10.48550 / arXiv.2302.02858) and "Detection" rely on supervised machine learning models trained on annotated datasets. These models are trained on datasets containing indoor scenes (i.e., furnished rooms). These scenes are represented based on patterns used to digitize them, typically 3D point clouds, camera feeds, or their 3D reconstructions.Currently used public datasets include SUN RGB-D (S. Song, SPLichtenberg, and J. Xiao, “SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite”, published in the 2015 IEEE Conference on Computer Vision and Pattern Recognition Proceedings, pp. 567-576, accessed November 16, 2022, [online], available at: https: / / openaccess.thecvf.com / content_cvpr_2015 / html / Song_SUN_RGB-D_A_2015_CVPR_paper.html) and ScanNet (A. Dai, AXChang, M. Savva, M. Halber, T. Funkhouser, and M. Nieβner, “ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes”). Scenes), arXiv, April 11, 2017, doi: 10.48550 / arXiv.1702.04405; and ARKitScenes (G. Baruch et al., “ARKitScenes: A diverse real-world dataset for 3D indoor scene understanding using mobile RGB-D data,” published at the 35th Conference on Datasets and Benchmark Tracking for Neural Information Processing Systems (Round 1), 2021, [online], accessible at: https: / / openreview.net / forum?id=tjZjv_qh_CE).

[0003] Specifically, FCAF3D and TR3D belong to the Convolutional Neural Network (CNN) class of 3D object detection methods, and their processes are as follows:

[0004] 1. Use 3D point cloud as input.

[0005] 2. Convert the point cloud into a voxel mesh, i.e., a 3D image representation. This is done by considering the mesh discretization of space; if at least one point exists within a given volume element in 3D space, the corresponding 3D voxel is considered occupied. We can think of this as a generalization of pixel-level 2D image representation to the 3D domain. An important parameter for this voxelization is the mesh size, typically between 0.5 cm and 30 cm. The larger the mesh size, the lower the resolution of the 3D representation, and therefore the lower the potential quality of subsequent predictions from the model. However, increasing the mesh size allows for a reduction in the computational and memory costs of subsequent processes.

[0006] 3. Apply neural networks consisting of multiple 3D convolutional layers. This can be accomplished using the Minkowski engine (C. Choy, J. Gwak, and S. Savarese, “4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks”, arXiv, June 13, 2019, doi: 10.48550 / arXiv.1904.08755), which allows for efficient computation of 3D convolutions on sparse (almost empty) voxel grids.

[0007] 4. Object detection methods output multiple predictions that can: a) be a large number (over a thousand) of detected objects, and b) have significant overlap with each other. An intuitive understanding of this is that the detection model will correctly identify, for example, multiple parts of a given chair as well as the chair itself, but will output them as different detections. A common post-processing algorithm introduced in R-CNN (R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation”, arXiv, October 22, 2014, doi: 10.48550 / arXiv.1311.2524) is Non-maximum suppression (NMS), which addresses these problems by removing smaller predictions within the same semantic category (e.g., “chair”) when the overlap between the smaller prediction and the larger prediction reaches or exceeds a set threshold. This post-processing algorithm does not require learning or training parameters. When the predicted values ​​are considered as bounding boxes, this overlap is measured as the intersection-union ratio (IoU ∈ [0,1]). For 3D bounding boxes, these union and intersection values ​​are calculated as volume; for clarity, Figure 5The 2D equivalent value is visualized as an area. The exemplary IoU thresholds for NMS are 0.25 and 0.5.

[0008] Leveraging the static nature of indoor scenes, advanced indoor 3D object detection methods preprocess the data representing a given scene by removing any temporal data associated with the acquisition method and simply grouping all inputs together. Therefore, they can be called "offline" methods. Offline methods enforce an invariant data acquisition order and force deep learning models to pay attention to spatial relationships and consider other objects within the room. However, this also reinforces the bias that the input scene has been fully scanned, causing the detection model to rely on other objects within the room to guide predictions, thus limiting its prediction quality in online detection scenarios. A reasonable requirement when considering using such models is the ability to provide detection feedback to the user during acquisition, i.e., predicting what the user has just scanned; this is known as online detection. The purpose of this requirement is to suggest whether further scanning of a given part of the scene is needed, or whether the user can move to other parts of the scene.

[0009] Furthermore, in order to provide feedback to users during the data collection process, such models must be evaluated holistically across continuous subsets of the entire scene (including points already collected by the user). This suboptimal approach requires redundant computation, thereby increasing the consumption of computer resources.

[0010] Apple's RoomPlan is an example of an offline approach that attempts to solve online problems ("3D Parametric Room Representation with RoomPlan", Apple Machine Learning Research, accessed: October 13, 2022, [online], available at: https: / / machinelearning.apple.com / research / roomplan). To enable real-time, repeated object detection, they used the following design choices:

[0011] 1. They chose a 3D CNN architecture with a voxel size as high as 15cm, making it extremely difficult to identify objects smaller than that (such as books). As shown in the FCAF3D paper, reducing the resolution of state-of-the-art 3D CNNs from 1cm to 3cm already degrades the overall detection quality. This seems to be why RoomPlan only focuses on detecting large objects: "lockers, sofas, tables, chairs, beds, refrigerators, ovens, stoves, dishwashers, washing machines or dryers, fireplaces, sinks, bathtubs, toilets, stairs, and televisions."

[0012] 2. During online scanning, they use a local detector to predict bounding boxes directly from disjoint subsets of the input point cloud, and then aggregate these bounding boxes using an undisclosed algorithm.

[0013] One might argue that outdoor object detection methods, typically designed for robotics or autonomous driving, can already perform online 3D object detection. However, they do require consideration of spatiotemporal relationships to accurately detect objects that may be occluded in dynamic scenes.

[0014] However, the task of outdoor models is to detect potential obstacles, i.e., large objects, while rooms (indoor scenes) contain both large furniture and small objects such as books. Furthermore, outdoor detection aims to detect objects that occupy vertical space, such as pedestrians or cars; in fact, state-of-the-art 3D outdoor detection models, such as PointPillars (AHLang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom's "PointPillars: Fast Encoders for Object Detection From Point Clouds"), are widely used. "Point Clouds", published in the proceedings of the 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 12697-12705, accessed on October 14, 2022, [online], accessible at https: / / openaccess.thecvf.com / content_CVPR_2019 / html / Lang_PointPillars_Fast_Encoders_for_Object_Detection_From_Point_Clouds_CVPR_2019_paper.html, and "BEVFusion (Z. Liu et al., "BEVFusion: Multi-Task Multi-SensorFusion with Unified Bird's-Eye View"), (Representation) "arXiv, June 16, 2022, doi: 10.48550 / arXiv.2205.13542) or EA-LSS (H. Hu et al., "EA-LSS: Edge-aware Lift-splat-shot Framework for 3D BEV Object Detection" arXiv, August 29, 2023, doi: 10.48550 / arXiv.2303.17895) explicitly integrates this prior by embedding multi-sensor inputs into a unified 2D bird's-eye view (i.e., top-down). This simplification of the 3D problem to a 2D problem will not be applicable to indoor scenes containing objects arranged in a truly 3D manner, such as hanging plants or books on a table.

[0015] Therefore, improved solutions are needed to output localized representations of objects in 3D point clouds representing real 3D scenes (e.g., for object detection or scene segmentation). Summary of the Invention

[0016] Therefore, a method for a computer implementation of a machine learning function is provided. The function is configured to take 3D point cloud frames of a real-world scene as input and output localized representations. Each output representation is a corresponding object relative to the real-world scene. The method includes obtaining a dataset of sequences of 3D point cloud frames. Each frame is associated with a time interval in the sequence. Each frame contains a localized representation of each corresponding object. The method also includes training the function based on the obtained dataset. Training includes: for each sequence of the dataset and for each given frame of the sequence, training the function to output a localized representation of the object in the given frame based on the given frame and at least the frame from the previous time interval in the sequence.

[0017] The method may include one or more of the following:

[0018] This function includes:

[0019] o A first neural network, configured to take a frame as input and output at least a feature vector corresponding to a localized representation of each corresponding object in the frame; and

[0020] o A second neural network configured to aggregate feature vectors output by the first neural network for the given frame and at least the previous frame in the sequence;

[0021] The first neural network is configured to take frames as input and output two or more feature vectors corresponding to different resolution levels, and the second neural network is configured to perform two or more aggregations of feature vectors corresponding to the same resolution level.

[0022] The first neural network is a convolutional neural network (CNN), and / or the second neural network is a combination of a transformer neural network and convolution;

[0023] Each 3D point cloud frame of a real-world scene represents a partial view of the real-world scene.

[0024] Training includes batch training;

[0025] Each batch follows the temporal order of the frames, ensuring that during batch training, the function does not localize the output representation for that batch based on frames associated with future times;

[0026] In the obtained dataset, each point cloud frame sequence (X1, X2, ..., X...) T ),in, It is a point cloud frame at time t and contains n t Each point is defined according to the following rules:

[0027]

[0028] Where ceil is the floor function, and (I1,I2,…,I…) V ) is a sequence of depth images of the scene, and where, It is a backprojection operator that projects pixels of a depth image onto 3D points;

[0029] The scene is an indoor scene; and / or

[0030] The 3D point cloud frames of the obtained dataset are derived from physical measurements (e.g., physical scans) or from virtual measurements (e.g., virtual scans).

[0031] It also provides functions that can be obtained using this method.

[0032] A computer-implemented method for using this function is also provided. The method includes providing a sequence of 3D point cloud frames of a real-world scene. The method further includes, for each frame of the sequence, determining a localized representation of each corresponding object in the real-world scene within that frame by applying the function to that frame. The application of the function is based at least on feature vectors corresponding to the localized representations of each corresponding object in the previous frame.

[0033] A computer program is also provided, which includes instructions for performing the method and / or the method of use.

[0034] An apparatus is also provided, comprising a data storage medium having computer programs and / or functions recorded thereon.

[0035] The device can be formed or used as a non-transitory computer-readable medium (e.g., on SaaS (Software as a Service) or other services) or a cloud-based platform. The device may alternatively include a processor coupled to a data storage medium. The device can therefore form, wholly or partially, a computer system (e.g., the device is a subsystem of a larger system). The system may also include a graphical user interface coupled to the processor. Attached Figure Description

[0036] A non-limiting example will now be described with reference to the accompanying drawings, in which:

[0037] - Figures 1 to 6 The method is shown; and

[0038] - Figure 7 An example of the system is shown. Detailed Implementation

[0039] refer to Figure 1 The flowchart presents a method for a computer-based implementation of a machine learning function. The function is configured to take 3D point cloud frames of a real-world scene as input and output localized representations. Each output representation is a corresponding object relative to the real-world scene. The method includes obtaining a dataset of sequences of 3D point cloud frames. Each frame is temporally associated with the sequence. Each frame contains a localized representation of each corresponding object. The method also includes training the function based on the obtained dataset. Training includes: for each sequence of the dataset and for each given frame of the sequence, training the function to output a localized representation of the object in the given frame based on the given frame and at least the frame from the previous time in the sequence.

[0040] This method constitutes an improved solution for outputting localized representations of objects in a 3D point cloud representing a scene.

[0041] In practice, this method trains a function to perform this output using a sequence of 3D point cloud frames, each frame associated with a time interval in the sequence (i.e., the sequence is therefore temporal, with each frame corresponding to a time interval within the sequence). Such a sequence can typically correspond to a real-time 3D scan of a real-world scene, which progressively scans the scene, acquiring 3D point cloud frames continuously (i.e., at short, regular time intervals), each frame corresponding to a spatial portion of the scene and acquired at some point in the scanning process. For example, this could correspond to a user moving through a scene (e.g., in a furnished interior room) and operating a scanning device to scan the scene. By considering not only the current frame but also at least the previous frame in the sequence (i.e., considering frames at least the previous time interval in the sequence), the function is trained to output a localized representation for a given input frame. Therefore, the function learns to infer spatial relationships between different frames (i.e., spatial relationships between different regions of the scene) and temporal relationships between different frames in the sequence. This improves the accuracy of the output.

[0042] Furthermore, the function is trained with consideration for conserving computational resources during its use (also known as the online / inference phase / period). In reality, real-world scenes are captured as sequences of 3D point cloud frames (e.g., because it corresponds to acquisitions using 3D scanning, etc., which cannot physically obtain the same measurements of the entire scene from a single point cloud / image taken from a single viewpoint; alternatively, such a sequence could correspond to a user operating a scanning device (e.g., using their mobile phone) by walking through the scene and sequentially capturing the scene via a mobile device). Performing the determination of localization representation based on each individually captured frame can lead to insufficient accuracy, as spatial and / or temporal relationships between frames will not be considered, or at least not adequately considered. Conversely, performing the determination of localization representation using all frames together in the same computational step would increase the consumption of memory and computational resources in the computer system. This method provides a balance between these two possibilities by determining the localization representation in each given frame in the sequence using computations (e.g., only this frame, or the first two or three frames) already performed for at least the previous few frames in the sequence (e.g., feature vectors discussed below). As described above, this allows for consideration of spatial and temporal relationships between different frames of a sequence, thus providing accuracy while effectively reusing computations already performed. Specifically, in practice, for a given frame to be processed by the function, only computations performed for at least the previous frame (e.g., only the previous frame or the first two frames) need to be stored in the computer's RAM (Random Access Memory) or VRAM (Video RAM), or in the cache of the computer software application executing the method, so that the function can access these computations and use them with the given frame to output a localized representation of the given frame.

[0043] This method is used for machine learning on a function, which is a neural network (also called a "neural network function"), which can be a combination of neural networks and optionally a combination of one or more deterministic layers, which are themselves considered a neural network. Therefore, this method is a machine learning method for learning / training the function. As is known in the field of machine learning itself, the processing of inputs by a neural network involves applying operations to the inputs, which are defined by data containing weight values. Therefore, learning a neural network involves determining weight values ​​based on a dataset configured for such learning, which may be referred to as a learning dataset or a training dataset. For this purpose, the dataset comprises data blocks, each forming a corresponding training sample. The training samples represent the diversity of scenarios that the neural network will use after being learned. Any training dataset in this document may contain a number of training samples greater than 1,000, 10,000, 100,000, or 1,000,000. In the context of this disclosure, "learning a neural network (or function) based on a dataset" means that the dataset is a learning / training dataset for the neural network, on which weight values ​​(also called "parameters") are set. In this disclosure, the training dataset is a dataset of sequences of obtained 3D point cloud frames, on which the function is learned.

[0044] This function is configured (i.e., trained to) take 3D point cloud frames of a real scene as input and output a localized representation of each corresponding object in the real scene. For example, the function can take a sequence of point cloud frames as input and compute a localized representation for each frame in the sequence, using at least the previous frame (i.e., a frame at least one time in the sequence) or computations performed for said at least the previous frame (e.g., its feature vector as described above). Alternatively, the function can take each frame of the sequence as input individually and sequentially (one by one) according to the temporal order of the frames in the sequence, and compute a localized representation for each frame, using at least the previous frame (i.e., a frame at least one time in the sequence) or computations performed for said at least the previous frame (e.g., feature vector as described above), for example, by accessing the results of these computations (e.g., feature vectors) from the computer's RAM or VRAM or from the cache of the computer software application performing the method, as described above. In any case, the function can output the localized representation computed for each frame, or alternatively, a post-processing module can be applied to filter the predictions and retain the most relevant predictions before outputting them, as discussed further below.

[0045] A 3D point cloud frame is a 3D point cloud corresponding to a local view of a real-world scene; that is, a 3D point cloud representation of that local view. Any 3D point cloud or point cloud frame in this paper is a set of 3D points (i.e., each point is equipped with a triplet of coordinates in 3D space), each point representing a location within the scene (or a local view of the scene, where applicable) represented by the point cloud or point cloud frame. In the examples, each point may also be equipped with one or more additional coordinates (e.g., RGB coordinates) representing the color of that location. Therefore, in these examples, the function takes into account the colors in the scene. A real-world scene refers to a portion of the real world (e.g., a view of a real-world room, such as a kitchen). Any scene in this paper can be an interior scene, such as a furnished room. Therefore, the function is configured to take 3D point cloud frames as input (i.e., trained and structured to take 3D point cloud frames as input). This does not preclude the function from being used with multiple frames to output a localized representation of one of these frames while considering the others, as discussed further below.

[0046] This function outputs a localized representation of each corresponding object in the real-world scene (i.e., represented by the input point cloud frames). The localized representation of each object is data representing the object's geometric location within the scene, and data representing the object's semantic category (e.g., the object's type, such as furniture or a segmentation type used for scene segmentation). The data representing the geometric location can be a bounding box (e.g., a rectangle or circle) surrounding or substantially surrounding the object. The bounding box can be defined by a set of coordinates (x, y, z) representing the 3D location (e.g., its center), dimensions (w, l, h) (width, length, height), and orientation θ. The data representing the semantic category can be any suitable type of data, such as a label or a string description of the object within a descriptive box. The semantic category can be a segmentation semantic category (e.g., a segmentation type) or an object type (e.g., furniture or interior object type if the scene represents a room scene). All semantic categories in this document can belong to a predetermined set of semantic categories (e.g., 10 to 40 semantic categories, such as 32 semantic categories), for example, each semantic category relative to an object type (e.g., oven, dishwasher, refrigerator, etc.). Any object in this document can be a large object (e.g., large furniture such as a refrigerator) or a small object (e.g., large interior items such as books). In this document, a small object can be defined as having a volume less than 0.01m. 3 And / or objects with a maximum size less than 0.3m.

[0047] This method involves obtaining a dataset of sequences of 3D point cloud frames.

[0048] Therefore, the dataset consists of sequences, each a sequence of 3D point cloud frames. The point cloud frames of each sequence can all be associated with the same scene, meaning each frame of the sequence represents a partial view of the same scene (and thus relative to the sequence). The sequences in the dataset can be all or almost all associated with the same type of scene, such as scenes that are all or almost all interior scenes, for example, furnished rooms (e.g., all house interior scenes), or rooms of the same type (e.g., all kitchen scenes, or any other type of interior room). The extent to which all or almost all scenes represent the same type of room can vary and can depend on the intended use of the function. For example, if the function is intended to be specific to the same type of room (e.g., kitchen), then all or almost all scenes considered in the dataset could belong to this type (e.g., all kitchens). Alternatively, if the function is intended to be applied more generally to any interior scene, such as any interior room scene, then the dataset can contain sequences associated with various types of interior (e.g., room) scenes.

[0049] For each sequence, each frame is associated with a time within the sequence. This sorts the frames in the sequence according to time. For example, each sequence could be of type (X1, X2, ..., X...). T ),in, It is a point cloud frame associated with time t∈{1,…,T}. Therefore, any frame X t All have the previous frame X t-1 (Except for X1), also known as "the previous frame in the sequence", and the next frame X t+1 (X T Except for the time interval t), it is also called the "frame of the next time interval in the sequence". Each time interval t can be the acquisition time of the frame or can be inferred from that acquisition time (e.g., until rescaling, etc.).

[0050] Each frame contains a localized representation of each corresponding object. As described above, each localized representation (relative to the object) is data representing the geometric position of the object within the scene, and data representing the semantic category of the object (this category is, for example, the type of object, such as the type of furniture or the segmentation type used for scene segmentation). The data representing the geometric position can be a bounding box (e.g., a rectangle or a circle) surrounding or substantially surrounding the object. The bounding box can be defined by a set of coordinates (x, y, z) representing the 3D position (e.g., its center), dimensions (w, l, h) (width, length, height), and orientation θ. The data representing the semantic category can be any suitable type of data, such as the label of the bounding box or a string description of the object (or its category) within the descriptor box. The semantic category can be a semantic category of segmentation (e.g., a segmentation type) or a type of object (e.g., the type of furniture or interior object if the scene represents a room scene). All semantic categories in this paper can belong to a predefined set of semantic categories, for example, each semantic category relative to an object type (e.g., oven, dishwasher, refrigerator, etc.). Therefore, this function can be used to segment a real scene or to detect bounding boxes around objects in a scene and their associated semantic categories. This function can output localized representations of all or almost all objects (e.g., associated with categories belonging to a predetermined set of categories, such as a predetermined set of furniture and / or interior items). For this purpose, the training dataset can contain frames capturing appropriate variations and numbers of these objects, as is known in the field of machine learning.

[0051] Each sequence in the training dataset can be obtained from raw 3D data representing the scene corresponding to that sequence (e.g., data measured by physical sensors of the scene (such as scanning devices, e.g., 3D scanning), or data corresponding to virtual scans of the scene). An example of a process for obtaining sequences based on such raw 3D data will now be discussed. Obtaining the dataset may include performing this process, or alternatively retrieving (e.g., downloading) the dataset already obtained from this process from (e.g., remote) memory, a server, a database, or the cloud, where the dataset was stored prior to obtaining it.

[0052] The process begins with obtaining raw 3D data (i.e., for each sequence) as a video stream using a physical or virtual scanning device, where the value of each pixel corresponds to the distance between the object displayed in the image and the camera / scanning device (pixels may also contain RGB data as outlined earlier). Then, using known camera parameters (including its 3D position and orientation), each pixel is mapped to a point in 3D space according to a geometric backprojection operation T. Each of these images is called a depth image or depth map I and is backprojected into a different 3D point cloud T(I).

[0053] The resolution of any depth image in this paper can range from low resolutions of 128×128 to 256×192 (e.g., for consumer mobile devices) to 1920×1440 or higher (e.g., for professional devices). Each pixel corresponds to a 3D point, and the scanning device can also calculate additional confidence scores ranging from 1 (best) to 3 (worst) to estimate the accuracy of the position of each point, allowing for the elimination of potentially mismeasured locations. Furthermore, the device can select when to capture images based on heuristics indicating sufficient differences between each depth map, such as the time between captures, travel distance, or camera rotation. In this case, the camera's position and orientation can be obtained by combining, for example, odometry from the on-device accelerometer and / or registration algorithms.

[0054] As previously mentioned, any of the point clouds described above (e.g., point clouds obtained from depth maps) can be obtained from real devices in a real indoor scene or from a simulated camera moving within a virtual 3D scene. While real-world acquisition more closely matches the intended use case, simulated acquisition within a virtual environment with known geometry allows for the generation of robust synthetic data with rich semantic annotations without the need for a human annotator. Point cloud depth maps can be obtained, in particular, by applying the method for generating training datasets disclosed in European patent application EP23305001.2, which is incorporated herein by reference.

[0055] In the implementation, each point cloud frame sequence is represented as (X1, X2, ..., X...) in the obtained dataset. T ),in, It is a point cloud frame at time t and contains n t There are several points. The definition of this integer time t is deliberately loose because it depends on the implementation during the training phase. However, in the implementation, given the depth image sequence (I1, I2, ..., I...) as described above... V The resulting point cloud (i.e., the point cloud generated from the image) can be grouped into k image frames according to the following rules:

[0056]

[0057] Here, `ceil` is the floor function. Therefore, this process can achieve the rules described above. However, this process can alternatively regroup the images into frames based on other criteria, such as timestamps, relative positions, or differences between camera parameters. It should be noted that these frames can overlap, allowing any given object to appear (at least partially) in multiple frames of the sequence.

[0058] In the implementation, the values ​​of the above variables can include:

[0059] The number of images (or views) V depends on the scene. Within this range, the average number of images acquired in simulated scenarios is 101; for real-world devices, a given scene could correspond to tens of thousands of images.

[0060] The number of images k per frame can be arbitrary and can be used to control the trade-off between having a large number of frames but fewer points per frame (k=1) and reverting to an offline setting with a single point cloud frame (k=∞). The inventors tested... The values ​​within the range were ultimately selected as k=10, thus yielding the average value for T=10 frames for each scene. This grouping is important in testing: for k=1, the inventors found that the local network f, which calculates the spatial relationships within each frame, does not have enough spatial information to enable the online model to make accurate predictions.

[0061] The total number of points in a given scene can range from several thousand to tens of millions; for ease of calculation, this method can calculate X points per frame. t sampling For each point, setting the upper limit of the value range can yield better results.

[0062] It should be noted that other methods exist for obtaining point clouds, and these methods can be used in alternative implementations. For example, some specialized devices do not output depth maps but instead output 3D point clouds directly based on 360° scans. It should be noted that any such device still needs to address occlusion issues, requiring the user to move it to fully capture complex scenes. Therefore, regardless of how the point cloud is obtained, the dataset consists of frames (X1, X2, ..., X...). T The sequence is composed of ).

[0063] In addition to obtaining the training dataset, the method also includes training a function based on the obtained dataset (i.e., the obtained dataset is the training dataset used to train the function). Training includes: for each sequence of the dataset and for each given frame of the sequence, training the function to output a localized representation of the object in the given frame based on the given frame and at least the previous frame in the sequence (i.e., using only the previous frame in the sequence, or possibly using one or more previous frames).

[0064] This function may include (e.g., be or include a combination thereof) two neural networks. These two neural networks include a first neural network and a second neural network. The first neural network is configured to take frames as input and output at least a feature vector corresponding to a localized representation of each corresponding object in the real-world scene. The feature vectors, also known as "embeddings" in machine learning, form a compact representation used to capture the localized representations (e.g., all localized representations) in the frames (i.e., 3D point cloud frames) obtained by the first neural network as input. The second neural network is configured to aggregate the feature vectors output by the first neural network for the given frame and at least the frames from the previous time in the sequence. In other words, for each sequence encountered, the first neural network takes all frames of the sequence as input (continuously or in batches) and outputs a corresponding feature vector for each corresponding frame as input. For each feature vector output by the first neural network, the second neural network aggregates that feature vector with the feature vectors of the frames associated with the previous time in the sequence, and may also aggregate with one or more feature vectors corresponding to one or more frames associated with one or more times prior to the previous time (e.g., the previous time, two times prior, or three times prior). The first neural network, denoted by f, can be called the "local network" or "backbone network." This neural network performs an evaluation exactly once for each point cloud frame in the input sequence. The second neural network, denoted by g, can be called the "aggregation network." This neural network receives the outputs of the local neural network in sequence and outputs either the prediction for the last frame in the sequence or the prediction for each frame in the sequence. This function may also include a detection head applied to the aggregation result, as discussed further below.

[0065] The first neural network can be configured to take frames as input and output two or more feature vectors, each corresponding to a different resolution level. In other words, the first neural network outputs two or more feature vectors for each input frame, and each output vector corresponds to a different resolution level of the localized representation of the object in the frame. These different resolution levels allow for the consideration of objects of different sizes (i.e., objects of various sizes). In this case, the second neural network is configured to perform two or more aggregations, each aggregation corresponding to the feature vector at the same resolution level. In other words, for each frame (or feature vector) involved in the aggregation performed by the second neural network (i.e., a given input frame and one or more preceding frames), these frame feature vectors are aggregated two or more times, once for each resolution level (i.e., one aggregation is performed on all frame feature vectors at the highest resolution level, one aggregation is performed on all frame feature vectors at the second highest resolution level, and so on, up to the lowest resolution level). The first neural network can be a convolutional neural network (CNN). The second neural network can be a combination of a transformer neural network and convolution. The method can consider two, three, or four resolution levels. Each resolution level can be or corresponds to a depth level. The resolution levels for aggregation can be 8cm, 16cm, 32cm and 64cm (of which the 16cm and 32cm levels are particularly useful).

[0066] First, for each point cloud frame X... t Independently evaluate the local neural network (or backbone network) f and output intermediate features (or embeddings) Z. t =f(X) t In the implementation, these features are intermediate computations of a custom variant of TR3D (discussed in “TR3D: Towards Real-Time Indoor 3D Object Detection”, D. Rukhovich, A. Vorontsova, and A. Konushin, arXiv, February 8, 2023, doi: 10.48550 / arXiv.2302.02858, which is incorporated herein by reference), allowing the application of a lightweight parametric function (referred to as the “detection head” in the previously cited reference) h to provide initial frame-by-frame predictions. Therefore, these two functions The combination of these can have an architecture similar to a 3D object detection neural network and is heavily inspired by TR3D. In the implementation, these initial predictions... They are not calculated during the inference phase. They are used as part of the loss function when training the neural network.

[0067] Then, the convergent neural network g uses the intermediate features Z output by the local neural network. t To generate the final prediction in, like Figure 1 As shown in the figure, this diagram illustrates the combined architecture of local neural networks and convergent neural networks in the implementation, outputting a prediction. Only corresponds to the current frame X t The model considers the objects in {2, ..., t}, but depends on the intermediate features of the past δt frames. Due to hardware limitations, the model can consider at most all past frames by choosing δt = t. However, to save computational resources, it is preferable to consider smaller values ​​of δt, such as δt = 2, δt = 3, or δt = 4 (because fewer previous frames will be considered, thus affecting the computer memory for a given input frame). Therefore, δt can take any value in {2, ..., t}, but is preferably a value less than t, such as δt = 2, δt = 3, or δt = 4.

[0068] In this implementation, the aggregation network g corresponds to the sequence of the fusion aggregation module (FAM), with a detection head h appended to it. This is as follows: Figure 2As shown, it illustrates the architecture. FAM is based on TransPillars (discussed in "TransPillars: Coarse-to-Fine Aggregation for Multi-Frame 3D Object Detection" by Luo, G. Zhang, C. Zhou, T. Liu, S. Lu, and L. Pan, arXiv, August 4, 2022, doi: 10.48550 / arXiv.2208.03141, which is incorporated herein by reference), which uses a concept similar to 2D feature maps for online outdoor detection. This implementation applies this concept to 3D features as follows: transforming image markers (tiles), applying a transformer, and recombining the tiles in the output. As previously mentioned, and as is known in the CNN field, features at different resolution levels (or depths) can be output to account for different possible sizes of the object to be detected. For example, the previously discussed TR3D outputs two features for each input point cloud, while FCAF3D (discussed in D. Rukhovich, A. Vorontsova, and A. Konushin, “FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection”, arXiv, March 24, 2022, accessed October 11, 2022, [online], available at http: / / arxiv.org / abs / 2112.00322, which is incorporated herein by reference) outputs four features for each input. Therefore, the aggregation network implementation includes at least one FAM for each resolution level. Figure 2 An aggregation network with two resolution levels and one FAM for each resolution level is shown, which corresponds to the implementation of this method.

[0069] In this implementation, features from the high-resolution level are first processed and then passed to the next FAM (Features from another FAM) to fuse features from its resolution level with those from the higher-resolution level. The outputs of all FAMs are then fed into the detection head to output the final prediction. However, for clarity and readability, Figure 2The connection between the FAM and the detection head is not shown. In these implementations, each FAM consists of 3D convolutional layers and Transformer layers (discussed in A. Vaswani et al., “Attention Is All You Need”, arXiv, December 5, 2017, doi: 10.48550 / arXiv.1706.03762, which is incorporated herein by reference) and deformable attention (such as in X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai, “Deformable DETR: Deformable Transformers for End-to-End Object Detection”). ObjectDetection ("arXiv", March 17, 2021, accessed October 17, 2022, [online], accessible at: http: / / arxiv.org / abs / 2010.04159, discussed in this paper by reference). The FAM architecture consists of... Figure 3 As shown.

[0070] It should be noted that the above descriptions of the aggregation network g and FAM are implementation details of the neural network, which has been tested by the inventors and is considered to provide satisfactory results. However, suitable alternative architectures or modifications can be considered, such as using scaled dot product attention (discussed in the previously discussed reference A. Vaswani et al., “Attention Is All You Need”, arXiv, December 5, 2017, doi: 10.48550 / arXiv.1706.03762) instead of the deformable attention discussed previously, or using a coarse-to-fine fusion aggregation order instead of a fine-to-coarse aggregation order. Variants of FCAF3D or TR3D can be considered as the network for the local network f. These alternatives have all been tested and provide satisfactory results.

[0071] The function may optionally include a post-processing part that applies non-maximum suppression (NMS) to filter predictions and retain the most relevant ones.

[0072] Training may include labeled training samples, minimizing losses known in the art. Similar to most deep learning methods, the function can be trained in its implementation using a stochastic gradient descent algorithm, such as AdamW (discussed in I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization,” arXiv, January 4, 2019, doi: 10.48550 / arXiv.1711.05101, which is incorporated herein by reference). In tests conducted by the inventors, the proprietary annotated training, validation, and testing datasets used for the function were generated using a HomeByMe virtual scene synthesized using the method described in the previously discussed European patent application EP23305001.2, as stated above.

[0073] Training can include batch training. Each batch can follow the temporal order of the frames, so during batch training, the function does not output a localized representation for that batch based on frames associated with future times. In other words, for training / during training, the aggregation network can operate in batch mode, thus outputting all predictions corresponding to the input frames:

[0074]

[0075] This batch processing pattern follows a chronological order (also known as causality): much like prediction. Independent of Z t+1 , Independent of Z t Instead, it relies on all features with indices ≤ t-1. This allows for parallel training of the model across multiple frames or even multiple scenes using a graphics processing unit (GPU), while still providing a large number of predictions that can be used for the loss function.

[0076] In the implementation method, online neural networks It is implemented in Python, using the PyTorch library (see A. Paszke et al.'s "PyTorch: An imperative-style, high-performance deep learning library"). The paper is discussed in “Research Progress in 32 Neural Information Processing Systems” by H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox and R. Garnett, Eds, Cologne Associates, 2019, pp. 8024-8035, [online], accessible at: http: / / papers.neurips.cc / paper / 9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf, which is incorporated herein by reference). Automatic differentiation is performed using the Minkowski engine (see reference C. Choy, J. Gwak and S. Savarese, “4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks”). The paper discusses sparse convolution operations in "Networks" (arXiv, June 13, 2019, doi: 10.48550 / arXiv.1904.08755, which is incorporated herein by reference).

[0077] The training process is implemented by... Figure 4 As shown.

[0078] The inventors conducted tests using the above-described implementation method. The results are shown in Table 1 below:

[0079]

[0080] Table 1 - Online test results of the dataset obtained using the method of European patent application EP23305001.2

[0081] As shown in the table above, the quantitative results demonstrate that the implementation of this method far surpasses that of the baseline method. The baseline method can be described as follows: The inventors did not use an online model. Instead, the final δt frame (X) was cascaded. t ,X t-1 ,…,X t-δt+1Furthermore, it employs offline methods such as FCAF3D. This baseline method is essentially what a person skilled in the art would use to generate online 3D object detection models.

[0082] Commonly used metrics in 3D object detection are mean precision (mAP) and mean recall (mAR). These metrics are defined to evaluate the model's prediction results using the intersection-over-union (IoU) threshold. For a given class c, if a predicted bounding box can be paired with a ground truth bounding box such that the IoU (intersection-over-union ratio) between the two bounding boxes is higher than the given threshold, then the predicted bounding box is considered a correct prediction. The intersection-over-union ratio (IoU ∈ [0,1]) can be described as follows. For 3D bounding boxes, these union and intersection values ​​are calculated as volume; for clarity, Figure 5 The 2D equivalent is visualized as an area. Exemplary IoU thresholds for NMS (Non-Maximum Suppression) are 0.25 and 0.5. Pairing is unique because a given ground fact can be assigned at most one prediction, and vice versa. Pairing is performed in descending order of IoU until all ground facts or prediction bounding boxes are exhausted.

[0083] Consider the following definition:

[0084] • True Positive (TP): A correct model prediction, i.e., a prediction that matches a bounding box of the same class, such that IoU > threshold.

[0085] • False positive (FP): An incorrect model prediction, i.e. the model predicts a bounding box but does not match the corresponding ground truth annotation or IoU ≤ threshold.

[0086] • False negative (FN): Missing prediction. A note that does not match any prediction (i.e., the target exists but was not detected by the model).

[0087] Precision is the proportion of actually correct positive predictions:

[0088]

[0089] • Recall rate is the proportion of actual positive instances that the model can correctly identify:

[0090]

[0091] Precision-Recall Curve: The trade-off between the quality and quantity of predicted bounding boxes can be controlled after inference, typically in the form of a confidence score associated with each class and bounding box. This score can be thresholded to remove unreliable predictions, allowing for plotting precision and recall as a function of the confidence score threshold.

[0092] • Mean precision (AP) is the area under the precision-recall curve for each class (e.g., ... Figure 6 (As shown).

[0093] • Mean precision (mAP) is the average of AP across all categories. We typically use mAP_threshold to specify at which IoU threshold we calculated the metric (i.e., at which quality level).

[0094] • Frame-by-frame average precision (fmAP) corresponds to mAP in the online settings, where precision and recall are calculated per frame, not per scene.

[0095] • Average precision (AR), mAR, and fmAR can be defined in a similar way.

[0096] It also provides functions that can be obtained according to this method, namely, computer-implemented data structures that form the function on a computer system and have parameters (e.g., neural network weights) with values ​​to be obtained through training according to this method. Such functions can be obtained directly from this method, i.e., the values ​​of the parameters are directly derived from training according to this method. In any case, this data structure can be stored (and provided as is) in computer (e.g., non-transitory) memory.

[0097] A computer-implemented method is also provided, which uses a function that is available (e.g., learned) through the method. The method includes providing a sequence of 3D point cloud frames of a real-world scene. Providing the sequence of 3D point cloud frames may include: operating a 3D scanning device (e.g., integrated into a user's mobile phone) by user operation or automatic operation, and using the device to scan the scene continuously or iteratively. The scanning may directly generate 3D point cloud frames of the scene, or it may generate an image of the scene (e.g., a depth image). In the latter alternative, providing the sequence may include converting each image into 3D point cloud frames known in the art, thereby generating the sequence. Each frame of the sequence is associated with time, which may be derived from (e.g., until rescaling, etc.) for example, an acquisition time provided by the scanning device. The method also includes: for each frame of the sequence, determining a localized representation of each corresponding object in the real-world scene in that frame by applying a function to that frame. Therefore, the application of the function is performed iteratively frame by frame. The application of the function is based at least on feature vectors corresponding to the localized representation of each corresponding object in the previous frame. For example, the application of this function may include: for each frame, applying a first neural network to that frame to extract a feature vector for that frame. Then, the application of this function may include: applying a second neural network to perform aggregation of the feature vector extracted for that frame and the feature vectors extracted for at least the previous frame (meaning that, in the aggregation, the feature vectors of the previous frame are used, and optionally, the feature vectors of one or more frames from times prior to the previous time are also used). After aggregation, final detection of the localized representation can be performed by applying the detection head discussed earlier to the aggregation result. The learning method and the usage method can be integrated into the same computer implementation method, which includes performing the learning method and then performing the usage method of the function learned by the learning method.

[0098] Therefore, the trained function offers both computational efficiency and accuracy when operating in an online environment (i.e., when the method is applied). In fact, by design, the heavier local network (backbone network) evaluates only once per point cloud frame, while the lighter aggregation network is responsible for refining its output into frame-by-frame predictions. Thus, when the aggregation network is applied to perform feature vector aggregation, only the feature vectors of the current frame and the previous frame (and optionally, where appropriate, the feature vectors of one or more frames from one or more times prior to the previous time) need to be stored in the cache of the computer software application executing the method on the RAM (Random Access Memory) or VRAM (Video RAM) of the underlying computer. In contrast, existing offline indoor 3D object detectors must evaluate on a concatenation of all previous frames in the sequence to produce acceptable results. The accuracy stems from the fact that the aggregation network simultaneously computes spatial relationships (i.e., spatial relationships between different regions of the scene) and temporal relationships (i.e., temporal relationships between different frames in the sequence). This efficient use of RAM or cache, in particular, allows for the determination of localized representations of a scene in real time (e.g., almost instantaneously, as the scene is captured, such as when scanned by a user moving through the scene). It also allows for real-time visual feedback to the user to determine if scene acquisition was performed well.

[0099] Furthermore, during point cloud acquisition, users can move the scanning device to capture different parts of a given indoor scene. In reality, some parts of the scene may be occluded by other objects or not even within the device's field of view. In this context, the method, functionality, and usage allow for the output of 3D semantic information related to the most recently captured points, based on previously acquired data from the scene.

[0100] These methods are implemented by computers. This means that the steps (or essentially all steps) of the method are executed by at least one computer or any similar system. Therefore, the steps of the method may be executed by the computer fully or semi-automatically. In the example, at least some steps of the method may be triggered through user-computer interaction. The required level of user-computer interaction may depend on the anticipated level of automation and be balanced with the need to fulfill the user's wishes. In the example, this level may be user-defined and / or predefined.

[0101] A typical example of a computer implementation of this method is to use a system suitable for this purpose to execute the method. This system may include a processor coupled to memory and a graphical user interface (GUI), on which a computer program containing instructions for performing the method is stored. The memory may also store a database. The memory is any hardware suitable for such storage, and it may comprise several physically distinct sections (e.g., one section for the program, and possibly one section for the database).

[0102] Figure 7 An example of a system is shown, where the system is a client computer system, such as a user's workstation.

[0103] The client computer in this example includes a central processing unit (CPU) 1010 connected to an internal communication bus 1000 and random access memory (RAM) 1070 also connected to the bus. The client computer also includes a graphics processing unit (GPU) 1110 associated with video random access memory 1100 connected to the bus. The video RAM 1100 is also referred to in the art as a frame buffer. A mass storage device controller 1020 manages access to mass storage devices, such as hard disk drives 1030. Mass storage devices suitable for tangibly representing computer program instructions and data include all forms of non-volatile memory, for example including semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks and removable disks; and magneto-optical disks. Any of the foregoing may be supplemented or incorporated into a specially designed ASIC (Application-Specific Integrated Circuit). A network adapter 1050 manages access to a network 1060. The client computer may also include a haptic device 1090, such as a cursor control device, a keyboard, etc. A cursor control device is used on the client computer to allow the user to selectively position the cursor at any desired location on the display 1080. Additionally, the cursor control device allows the user to select various commands and input control signals. The cursor control device includes multiple signal generating devices for inputting control signals to the system. Typically, the cursor control device can be a mouse, with mouse buttons used to generate signals. Alternatively or additionally, the client computer system may include a sensitive pad and / or a sensitive screen.

[0104] A computer program may include computer-executable instructions, including means for causing the system to perform the method. The program may be recorded on any data storage medium, including the system's memory. For example, the program may be implemented in digital electronic circuits, or in computer hardware, firmware, software, or a combination thereof. The program may be implemented as a device, for example, a product tangibly embodied in a machine-readable storage device for execution by a programmable processor. The method steps may be executed by a programmable processor that executes the program of instructions to perform the function of the method by manipulating input data and generating output. Thus, the processor may be programmable and coupled to receive and send data and instructions from the data storage system, at least one input device, and at least one output device. The application program may be implemented in a high-level procedural language or an object-oriented programming language, or, if desired, in assembly language or machine language. In any case, the language may be a compiled or interpreted language. The program may be a complete installation program or update program. In any case, the application program generates instructions on the system for performing the method. Alternatively, the computer program may be stored and executed on a server in a cloud computing environment that communicates with one or more clients over a network. In this case, the processing unit executes the instructions contained in the program, thereby enabling the method to be executed in a cloud computing environment.

Claims

1. A computer-implemented method for a machine learning function, configured to take a 3D point cloud frame of a real scene as input and output a localized representation of each corresponding object in the real scene, the method comprising: A dataset of sequences of 3D point cloud frames is obtained, each frame being associated with time in the sequence, and each frame including a localized representation of each corresponding object; as well as The function is trained based on the obtained dataset, the training comprising: for each sequence of the dataset and each given frame of the sequence, training the function to output a localized representation of an object in the given frame based on the given frame and at least the previous frame in the sequence.

2. The method as described in claim 1, wherein, The functions include: A first neural network is configured to take a frame as input and output at least one feature vector corresponding to a localized representation of each corresponding object in the frame; and The second neural network is configured to aggregate the feature vectors output by the first neural network for the given frame and the previous frame in the sequence.

3. The method of claim 2, wherein: The first neural network is configured to take frames as input and output two or more feature vectors corresponding to different resolution levels; and The second neural network is configured to perform two or more aggregations corresponding to the feature vectors at the same resolution level.

4. The method as described in claim 2 or 3, wherein: The first neural network is a convolutional neural network (CNN); and / or The second neural network is a combination of a transformer neural network and convolution.

5. The method according to any one of claims 1 to 4, wherein, Each 3D point cloud frame of the real scene represents a partial view of the real scene.

6. The method according to any one of claims 1 to 5, wherein, The training includes batch training.

7. The method of claim 6, wherein, Each batch follows the temporal order of the frames, such that during training of the batch, the function does not localize the output representation for the batch based on frames associated with future times.

8. The method according to any one of claims 1 to 7, wherein, In the obtained dataset, each point cloud frame sequence (X1, X2, ..., X...) T ),in, It is a point cloud frame at time t and contains n t Each point is defined according to the following rules: Where ceil is the floor function, and (I1,I2,…,I…) V ) is a sequence of depth images of the scene, and wherein, It is a backprojection operator that projects pixels of a depth image onto 3D points.

9. The method according to any one of claims 1 to 8, wherein, The scene described is an indoor scene.

10. The method according to any one of claims 1 to 9, wherein, The 3D point cloud frames of the obtained dataset are derived from physical measurements or from virtual measurements.

11. A function that can be obtained by the method according to any one of claims 1 to 10.

12. A computer-implemented method using the function according to claim 11, the method comprising: Provides a sequence of 3D point cloud frames for a real-world scene; as well as For each frame of the sequence, the function is applied to the frame at least based on the feature vector corresponding to the localized representation of each corresponding object in the previous frame to determine the localized representation of each corresponding object in the real scene in the frame.

13. A computer containing instructions that, when executed by the computer, cause the computer to perform the method as claimed in any one of claims 1 to 10 and / or the method as claimed in claim 12.

14. A computer-readable data storage medium having a computer program as claimed in claim 13 and / or a function as claimed in claim 11 recorded thereon.

15. A computer system, including a processor coupled to a memory, the memory storing a computer program as claimed in claim 13 and / or a function as claimed in claim 11.

Images