Object classification

Abstract

Examples described herein relate to methods and apparatus for classifying an object. In one example there is provided a method of classifying an object. The method comprises obtaining a plurality of 2D images (250, 260) of an environment (100), the received 2D images associated with different poses (136), and performing object segmentation on the obtained 2D images to identify one or more object masks (255X, 255Y, 265X, 2565Z) associated with respective obtained 2D images, the objects masks associated with respective detected objects and object mask classifications. The method determines an object (385) within a 3D map (380) of the environment; matches the determined object (385) with the detected objects in at least some of the object masks (255X, 255Y, 265X, 265Z); and classifies the determined object using the object mask classifications of the object masks associated with matching detected objects.

Description

[0001]CLASSIFYING AN OBJECT Technical Field Examples of the present disclosure relate to visual object detection and classification. Background Visual object detection and classification is useful in many fields including augmented or extended reality (XR), autonomous vehicles, robotics and surveillance systems. Object classification may be based on machine vision and machine learning technology including the use of deep learning models such as Mask Region Based Convolutional Neural Network (R-CNN) trained with large datasets to reach high classification accuracies. However, these models are still trained with limited data in which it may not be possible to generalize to recognize an object in every situation in real-world scenarios. This situation may be exacerbated in multi-view scenarios where there is significant variation in the object’s appearance across different views or if the object is occluded or partially visible in some views. This may cause subpar classifications in situations which they are not trained for, such as different viewpoints, lighting conditions, and object classes. A geometric characterization or three dimensional (3D) representation of objects in an environment may be performed using simultaneous localization and mapping (SLAM) algorithms. This allows a device such as a Smartphone or extended reality devices such as Smart glasses to geometrically map the environment as 3D data and simultaneously localize itself within the environment. Machine learning models such as object detection and segmentation networks may then be used to semantically understand what those 3D data mean. Instance segmentation may be employed for classifying each pixel of an image into a set of object classes. However, running such processes simultaneously with other processes such as SLAM is computationally demanding. Such an approach is described in Chilbane, Julian, et al. “Box2Mask: Weakly Supervised 3D Sementic Instance Segmentation using Bounding Boxes.” Computer Vision-ECCV 2022: 17thEuropean Conference, Tel Aviv, Isreal, October 23-27, Proceedings, Part XXXI. Cham: Springer Nature Switzerland, 2022. In an alternative approach, Pillai, S., & Leonard, J. (2015). “Monocular slam supported object recognition”. arXiv preprint arXiv:1506.01732 uses 2D images for semantic understanding, using bounding boxes. Summary It is an object of the present disclosure to provide improved methods and apparatus for object classification. In one aspect there is provided a method of classifying an object. Obtaining a plurality of two dimensional (2D) images of an environment, the received 2D images associated with different poses. Performing object segmentation on the obtained 2D images to identify one or more object masks associated with respective obtained 2D images, the objects masks associated with respective detected objects and object mask classifications. Determining an object within a 3D map (380) of the environment. Matching the determined object with the detected objects in at least some of the object masks. Classifying the determined object using the object mask classifications of the object masks associated with matching detected objects. Using semantic information from object masks from multiple 2D images of an object to classify the object within a 3D map enables object classification to be performed with reduced processing requirements. This may enable real-time processing even on reduced processing devices such as XR glasses and Smartphones. In another aspect there is provided an apparatus configured to perform the following. Obtain a plurality of 2D images of an environment, the received 2D images associated with different poses. Generate a 2D image based on a said obtained 2D image depending on the pose of said obtained 2D image. Generate a sparse representation of an object corresponding to an object detected in said obtained 2D image, wherein the sparse representation of the object is associated with fewer pixels than the object detected in said obtained 2D image. According to certain embodiments described herein there is also provided a computer program comprising instructions which, when executed on a processor, cause the processor to carry out methods described herein. The computer program may be stored on a non-transitory computer readable media. Brief Description of the Drawings For a better understanding of examples of the present disclosure, and to show more clearly how the examples may be carried into effect, reference will now be made, by way of example only, to the following drawings in which: Figure 1 is a schematic view of an object recognition and classification system for an environment and according to an example; Figure 2 illustrates different views and segmentation masks for the environment of Figures 1, according to an example; Figure 3 illustrates a partial 3D representation of the environment of Figure 1, according to an example; Figure 4 illustrates displays representing the environment of Figure 1, according to an example; and Figure 5 is a flow chart for a method of classifying an object, according to an example; Figure 6 is a flow chart for a method of representing an object, according to an example; and Figure 7 illustrates the projection of 3D object points into two dimensional (2D) images having object mask classifications, according to an example. Detailed Description The following sets forth specific details, such as particular embodiments or examples for purposes of explanation and not limitation. It will be appreciated by one skilled in the art that other examples may be employed apart from these specific details. In some instances, detailed descriptions of well-known methods, nodes, interfaces, circuits, and devices are omitted so as not obscure the description with unnecessary detail. Those skilled in the art will appreciate that the functions described may be implemented in one or more nodes using hardware circuitry (e.g., analog and / or discrete logic gates interconnected to perform a specialized function, ASICs, PLAs, etc.) and / or using software programs and data in conjunction with one or more digital microprocessors or general purpose computers. Nodes that communicate using the air interface also have suitable radio communications circuitry. Moreover, where appropriate the technology can additionally be considered to be embodied entirely within any form of computer-readable memory, such as solid-state memory, magnetic disk, or optical disk containing an appropriate set of computer instructions that would cause a processor to carry out the techniques described herein. Hardware implementation may include or encompass, without limitation, digital signal processor (DSP) hardware, a reduced instruction set processor, hardware (e.g., digital or analogue) circuitry including but not limited to application specific integrated circuit(s) (ASIC) and / or field programmable gate array(s) (FPGA(s)), and (where appropriate) state machines capable of performing such functions. Embodiments relate to classifying visual objects in an environment using an efficient computations approach. This may be suitable for resource constrained devices such as XR glasses and Smartphones. Figure 1 is a schematic view of an object recognition and classification system for an environment and according to an example. A device or apparatus 120 moves through an environment 100 along a path indicated by 134. The environment comprises a number of objects 110A, 110B, 110C and the device 120 obtains a plurality of 2D images of the objects from different views 136. Each view 136 or 2D image is associated with a viewing position and orientation referred to herein as a pose within the environment. The device 120 may be configured to perform object detection on the 2D images to identify objects within the environment. In an example, this may be implemented using a segmentation approach which results in object masks for each object recognized in an obtained 2D image. Each object mask comprises pixels corresponding to a recognized object, these pixels associated with an object mask classification. Pixels corresponding to the recognized object will be associated with an object mask classification such as “chair”, other pixels of the 2D image may have a null value. Segmentation for generating the object masks may be performed in any suitable manner, for example using a trained machine learning model such as the Mask R-CNN model. In some examples, the object masks may be downsampled to reduce the number of pixels in order to reduce subsequent processing. The downsampling may utilize object feature recognition such as edges or specific shapes and filter out pixels not contributing to these object features. The device 126 may be configured to determine a 3D map of the environment, for example using a simultaneous localization and mapping (SLAM) algorithm. This will generate 3D data typically in the form of a point cloud in three dimensions of the environment, although a 2D map may be sufficient in some applications. Points in the point cloud may be clustered to determine objects within the 3D map. Different types of clustering may be employed, such as spatial clustering and / or texture or color based clustering. Feature recognition may also be utilized to determine the objects within the 3D data. In order to reduce subsequent processing, a sparse representation of these objects may be used. A sparse representation uses only some of the points that could be provided for in the 3D map, for example using a lower than maximum point resolution and / or one or more predetermined geometric features such as edges or circumferences. A sparse 3D map may be created from a sequence(s) of 2D images. An algorithm such as ORB or SIFT may be used to find edge like features in each image. Then by matching the features in the two or more 2D images geometric constraints can be imposed on the feature position relative to the poses of camera(s) taking the images. The geometric constraints correspond to the feature being in 3D space and where the feature’s position in the 2D image relates via a simple projection from 3D space to 2D. Each feature can therefore be positioned in 3D space via bundle adjustment or similar optimization procedures which find the optimal value of the position of the feature in 3D space to minimize a goodness of fit criteria, such as least squares. The object recognition and 3D mapping functions may be performed in parallel; with their outputs used for further processing as described below. Using sparse representation of an object in the 3D map and / or downsampling of the object masks reduces the processing requirement of the apparatus and may allow for real time processing even for lower powered devices such as XR glasses, Smartphones, or drones. Whilst the device may perform these functions, in other examples, the device 120 may request object detection from an external source, such as a third party computing resource coupled to the device by a communications network. Similarly, the device may determine the 3D map of the environment by downloading this from an external source or using a non-SLAM algorithm. In a further alternative, a separate imaging device may be employed to obtain 2D images of the environment together with corresponding poses. The functionality of the device described below may also be distributed in other ways. The device 120 may generate and display a 2D image corresponding to an obtained 2D image, depending on the current pose of the device within the environment 100. The device 120 may be configured to display a representation of and / or object information about recognized objects 110A, 110B, 110C. The representation and / or object information may depend on how the recognized objects are classified. From some views 136, some objects may be partially occluded and so a representation or information about the object may be inaccurate. For example, a chair in which only one corner of the seat and a leg are seen from a first view may be incorrectly classified as a table and a representation of a table or information about a table displayed. From a second view, the chair may be fully seen and then correctly classified as a chair. A representation of a chair or information about a chair may then be displayed. As the device 120 moves about the environment 100, the representations of and / or object information about the object may change. In some examples, the device integrates the object recognition data from multiple different views or poses 136 within the environment with geometric data determined from the 3D map. Points from an object determined within the 3D map may be projected into multiple obtained 2D images or object masks and matched with corresponding pixels in the 2D images or object masks. Those object points in the 3D map may then be associated with classifications associated with the matching pixels from the object masks. In some views or 2D images these classifications may be different. A classification for each point of a determined object or all points of the determined object in the 3D map may then be determined based on a function of the corresponding classifications from the object masks having matching pixels. For example, a majority decision may be made so that if most matching pixels have a “chair” classification but some have a “table” classification, the classification of the determined object in the 3D map is assigned as “chair”. The classification assigned to the object determined in the 3D map may then be used for various applications such as updating the classification of all matching pixels in the object masks from the obtained 2D images. This may improve the semantic understanding of the environment, including for example the accuracy of the object masks, displayed representations of the objects and / or displayed object information. This approach also reduces the amount of processing required in 3D, which is typically heavy, as object recognition processing is carried out in 2D. Semantic understanding is performed in 2D not 3D, using the segmentation of obtained 2D images. Projection between 3D points and 2D pixels of object masks is then performed, and classifications from pixels matching these points are then associated with these points. Therefore, semantic understanding or object recognition is not performed on 3D object representations. To further reduce processing, generation of the 3D map of the environment may be carried out using a sparse geometrical representation and determined objects registered with or matched with the objects recognized in 2D. This is significantly less processing intensive than 3D object recognition. This may allow the device to avoid using a graphical processing unit (GPU) which might otherwise be required for processing based on more dense representations. In an example, the device 120 of Figure 1 comprises a processor 122, memory 124, a 2D image capture device 130 such as a camera, and a display 132. The memory 124 comprises instructions 126 and a 2D object recognition model 128. The memory may also comprise a SLAM algorithm. The instructions 126, when executed on the processor 122, may cause the processor to carry out a number of steps 140 – 148. At 140, the processor obtains a plurality of 2D images of an environment 100 and which are associated with different poses or views 136. The obtained 2D images may be associated with different times, for example as an apparatus obtaining the images device moves through an environment. Such an apparatus may be a 2D capture device such as a camera. The obtained 2D images are obtained together with position data for the apparatus obtaining the 2D images A pose for a 2D image may be determined using accelerometers, gyroscopes, GPS coordinates, movement tracking and any other suitable mechanism associated with the apparatus capturing the 2D image. At 142, the processor performs object segmentation on the obtained 2D images to identify one or more masks associated with respective obtained 2D images. The object masks are associated with respective detected objects and object mask classifications (e.g. “chair”). The processor may implement this using the 2D object recognition model 128. At 144, the processor determines an object within a 3D map of the environment. This may be performed using a SLAM algorithm to generate object points or by downloading the map from an external source. The object may be determined by clustering object points within the 3D map, for example using spatial and / or color based clustering. At 146, the processor matches the determined object with the detected objects in the object masks. This may be implemented by mapping the geometric coordinates of the object points in the 3D maps into the 2D object masks of the obtained 2D images and identifying any object mask pixels that match the position of the 3D points. This is known as reprojecting from 3D into 2D, and uses position and orientation information of the object points within the 3D map as well as corresponding position and orientation information derived from the poses associated with the object masks. At 148, the processor classifies the determined object using the object mask classifications of the object masks having matching pixels. This may be implemented by associating object points of the determined object with classifications of matching pixels in the object masks. If all of these classifications are the same (e.g. “chair”), then the object points may be assigned this classification. If the classifications are different, then an aggregating or arbitrating function may be employed such as majority decision. As noted, these instructions or functions may be performed within the device 120, or may be distributed across other devices such as external computing resources, data stores for 3D maps or 2D images and so on. Figure 5 illustrates a method of classifying an object according to an example. The method 500 may be implemented by the device 120 or by any other suitable equipment. Reference is also made to Figures 2 – 4, and 7. At 505, the method obtains a plurality of 2D images of an environment, for example using an image capture device such as a camera or downloading the 2D images. The 2D images are associated with different poses or views of the environment. At 510, the method performs object segmentation of the 2D images to identify object masks having pixels with object mask classifications, as previously described. The object masks may be downsampled to reduce the number of pixels and hence the amount of processing required. This may involve feature matching to identify pixels corresponding to edge or other features. Figure 2 illustrates two different 2D images 250, 260 of an environment, the 2D images associated with different poses within the environment. These images are simplified for simplicity of explanation, however more realistic images are illustrated in Figure 7. The two 2D images 250, 260 each show a view or representation of objects 210A1, 210B1, 210C1 and 210A2, 210B2, 210C2, which correspond to objects 110A, 110B, 110C respectively of Figure 1. Below each 2D image 250, 260, are shown two object masks 255X, 255Y and 265X, 265Z respectively. These object masks correspond to recognized objects in the two 2D images. Each object mask comprises pixels corresponding to the pixels of the 2D image, each object mask pixel corresponding to the detected object is associated with an object mask classification, such as “chair” or “table”. The other object mask pixels not associated with this classification may be associated with a null value or removed from the object mask. As the two 2D images 250, 260 comprise different views of the objects 210A – 210C, different objects are recognized in each set of object masks. For example, object mask 255X recognizes objects 210A1, 210B1 as a single object (the long rectangle) in which object 210B1 is recognized as part of object 210A1. By contrast, in the different view of 2D image 260, object 210A2 is recognized as a single object (the shorter rectangle) and object 210B2 is recognized as a separate object (the circle). Object 210C1 is only recognized in the first 2D image 250. At 515, the method comprises determining a sparse representation of an object having object points or features in a 3D map of the environment. This is performed using a separate process such as a SLAM algorithm. Such sparse representation processing is often performed at the beginning of a more detailed map generation process. The 3D map determining process and the object recognition process may be performed at the same time in parallel. Figure 3 illustrates a 3D map 380 having a point cloud 310B representing a determined object 385, which corresponds to object 110B in Figure 1. For simplicity only this determined object 385 and associated point cloud 310B is shown, however it will be appreciated that point clouds may be used to represent other objects. The determined object 385 may be determined by clustering points in the 3D map 380. Clustering could be based on geometry or spatial clustering and / or other types of clustering such as color based using RGB values. The object 385 may also or alternatively be determined using feature recognition such as edges 382, shapes or other features. At 520, the method matches object points from the sparse representation of the object in the 3D map with pixels from the object masks from different 2D images. In this way, an object point from the sparse representation may be matched with a pixel from more than one object mask. This may be implemented by reprojecting the points from the 3D map into the 2D images / object masks using location data from the 3D map and location data for the object masks based on the pose information for the corresponding 2D image. In Figure 2, the second 2D image 260 shows 2D points projected from the 3D map into the 2D image.2D points 270 matches recognized object 210B2 and 2D point 272 falls outside of the recognized object 210B2. The reprojected points of the 3D determined object that match pixels in the object mask 265Z can then be associated with the classification associated with the object mask. This process may be repeated for different 2D images and the object masks of those 2D images – that is reprojecting 3D points from the determined object in the 3D map into different 2D images / object masks and assigning the classification of matching pixels from the object masks back to the 3D points. Figure 7 illustrates two 2D images 750, 760 of an environment and associated with different poses. The 2D images show chairs and tables 710A, 710B, 710C overlaid with pixels 712T, 712C, 712L from object masks which correspond to 3D points projected from a sparse representation of determined objects from a 3D map. The sparse representation of the objects corresponds to a point cloud in a 3D map in contrast to a dense texture based dense representation for example.. The pixels 712T, 712C corresponding to the sparse representation of the 3D object may be associated with a classification if they fall within the boundary of an object mask for the 2D. In the illustrated example, “x” represents a “table” classification, and a triangle represents a “chair” classification. It can be seen that in the upper image 750, the table 710C overlaps with some pixels having a “chair” classification (triangles) and so may incorrectly be classified as a chair. In the lower image, with a better view of table 710C, the table overlaps with pixels having the correct "table“ classification (“X”) and so is likely to be correctly classified as a table in this image 760. Some pixels 712L corresponding to an object point in the 3D map may have a low prediction score in the object mask (illustrated as small squares) and may then be filtered out to avoid degrading the classification of the 3D object. Noise 716 (illustrated as small circles) from the object masks may also be filtered out. At 525, the method associates object points of the sparse representation with object mask classifications of object mask pixels matching the object points. Because the objects may be defined or classed differently in different 2D images, an object point in the 3D sparse representation may initially be associated with different classifications. The classification of the correspondence pixels 712T, 712C may be associated with the corresponding object point in the 3D map. The classification of pixels in different object masks and 2D images corresponding to the same object points in the 3D map may also be associated with these object points. At 530, the method classifies the determined object using the associated object mask classifications. Where multiple object mask classifications are associated with a 3D object point, a function is used to assign a classification to the determined object. In one example, this may be based on a majority decision, but alternative functions may be used. A classification for an object point or an object to which this belongs can then be determined based on the accumulation of classifications from corresponding pixels in different object masks. Various algorithms may be used for determining a classification for an object in the 3D map using the corresponding input object mask classifications – for example majority voting as noted previously, but other functions are possible and may depend on other factors such as weighting classifications based on pose, distance to object and so on. This approach helps to correct inaccurate classifications that may be derived from some 2D images that may have an occluded or otherwise poor view of the object. As previously described, 3D objects in the map may be determined using clustering, for example spatial and / or textual clustering.3D points from each cluster or determined object may then be projected into the 2D object masks from different 2D images or views in order to collect classifications from matching pixels. The most common classification may then be assigned to the entire cluster. Alternatively the classifications may be retained as semantic descriptors. At 535, the method may perform further processing using the classification of the object. In one example, some object masks having incorrect classifications may be updated or corrected so that their pixels corresponding to the 3D determined object are changed depending on the classification from corresponding pixels in other object masks. For example, the classifications for these pixels in the different object masks may all be updated to the most common classification. In another example, a display showing an obtained 2D image having representations of objects may be updated to also show a sparse representation of these objects based on the spare representation from the 3D map. This is illustrated in Figure 4 in which a first generated 2D image 432-1 reflects an obtained 2D image. The second image 432-2 displays a sparse representation of the objects derived from the 3D map. This may show hidden information about an object which is partially occluded in the displayed 2D image 432-1. The sparse representations may be overlaid on the generated 2D image 432-1. The display 432-2 may alternatively or additionally be updated to show object information 490 about the objects, such as their classification or derivative information that depends on their classification. Referring to Figure 6, another method according to an example is illustrated. This method 600 may be implemented by the device 120 of Figure 1, or any other suitable apparatus. At 605, the method 600 obtains a plurality of 2D images of an environment, the 2D images associated with different poses. This corresponds with step 505 of method 500 in Figure 5 and can be implemented in similar manner. At 610, the method 600 displays a generated 2D image based on an obtained image and depending on the pose of the obtained image. At 615, the method generates a sparse representation of an object corresponding to an object detected in the obtained 2D image. The sparse representation of the object is associated with fewer pixels than the object detected in said obtained 2D image. The sparse representation may simply show an outline of the object or certain features such as edges. The sparse representation in the generated 2D images may be updated over a sequence of poses of the device 120, corresponding to changing views of the object. As some views of the object may be better than others, the accuracy or quality of the representation may similarly change. In some examples, as the classification of the object accumulates over the sequence of poses, an accumulated or classification may be determined and this may be retained for subsequent poses where the view of the object is poor. The sparse representation, based on the classification, may then also be updated over the sequence of poses and retained for subsequent poses where the view of the object is poor. For example, the updated representation of the determined object may be retained when returning to one of the poses of the sequence of poses. The sparse representation may be generated based on multiple of the plurality of 2D images which may improve the displayed sparse outline or edge features in some generated 2D images. For example, these features may be less apparent in some poses. The sparse representation may be generated using a sparse representation of the object in a 3D map. This 3D representation of the object may be determines using a SLAM algorithm for example. The determined object in the 3D map may be updated with information gathered from the plurality of obtained 2D images, for example improving boundaries and classifications. Examples provide a number of advantages. These include avoiding heavy semantic calculations in 3D – examples working with sparse reprojected points in 2D space where semantic processes are run. This reduces the computational burden compared with known approaches using 3D dense reconstruction to understand an environment semantically. For resource constrained apparatus such as XR-glasses, mobile phones, or beyond visual line of sight unmanned aerial vehicles, this reduction in computational power enables real-time processing and / or reduced costs. When such apparatus cannot rely on shared maps or offloading of computation, the 3D calculations may become prohibitive and the benefits of multi- view object detection cannot be realized. Using the much-reduced computational effort of examples makes the calculation effort tractable even in such cases. Aggregating object detection (segmentation) data from different poses or views leverages an implicit behavior from feature extraction algorithms to refine borders of segmented objects, which is also typically the object mask border. Further, the accuracy of object classification may also be improved using multi-view aggregation scheme of the examples. This may be achieved whilst using limited processing. It should be noted that the above-mentioned examples illustrate rather than limit the invention, and that those skilled in the art will be able to design many alternative examples without departing from the scope of the appended statements. The word “comprising” does not exclude the presence of elements or steps other than those listed in a claim or embodiment, “a” or “an” does not exclude a plurality, and a single processor or other unit may fulfil the functions of several units recited in the statements below. Where the terms, “first”, “second” etc are used they are to be understood merely as labels for the convenient identification of a particular feature. In particular, they are not to be interpreted as describing the first or the second feature of a plurality of such features (i.e. the first or second of such features to occur in time or space) unless explicitly stated otherwise. Steps in the methods disclosed herein may be carried out in any order unless expressly otherwise stated. Any reference signs in the statements shall not be construed so as to limit their scope.

Claims

Claims 1. An apparatus configured to: obtain a plurality of 2D images of an environment, the received 2D images associated with different poses; generate a 2D image based on a said obtained 2D image depending on the pose of said obtained 2D image; generate a sparse representation of an object corresponding to an object detected in said obtained 2D image, wherein the sparse representation of the object is associated with fewer pixels than the object detected in said obtained 2D image.

2. The apparatus of claim 1, wherein the sparse representation of the detected object is based on two or more classifications of the detected object determined from two or more of said plurality of 2D images.

3. The apparatus of claim 1 or 2, wherein the apparatus is operative to update the sparse representation of the object over a sequences of poses, and to retain the updated sparse representation of the object when returning to one of the poses of the sequence of poses.

4. The apparatus of any one preceding claim, wherein the apparatus comprises one or more of the following: a 2D image sensor, a 2D image display, a processor and a memory.

5. The apparatus of any one preceding claim, wherein the sparse representation of the object comprises a predetermined geometric feature.

6. The apparatus of claim 5, wherein the predetermined geometric feature is an edge.

7. The apparatus of any one preceding claim, not comprising a graphical processing unit.

8. A method of classifying an object, the method comprising: obtaining a plurality of 2D images of an environment, the received 2D images associated with different poses; performing object segmentation on the obtained 2D images to identify one or more object masks associated with respective obtained 2D images, the objects masks associated with respective detected objects and object mask classifications; determining an object within a 3D map (380) of the environment; match the determined object with the detected objects in at least some of the object masks; and classifying the determined object using the object mask classifications of the object masks associated with matching detected objects.

9. The method of claim 8, comprising generating a sparse representation of the determined object, wherein the sparse representation of the determined object is associated with fewer pixels than one or more of the detected objects in the at least some of the object masks.

10. The method of claim 9, wherein the sparse representation of the determined object comprises a predetermined geometric feature.

11. The method of claim 10, wherein the predetermined geometric feature is an edge.

12. The method of any one of claims 8 to 11, wherein the object classification is a function of the object mask classifications of more than one of the at least some of the object masks.

13. The method of claim 12, wherein the function is the most common object mask classification.

14. The method of any one of claims 8 to 13, wherein the obtained 2D images are associated with different times and are obtained together with position data for an apparatus generating the 2D images.

15. The method of any one of claims 8 to 14, comprising generating a 2D image based on an obtained 2D image and comprising a representation of the determined object.

16. The method of any one of claims 8 to 15, comprising generating a 2D image based on an obtained 2D image and comprising object information for the determined object, the object information based on the classification of the determined object.

17. The method of any one of claims 8 to 14, comprising: displaying, on an apparatus, a 2D image based on an obtained 2D image, depending on the pose of the apparatus within the environment; displaying, on the apparatus, one or more of the following: a representation of the determined object; object information for the determined object, the object information based on two or more object mask classifications from two or more of said plurality of received 2D images.

18. The method of claim 17, comprising updating the representation of the determined object or the object information over a sequences of poses of the apparatus, and retaining the updated representation of the determined object or the object information when returning to one of the poses of the sequence of poses.

19. The method of any one of claims 8 to 18, wherein determining the object within the 3D map of the environment comprises determining a plurality of object points; and wherein matching the determined object with the detected objects in at least some of the object masks comprises matching at least some of the object points with at least some pixels of the at least some of the object masks.

20. The method of claim 19 comprising: updating a number of the matching pixels of at least one object mask using the classification of the determined object associated with the number of matching pixels.

22. The method of any one of claims 8 to 21, wherein the object masks comprise pixels associated with prediction scores for the respective object mask classification, and wherein the pixels are filtered using the prediction scores.

23. A computer program comprising instructions which, when executed on a processor, cause the processor to carry out the method of any one of claims 8 to 22.

24. A computer program product comprising non-transitory computer readable media having stored thereon a computer program according to claim 23.

Images